anti crawler - Python
<p>I know to write Python crawler with beautiful soup module. Now I want to detect if someone crawls my website. How to do that. Can someone point me to pesudo code or sourcecode. Basically I am looking to write anti-crawler in python. </p>

<p>It's hard, but can do things to filter crawlers.</p>

<p><strong>Auth</strong></p>

<p>Show pages only to authorized users. </p>

<p><strong>Strong Captcha</strong></p>

<p>If your captcha system strong enough, can anti a part of crawlers.</p>

<p><strong>User Agent</strong></p>

<p>Requests from crawler may not set user agent, so you can block those requests have no user agent(or not right user agent).</p>

<p><strong>Request Frequency</strong></p>

<p>Some crawler requests your website more faster than human, block their IP.</p>

<p><strong>JS load</strong></p>

<p>Using javascript to load your html, normal crawl only get the base html before javascript loading completed.</p>

<p><strong>Temporary URL</strong></p>

<p>Your can encode your url with timestamp(or something else) and public the url to user for visiting, making some crawlers cannot find entrance.</p>

<p><strong>Or Any other</strong></p>

<p>....</p>

python python-2.7 python-3.x web-crawler google-crawlers