anti crawler - Python
<p>I know to write Python crawler with beautiful soup module. Now I want to detect if someone crawls my website. How to do that. Can someone point me to pesudo code or sourcecode. Basically I am looking to write anti-crawler in python. </p>

<p>What about assuming that not all crawlers are nasty? Most do respect the <a href="http://www.robotstxt.org/" rel="nofollow">robots directives</a>. Of course you can implement all sorts of heuristics to block robots but the first thing you do would be to have </p>

<pre><code>User-agent: *
Disallow: /
</code></pre>

<p>in a robots.txt at the root of your site.</p>

<p>Then if you really want to make things hard for the ones who do not follow robots.txt then use Javascript for all the links : not 100% guaranteed to block robots but it will make their lives a lot more difficult.</p>

python python-2.7 python-3.x web-crawler google-crawlers