witten/luminotes
Archived
1
0
Fork 0
Code Releases Activity

Disallowing one particular broken bot, and adding some comments.

Browse Source
This commit is contained in:
Dan Helfman 2007-11-07 19:43:09 +00:00
parent 3b7a5422ac
commit 4807db7feb
1 changed files with 5 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
static/html/robots.txt
View File

@ -1,2 +1,7 @@
# prevent bots from signing up for demo accounts
User-agent: *
Disallow: /users/demo
# this crawler is completely broken and requests many invalid URLs
User-agent: panscient.com
Disallow: /