witten
/
luminotes
Archived
1
0
Fork 0
Code Releases Activity

Various updates to disallowed robot pages.

This commit is contained in:
Dan Helfman 2008-10-15 12:25:48 -07:00
parent b18cd1dca2
commit c3a0c1bb4c
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
static/html/robots.txt
View File

@ -2,7 +2,9 @@
User-agent: *
Disallow: /users/demo
Disallow: /users/thanks
Disallow: /notebooks/download_html/
Disallow: /users/thanks_download
Disallow: /notebooks/export_csv/
Disallow: /notebooks/export_html/
# this crawler is completely broken and requests many invalid URLs
User-agent: panscient.com