witten
/
luminotes
Archived
1
0
Fork 0
Code Releases Activity

Prevent web crawlers from spidering the "download as html" version of a notebook.

This commit is contained in:
Dan Helfman 2008-01-02 03:27:31 +00:00
parent e7ae481eea
commit c4a2cf6713
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
static/html/robots.txt
View File

@ -1,6 +1,7 @@
# prevent bots from signing up for demo accounts
User-agent: *
Disallow: /users/demo
Disallow: /notebooks/download_html/
# this crawler is completely broken and requests many invalid URLs
User-agent: panscient.com