Dan Helfman
|
994e31884d
|
Web spiders should not be creating forum threads.
|
2008-11-12 22:44:25 -08:00 |
Dan Helfman
|
c3a0c1bb4c
|
Various updates to disallowed robot pages.
|
2008-10-15 12:25:48 -07:00 |
Dan Helfman
|
f6538ad3a5
|
First pass at "thank you" page (and related processing/error pages) is complete.
|
2008-01-11 22:55:04 +00:00 |
Dan Helfman
|
c4a2cf6713
|
Prevent web crawlers from spidering the "download as html" version of a notebook.
|
2008-01-02 03:27:31 +00:00 |
Dan Helfman
|
4807db7feb
|
Disallowing one particular broken bot, and adding some comments.
|
2007-11-07 19:43:09 +00:00 |
Dan Helfman
|
7ebdf4aa26
|
New robots.txt file disallowing /users/demo so search engine spiders don't create a bunch of demo accounts.
|
2007-10-19 22:53:41 +00:00 |