All,<br><br>Referencing this thread (<a href="http://www.haskell.org/pipermail/haskell-cafe/2007-May/025769.html">http://www.haskell.org/pipermail/haskell-cafe/2007-May/025769.html</a>), I have prepared a small update to robots.txt
. It doesn't do much, but should exclude all dynamic pages from being spidered. Please vet it and let me know of any concerns. Note I'm not really interested in reconfiguring the wiki to have nicer URLs - what is there is what I'm working with right now.
<br><br>Here's the updated robots.txt (availabe as <a href="http://www.haskell.org/robots2.txt">http://www.haskell.org/robots2.txt</a> right now):<br><br> User-agent: *<br> Disallow: /haskellwiki/Special:<br> Disallow: /haskellwiki/Special%3A
<br> Disallow: /haskellwiki/?<br> Disallow: /haskellwiki/%3f<br><br>Any feedback, let me know. Thanks!<br><br>Justin<br>