[Haskell-cafe] haskell wiki indexing

Vincent Kraeutler vincent at kraeutler.net
Tue May 22 08:45:01 EDT 2007


as was pointed out on the programming reddit [1], crawling of the
haskell wiki is forbidden, since http://www.haskell.org/robots.txt contains

User-agent: *
Disallow: /haskellwiki/

and indeed, a google search gives the old wiki
http://www.google.ch/search?q=haskell+wiki
i.e.
http://haskell.org/hawiki/FrontPage
rather than
http://haskell.org/haskellwiki/Haskell

in other words, it seems like the most relevant search engines do not
index the haskell wiki. i have reported this on #haskell about a week
ago[2], and this setting was acknowledged to be a measure deliberately
taken to curb excessive load from spiders crawling the wiki. i still
believe this a highly harmful stance, and just to make sure that whoever
may feel concerned is at least aware of the problem, i am now
cross-posting to this list as well.

kind regards,
v.

[1] http://programming.reddit.com/info/1qzjx/comments/c1r3ok
[2] http://tuukka.iki.fi/tmp/haskell-2007-05-16.html







-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 249 bytes
Desc: OpenPGP digital signature
Url : http://www.haskell.org/pipermail/haskell-cafe/attachments/20070522/b53fdc48/signature-0001.bin


More information about the Haskell-Cafe mailing list