robots.txt Crawling and Stats

I found robots.txt Adventure really compelling.

Anyone with the time and inclination to do broad studies of all the web, or at least as wide a portion as you can get your arms around, is fine by me.

It’s impressive enough when someone like Google does it, as in 2005’s Web Authoring Statistics, but when an individual does it, that’s some compelling stuff.

The uncov boys are doing something pretty interesting with Persai, but haven’t showed anything public yet.