robots.txt Crawling and Stats

I found robots.txt Adventure really compelling.

Anyone with the time and inclination to do broad studies of all the web, or at least as wide a portion as you can get your arms around, is fine by me.

It’s impressive enough when someone like Google does it, as in 2005’s Web Authoring Statistics, but when an individual does it, that’s some compelling stuff.

The uncov boys are doing something pretty interesting with Persai, but haven’t showed anything public yet.

posted this 16 years ago.
What else did he post in September 2007?

(Sunday September 23rd 2007 at 4:37pm)

Leave me a comment!

This site uses Akismet to reduce spam. Learn how your comment data is processed.