robots.txt Crawling and Stats

I found robots.txt Adventure really compelling.

Anyone with the time and inclination to do broad studies of all the web, or at least as wide a portion as you can get your arms around, is fine by me.

It’s impressive enough when someone like Google does it, as in 2005’s Web Authoring Statistics, but when an individual does it, that’s some compelling stuff.

The uncov boys are doing something pretty interesting with Persai, but haven’t showed anything public yet.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

This post accepts webmentions. If you link to and write about this on your website, enter the web address of your post below: