robots.txt Crawling and Stats2007Sep23

I found robots.txt Adventure really compelling.

Anyone with the time and inclination to do broad studies of all the web, or at least as wide a portion as you can get your arms around, is fine by me.

It’s impressive enough when someone like Google does it, as in 2005’s Web Authoring Statistics, but when an individual does it, that’s some compelling stuff.

The uncov boys are doing something pretty interesting with Persai, but haven’t showed anything public yet.

wrote this Sunday September 23rd 2007 at 4:37pm That was 13 years ago. Leave a comment

Leave a Reply