ArtLung

robots.txt Crawling and Stats 2007 Sep 23

I found robots.txt Adventure really compelling.

Anyone with the time and inclination to do broad studies of all the web, or at least as wide a portion as you can get your arms around, is fine by me.

It’s impressive enough when someone like Google does it, as in 2005’s Web Authoring Statistics, but when an individual does it, that’s some compelling stuff.

The uncov boys are doing something pretty interesting with Persai, but haven’t showed anything public yet.

blogged this at 4:37pm in 2007 in September. The 23rd was a Sunday. You are reading this 12 years later. Comment. There are no comments Tweet. Send email. It has no hastags.

Leave a Reply

Comments Open; Trackbacks Open.