In case you came here looking for a reference to netflix based on this, well, I like looking at peoples’ robots.txt files.
Truth be told, I learn a great deal by looking at how sites do their thing. I remember visiting amazon a long time ago and seeing how they generate extra url stuff and wondering what it was. I distinctly remember opening and closing browsers to see how I could mess with it and see if I could fool it. I also love tweaking url strings to see if sites still work when the urls are messed with. As you see what happens as you add and remove elements you get a sense for what’s happening “under the hood.”
I suppose my behavior is an amplified version of a a monkey with a twig poking at anthills.
