search engines

You are currently browsing articles tagged search engines.

Bagels & Lox

June 8, 2002 in imported | No comments

Bagels & Lox

Today started with an early morning breakfast with my friend Steve E. at local deli D.Z. Akins. We talked of course about my current state of affairs, the separation, work, the web, search engines, the list. D.Z.A. is a great deli. Now, it’s not a Cantor’s or Carnegie Deli, but for San Diego, it rocks. I had something I’ve not had in a long time—a bagels and lox platter. What a sublime invention – this erector set of a breakfast. Assemble fat slices of fresh tomato, red onion, cream cheese, and of course the smoked salmon itself, all piled onto a bagel, and that is some great eating. Now, mind you, my appetite since this all started has been shot to hell. I’m continuing to lose weight too, less eating, more walking and exercise—feels good. But breakfast was inspiring.

Combine good food with good company—wonderful. Steve I met at a local user group some years ago at the Digital Design SIG —and it was he who sent me the Frank Zappa/Mothers ticket stub on my Zappa fan piece. We talked more about migrating the list off of yahoogroups. It’s going to be work, but I think it’s probably necessary. This summer, I keep seeing. When I can focus some attention on it, I need to figure out what features I’m using on yahoogroups and then begin to migrate.

After that, I went to racquetball. The problem is that the joint where I usually play was closed. So instead, my Aunt and I got our cars detailed. Banal, but my car looks great! For a 14 year old car that’s been cross-country twice and all over, it looks rejuvinated.

The next cancellation was the Barbecue I was supposed to go to today. The hostess got sick, so no party. So instead of making macaroni salad, I went to a thrift store and a comic book store.

Jenny called, left a message. She needs to come over and pick up some stray mail and clothes. Sometime in the next few days probably. We’ll see.

Tomorrow I should probably do laundry.

I think I’m through the crushing shock part of separation. Now I’m in the living part of it. I’m just trying to have a life, and be myself. It’s not impossible. I bet if you asked me that one month ago I’d have something entirely different to say. But life goes on.

Onward.

Tags: , , , ,

Search Engine Upgrade to ht://Dig

January 27, 2002 in imported | No comments

Search Engine Upgrade to ht://Dig
Last night and this morning I installed htdig as the new search engine for this site internally. Back in August I mentioned that I would start using google because the solution I was trying had stopped working with any reliability. ht://Dig is open source and originated here in San Diego at SDSU.

Total time for installation and customization was about 5 hours total. This is valuable information in case I ever need to install an htdig search engine for a client. Lots of small details in doing this installation. I downloaded the installation as a tar.gz file, then decompressed that to a suitable location (cgi-bin). Then I had to do configure, make, make install. Installing unix software is always an adventure. This site runs FreeBSD (see: colophon, and I was delighted that it went pretty smoothly.

Then I was ready to start running it. This got tricky, but it was straightforward as I was able to tweak the conf/htdig.conf file to do what I like. rundig is the key to indexing a site. At first I had broken images, but it was working properly. The site initially indexes the htdig site itself. Just like any web robot, it goes out and looks at that site just as a browser would. This put my mind at ease, as I was not sure how it would deal with databased content, or the fact that the pages on my site are very include() driven. I was also concerned that because it is a local search engine, it would index files I don’t want indexed. The perl search engine I had originally installed had this problem. It would find older versions of files and garbage files that had become garbage for a reason.

As I got it working, and pointed it at artlung.com, I found a problem. The indexing process was taking far too long. Seems I had an infinite loop happening! In my accessibility slideshow from 1999 I had a problem. The [next] and [previous] links did not give any thought to whether they should actually show or not. The php for that I had written when I really knew very little php, and I ended up with the search engine indexing not just /words/accessibility/?i=0 to /words/accessibility/?i=10, but it was iteratively visiting the “next” and “previous” links like crazy. ?i=-1, ?i=-2, ?=-3, and on until I stopped it at ?i=-115. That would have been 115 versions of the “previous” page that was no different than the “first” poge. The PHP I had written in 1999 was smart enough to handle bad values for $i, but not smart enough to realize that there was no “previous” pages for those pages. The “next” links had the same problem. The htdig indexer was not smart enough to know that it was indexing hundreds of nearly identical pages. The solution was to fix the slideshow code so that it would not produce spurious links like that. After that fix, it was indexed properly and quickly. This is probably another reason that many search engines simply won’t touch pages with querystrings.

The next problem I had was that it was showing bad search results for certain pages. Example: I searched for the word “Zappa” – and I got far more results than I would have expected. Granted, I am a Frank Zappa Fan, but why would the bio page come up in a result for that? Turns out the indexer found the entry inside the bottom

Subscribe To The Feed