Perl

Java, Pascal Classes; Fall 1998; Becoming
In the Fall of 1998 I started to take classes in Pascal and Java at Santa Monica College – the Pascal class was interesting for a while, but I dropped it. My own reading at the time was doing what I needed. At the time the only programming I was doing was some Perl and some JavaScript. The Java class was pretty mind-blowing. I grokked object oriented code, but the class rapidly got ahead of my own experiences of OO code. I remember being right there with the instructor for about 3 weeks, then I was in way over my head. I’m glad I took those classes, but I am only a dilettante programmer. My skills are multivariate; this is my strength. I need to get comfortable with that.

I have been going through all my old papers and objects. This morning I came across my notebook from SMC. I write it here because I’m throwing out the notebook.

I’ve got a zen calm about letting go of these things I have packratted.

I am ever in the process of becoming. Refining. Rehabilitating. Remaking. Reclaiming.

Onward.

Search Engine Upgrade to ht://Dig
Last night and this morning I installed htdig as the new search engine for this site internally. Back in August I mentioned that I would start using google because the solution I was trying had stopped working with any reliability. ht://Dig is open source and originated here in San Diego at SDSU.

Total time for installation and customization was about 5 hours total. This is valuable information in case I ever need to install an htdig search engine for a client. Lots of small details in doing this installation. I downloaded the installation as a tar.gz file, then decompressed that to a suitable location (cgi-bin). Then I had to do configure, make, make install. Installing unix software is always an adventure. This site runs FreeBSD (see: colophon, and I was delighted that it went pretty smoothly.

Then I was ready to start running it. This got tricky, but it was straightforward as I was able to tweak the conf/htdig.conf file to do what I like. rundig is the key to indexing a site. At first I had broken images, but it was working properly. The site initially indexes the htdig site itself. Just like any web robot, it goes out and looks at that site just as a browser would. This put my mind at ease, as I was not sure how it would deal with databased content, or the fact that the pages on my site are very include() driven. I was also concerned that because it is a local search engine, it would index files I don’t want indexed. The perl search engine I had originally installed had this problem. It would find older versions of files and garbage files that had become garbage for a reason.

As I got it working, and pointed it at artlung.com, I found a problem. The indexing process was taking far too long. Seems I had an infinite loop happening! In my accessibility slideshow from 1999 I had a problem. The [next] and [previous] links did not give any thought to whether they should actually show or not. The php for that I had written when I really knew very little php, and I ended up with the search engine indexing not just /words/accessibility/?i=0 to /words/accessibility/?i=10, but it was iteratively visiting the “next” and “previous” links like crazy. ?i=-1, ?i=-2, ?=-3, and on until I stopped it at ?i=-115. That would have been 115 versions of the “previous” page that was no different than the “first” poge. The PHP I had written in 1999 was smart enough to handle bad values for $i, but not smart enough to realize that there was no “previous” pages for those pages. The “next” links had the same problem. The htdig indexer was not smart enough to know that it was indexing hundreds of nearly identical pages. The solution was to fix the slideshow code so that it would not produce spurious links like that. After that fix, it was indexed properly and quickly. This is probably another reason that many search engines simply won’t touch pages with querystrings.

The next problem I had was that it was showing bad search results for certain pages. Example: I searched for the word “Zappa” – and I got far more results than I would have expected. Granted, I am a Frank Zappa Fan, but why would the bio page come up in a result for that? Turns out the indexer found the entry inside the bottom box for my frank zappa piece. So the search engine was indeed finding an instance of the word “Zappa,” but not a useful one. The solution is to not include the bottom navigation in the pages served to the search engine. I also did the same thing with the blog such that the archived pages don’t show the outbound links to the indexer. In this way you don’t get each and every instance in navigation when you use the search engine. I suspect there will be more tweaks like this.

Next I began playing with the look and feel to match the rest of artlung.com. I used my preexisting styles and made graphical widgets for search results. They are pretty cute, actually. Buttons should look like they want to be clicked, and these even have a bevel! You can’t go wrong with a subtle bevel. I added the ht://Dig logo in a way to my liking (a banner along the bottom). I ended up changing several pages in htdig’s common directory: footer.html , header.html, long.html , and nomatch.html. I also edited the template_map variable to point to my own long.html file.

Once I got it working to my satisfaction I reran rundig manually for the last time, and swapped out the old google include for the new htdig include on the search page. With a highly include-driven site it’s very easy to make these kinds of changes.

The last thing I did was add a daily unix system cron job to reindex the site daily. The indexing process takes about 2 minutes and the best part about it is that I shouldn’t have to think about it. And it should always be up to date (plus or minus 24 hours I suppose).

With luck, this search engine will work well. If you have comments or questions, feel free to ask!

Tags: bad search results, htdig search engine, htdig site, html, include-driven site, local search engine, Perl, perl search engine, PHP, san-diego, search page, search engine, search engines, search results, unix, unix software, unix system, web robot

Comparative Anatomy for Server Side Scripting

August 20, 2001 in imported | No comments

Comparative Anatomy for Server Side Scripting – two small, trivial examples of the same thing, done different ways – in Perl, in Cold Fusion, in PHP, and ASP.

Tags: Perl

Initially this site had a search powered by a script from Matt’s Script Archive…

August 5, 2001 in imported | No comments

Initially this site had a search powered by a script from Matt’s Script Archive. Based on the timestamps on files I think I installed it in March of 1999. Last month I replaced it with another service. Problem is, a few weeks later, based on testing it now and again – it didn’t work more than it did work.

So what’s the answer?

Google! of course. Google comes around and indexes roughly monthly. What that means is that the blog will not always be instantly searchable. And day-to-day changes will only be searchable with a lag. But Google is good, and predictable.

If I can find a PHP solution that’s painless I may change again.

I learned a great deal on that first perlbased search though. Installing, configuring, testing.

Joe Crawford

You're looking at artlung.com, the personal website of Joe Crawford. That's me. I make websites professionally and as a hobby. I'm married to Leah, I'm a stepdad, I blog. I live in Moorpark, California, USA. I work in Los Angeles.
Phone: 805-857-3951.
I'm not seeking employment/
I'll be available in June 2008 for new work.
Email: joe at artlung dot com.

www.flickr.com

Recent Comments

• ArtLung Blog · Murderball on Upcoming Documentaries
• Joe Crawford on After Action Report: SIGGRAPH, Petersen Auto Museum, Chris, Canter’s
• BinaryWolf on After Action Report: SIGGRAPH, Petersen Auto Museum, Chris, Canter’s
• Joe Crawford on Upcoming Documentaries
• susan m on Upcoming Documentaries

Recent Posts

• Murderball
• Upcoming Documentaries
• Three by Missing Persons
• After Action Report: SIGGRAPH, Petersen Auto Museum, Chris, Canter’s
• Passing Strange on iTunes May 27, 2008
• Seven Nominations for Passing Strange
• Passing Strange on The View
• As a matter of fact…
• Misc, Art, Mother’s Day
• Iron Man, Redesigned

I've been doing this for a while

Select Month May 2008  (21) April 2008  (40) March 2008  (43) February 2008  (12) January 2008  (17) December 2007  (28) November 2007  (18) October 2007  (32) September 2007  (32) August 2007  (38) July 2007  (20) June 2007  (25) May 2007  (16) April 2007  (38) March 2007  (33) February 2007  (41) January 2007  (60) December 2006  (45) November 2006  (26) October 2006  (45) September 2006  (16) August 2006  (24) July 2006  (16) June 2006  (19) May 2006  (35) April 2006  (35) March 2006  (16) February 2006  (44) January 2006  (20) December 2005  (35) November 2005  (17) October 2005  (18) September 2005  (9) August 2005  (54) July 2005  (61) June 2005  (52) May 2005  (31) April 2005  (31) March 2005  (8) February 2005  (5) January 2005  (16) December 2004  (32) November 2004  (13) October 2004  (10) September 2004  (16) August 2004  (17) July 2004  (42) June 2004  (38) May 2004  (13) April 2004  (13) March 2004  (51) February 2004  (60) January 2004  (96) December 2003  (46) November 2003  (46) October 2003  (44) September 2003  (27) August 2003  (52) July 2003  (60) June 2003  (30) May 2003  (36) April 2003  (27) March 2003  (39) February 2003  (48) January 2003  (85) December 2002  (49) November 2002  (20) October 2002  (48) September 2002  (82) August 2002  (87) July 2002  (42) June 2002  (42) May 2002  (46) April 2002  (56) March 2002  (57) February 2002  (65) January 2002  (72) December 2001  (70) November 2001  (38) October 2001  (38) September 2001  (42) August 2001  (26) July 2001  (30) June 2001  (20) May 2001  (15) April 2001  (21) March 2001  (15) February 2001  (9)