When I initially wrote the search functionality for the EoE, I used the PLucene search engine, which is a pure-perl implementation of the Apache Lucene project. It worked fairly well, providing good search results and the ability to create a rich query language.

Unfortunately, it is very slow. By the time the site reached 500 indexed documents, it was taking 8 minutes for the indexer to run. Obviously this wasn’t going to scale…

So I started googling around, and eventually came across KinoSearch. It is a loose port of Lucene, with perl interfaces but lots of compiled C code. It is fast… we are now indexing 1500 much more complex documents in less than 40 seconds. Search query time appears to be at least an order of magnitude faster as well.

In addition, KinoSearch has given us highlighted excerpts, which work very, very well. Here is an example of the search engine at work:

http://www.eoearth.org/search?q=climate+change

This entry was posted on Tuesday, January 9th, 2007 at 9:45 pm and is filed under Development. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.