We just rolled out a new site search feature at DS, using Apache Solr as the search backend. We are making use of several of it’s more resource intensive features, including faceting (on a bunch of fields) and the dismax query parser.

Once we started loading large amounts of data into the index, while at the same time running queries against the index, the solr process started crashing with out of memory errors.

The errors would start with “java.lang.OutOfMemoryError: GC overhead limit exceeded“, followed with lots of “java.lang.OutOfMemoryError: Java heap space“.

I tried tweaking solrconfig.xml to better suit our dataset, and giving the java VM 2GB of heap space, but the process was still occasionally running out of memory. This didn’t seem very reasonable to me, as our data set isn’t very large, all things considered, and 2GB seems like it ought to be enough memory.

Fortunately, Joan pointed me to this page on Java VM Garbage Collection Tuning. The section on available collectors cuts to the chase, and suggests that I try the concurrent garbage collector.

I have restarted solr with “-XX:+UseConcMarkSweepGC“, and so far it seems like it might be helping.

This entry was posted on Saturday, November 10th, 2007 at 12:03 pm and is filed under DailyStrength, Administration, Development. You can follow any responses to this entry through the RSS 2.0 feed. Responses are currently closed, but you can trackback from your own site.