Solr Full Import failed (OutOfMemory)

solrA minor headache over a > 100 000 document full import on a 512 MB VPS running solr on tomcat (ubuntu).

The full import failed at ~70 000 documents.

Solved by a minor adjustment

In solrconfig.xml, reduce mergeFactor and maxBufferedDocs by 50%

    <mergeFactor>5</mergeFactor>
    <maxBufferedDocs>500</maxBufferedDocs>

As pointed out by this post, larger datasets might need further adjustments.

Here’s the error from catalina.out:

org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:424)
        at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:242)
        at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:180)
        at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:331)
        at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:389)
        at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.index.TermsHash.getPostings(TermsHash.java:225)
        at org.apache.lucene.index.TermsHashPerThread.morePostings(TermsHashPerThread.java:75)
        at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:425)
        at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:175)
        at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:244)
        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:781)
        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:764)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2634)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2606)
        at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:241)
        at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61)
        at org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:75)
        at org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:292)
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:392)
  1. Add -XX:MaxPermSize=256m to your JVM args and try again :)

  2. Awesome, new boss, I’ll try that one!

    • Kamal
    • March 19th, 2013

    @Edvin Syse

    Can u explain in detail where i need to pass the JVM args ??

  1. No trackbacks yet.