Details
-
Type: Improvement
-
Status: Closed (View Workflow)
-
Priority: Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 1.7
-
Component/s: o.c.jsword.index
-
Labels:
-
Environment:
Android
Description
LuceneIndex requires too much memory for use on Android due to it's use of RAMDirectory. The main suggestion is to remove RAMDirectory.
I attach the low memory index creator AB uses for comparison. Unfortunately this code is based on the old pre-av11n code which has changed recently so I have tried to list the main changes I made to allow possible update of the latest code. The main changes I made were:
1. Move Analyzer creation inside try/catch block because it can throw errors
2. Reduce ram use by replacing RAMDirectory and setting RAMBufferSize:
Directory destination = FSDirectory.open(new File(tempPath.getCanonicalPath()));
writer = new IndexWriter(destination, analyzer, true, IndexWriter.MaxFieldLength.UNLIMITED);
writer.setRAMBufferSizeMB(MAX_RAM_BUFFER_SIZE_MB); // 1 Mb
3.book.getGlobalKeyList() took absolutely ages (possibly optimized recently) so I changed it as follows:
Key keyList = null;
if (book.getBookCategory().equals(BookCategory.BIBLE)) {
// this method is so much faster than book.getGlobalKeyList but not accurate e.g. some bibles are only NT
keyList = PassageKeyFactory.instance().getGlobalKeyList();
} else {
keyList = book.getGlobalKeyList();
}
4. Minor point but it caused me problems - when not using RAMDisk you may have temp files left on failure so need to delete temp files in the finally block:
// ensure the temp path is gone - errors can leave it there and cause further problems
CommonUtils.deleteDirectory(tempPath);
5. Allow possible reduction of field list to what is critical. I reduced it to keyField, bodyField, and strongField.
This might arouse discussion as there are various new fields being suggested but every new field increases resources required for index creation.
6. Don't report progress after each inner loop because it takes a long time to update the screen:
// report progress but not all the time for efficiency
if (subCount%50 ==0) {...
7. The old fallback when struggling with memory constraints:
// and force a garbage collect every so often
System.gc();