This is something that came up in the Lucene users list, and might be problem on massively scaled (600 million rows +) data. Which is that many operating systems have a limit on the directory size limit or will run out of memory on a Lucene search. We should investigate how the index will scale and figure out when/if we need to start sharding indexes. The result of this investigation should 1. be an upper bound on a single index. 2. the system resources that are required to do searches on very large applications.