People generally wind up creating custom databases, or at least highly custom setups of standard database software.

You are talking about indices containing many millions of records.  It isn't as easy as just telling your SQL db to index a column.  People often turn to tools like redis for such huge datasets.