DPs don't need fast lookup.
If the idea is to find quick solutions, you need to include more DPs in your searches, so I assume you need more frequent queries. And to keep both working without creating bottlenecks, you need fast access and compact databases, because not everyone has hundreds of terabytes of storage or supercomputers.
What happens is that you extrapolate everything to environments that go hand in hand with Google, Amazon, and Microsoft, and these scripts try to solve things faster with solutions within reach of the average user.
A separate thread works just fine. The only bottleneck would be having more DPs generated than the insertion/lookup can handle. However when you only have just a few DPs per second / minute, while the database can do several hundreds of thousands/s anyway, not sure where's the benefit of speeding that up another 10.000 times (e.g. having it in RAM). At some point, the RAM won't be enough anyway.
Don't need to complicate things so much.