The method is good as idea but you will lose significant amount of speed because you need to do scalar multiplication for each key, the secp256k1 sequential tricks will not work here.
So the new question will be arises, Is it worth to lose that much of speed to avoid that amount of keys?
I also faced this issue, so I played the data analyst role and analyzed the data haha (sorry for the joke).
The answer is no, we don't need to perform scalar multiplication for all the keys. This part is a bit more complex and was the most challenging, but everyone will understand.