By the time ECDSA actually gets broken, there might be more than a few million Bitcoins that are vulnerable still (forgotten or lost used P2PKH, just normal P2PK, etc) . A sufficiently long time for transition would be required, though arguably you're right in a sense that it does rob people of what is rightfully theirs.
There are going to be coins robbed, no doubt. However, I wouldn't take it for granted there will be millions. Sure, there are millions in P2PK, but perhaps they get spent until then; especially after the cryptographic community accepts some quantum safe alternative.
Simple and pure mathematics is what keeps the coins safe
I'm genuinely curious to which math you're referring to. Those I know, which are discrete math and cryptographic-inclusive math, are certainly not pure and simple!
