This.

If it is possible to break the encryption on a private key, then we can break the encryption on a lot of other things which use similar or less-secure methods. Banking would collapse. Commerce, trade, and shopping would collapse. Many databases would be easily broken in to, including things like individual's health records, government identity databases, criminal databases, etc. The loss of bitcoin would be the absolute least of your worries.

Having said all that, if something like Quantum Computing or similar ever did become a credible risk, we could simply soft-fork to a new algorithm to mitigate it.