It would be damaging or more like inconvenient but I don't think it would be game over since all of these popular wallets have a website where they self-host their binaries and also there is github where they share the source code and some of the also publish the binaries there as a mirror.
If this happens with iOS, we iPhone users are cooked. At least with Google (android) you can install a self compiled or trusted (verifiable) APK. But for iOS, you literally can't install an app unless you buy yourself a developer accounts ($100 per year) to sign your own app and sideload it.
It's also a lot harder to do that. 
I think this might happen sooner or later, even if they backed down for now.
