I'm starting to wonder why, since Fiat (physical cash) is not as anonymous as one thought it would be.
Wonder why you say so(assuming you mean physical currency, which are still quite anonymous as far as I can tell.)?If by any means criminals start adopting crypto in mass (especially privacy coins), we could expect a full government crackdown on the entire industry. Which is why I think it's best for criminals to continue using Fiat for the sake of the crypto/Blockchain industry's growth worldwide. But we'll never know what will happen in the future. Thoughts? 
Well, if that happens, a transparent/good cryptocurrency will be used legitimately without much problem if the users/community do things right. No Just government will go after good cryptocurrency networks because of the bad/non-transparent ones. By the way, members of a good Cryptocurrency Network should be willing to help or cooperate with any government that has reasonable issues with them, as long as the government respects the Community/Network cheerished values like decentralization, transparency, immutablity, privacy/anonymity, etc. You can't crackdown on things that no central control especially if they are good things. The so called crackdown should be directed at those abusing the good things.