I think it's risky to rely on (under the radar) methods for handling Bitcoin because that can lead to legal problems and financial issues down the line. If we only focus on inflation, we might miss how tax policies impact the market. An increase in taxes could cut down the money we have for investing, which could slow down Bitcoin's adoption. This kind of economic uncertainty could create instability that hurts Bitcoin's value. So, we really need to take tax concerns just as seriously as inflation.
It's a riskier process than dealing with an exchange, that's for sure, we shouldn't rely on them. However, there are ways to stay relatively under the radar, at least for minor amounts under $500 per transaction, or more major ones through P2P exchanges. I highly doubt the average joe is going to face issues with the authorities, unless we're talking about extravagant amounts of money that are impossible to go undetected.
Taxes decrease our available income, and increase the price of goods and services which increase inflation even further in the long run. Having imposed taxes on cryptocurrencies may spark controversies and decrease their adoption.