Lets work this math out: Lets assume I own $1 Million in BTC and the money supply is $1 Billion in BTC growing at 10% per year. This means that in one year (all else being equal), I have lost $100,000 in inflation paid to miners. If I want to prevent this loss I would have to invest in 0.1% of the hashing power of the network so that my earnings from mining offset my losses due to inflation.
Now if I had the option of only investing 0.05% of the hashing power to earn enough BTC to offset inflation, then I am still maintaining my position and have profited in the process by having to purchase less hashing power! The network would also have a slightly lower hash rate even thought he BTC/difficulty yield would be unchanged.
Everyone involved with bitcoin mining would like to make more bitcoin with less hash power. If all the Miners agreed at once to cut their mining in half, then the difficulty would go down by half. However, that's never going to happen, because anyone who doesn't agree would have 1 + X reward where X is the portion of miners agreeing (i.e. if 90% agree, X = 0.9)
Anyway, it's the miners, not the holders of bitcoin who get to vote on inflation.