Lets work this math out: Lets assume I own $1 Million in BTC and the money supply is $1 Billion in BTC growing at 10% per year. This means that in one year (all else being equal), I have lost $100,000 in inflation paid to miners. If I want to prevent this loss I would have to invest in 0.1% of the hashing power of the network so that my earnings from mining offset my losses due to inflation.
Now if I had the option of only investing 0.05% of the hashing power to earn enough BTC to offset inflation, then I am still maintaining my position and have profited in the process by having to purchase less hashing power! The network would also have a slightly lower hash rate even thought he BTC/difficulty yield would be unchanged.
Everyone involved with bitcoin mining would like to make more bitcoin with less hash power. If all the Miners agreed at once to cut their mining in half, then the difficulty would go down by half. However, that's never going to happen, because anyone who doesn't agree would have 1 + X reward where X is the portion of miners agreeing (i.e. if 90% agree, X = 0.9)
Anyway, it's the miners, not the holders of bitcoin who get to vote on inflation.
A miner and holder could be the same person. If I owned 10% of the BTC ever mined I would mine for 50% of the reward assuming my difficulty target was half of what everyone else was doing. I would win half as much twice as often, but every time I won the inflation rate would be 50% less, but the average block rate would still be 6/hr. Even if I only had 0.1% of the hashing power, I would be better off mining for 50% reward at 50% difficulty than mining at full reward because my 'mining vote' counts 2x as much as someone mining for 2x the fees.