The mining pools are pretty much the biggest controller of the fees and they don't want them to get TOOOO high or it would be bad for them too. IMO
There is no cartel system among the mining pools. Although 82% or more of the hashpower is located in China, the biggest pools such as BTC.com, AntPool, ViaBTC and SlushPool are not in talking terms with each other. And this hinders their ability to take joint actions which can result in fee manipulation. So I would say that right now there is no proof to say that the mining pools are controlling the fee. IMO, it is being solely decided by the volume of the Bitcoin transactions and the volatility in the market.