The mining pools are pretty much the biggest controller of the fees and they don't want them to get TOOOO high or it would be bad for them too. IMO
The mining pools are responsible for the high fees. I still remember what happened in 2017, when some of the mining pools rejected a lot of transactions with low fee, although they were having a lot of space left with the 1 MB block size. Instead of using the full 1 MB size, some of them were using only 300KB or 400 KB. I don't have anything against prioritizing high fee transactions. But rejecting the low fee transactions and asking for a higher fee looks like plain blackmail for me.
it wasn't like that. it was done to increase their node's performance at the time because the mempool size was gigantic. so what they did was that they increased their minrelayfee so that they only receive transactions that they were including in blocks so imagine there is 100 MB transaction and only 20 MB of it is paying high fees and you can only select 1 MB of it, that means you are wasting 80 MB and that change saves that much.
the problem however was that they were lazy to change it back when the mempool size decreased so at some point they were still rejecting transactions while their blocks had empty space for them.