You can't have a decentralised exchange. Where would you store the flat?
The system doesn't store the fiat. Brokers in the system handle that in the thousands of ways fiat is transferred and stored currently.
The banks might decided to refuse dealing with them. The brokers are then the single point of failure.
If the banking sector puts a prohibition on transfers in and out of Bitcoin and other digital currencies then it makes no difference whether we have a broker/ decentralised exchange system or the present set up.
But people would still find other ways to use Bitcoin, it would just be made harder getting in and out.