You can't have a decentralised exchange. Where would you store the flat?
The system doesn't store the fiat. Brokers in the system handle that in the thousands of ways fiat is transferred and stored currently.
The banks might decided to refuse dealing with them. The brokers are then the single point of failure.