In bitcoin, those authorities are back, now they are mathematics and network. People have no problem to trust the authoritative of mathematics (law of nature), but they might not trust the network because it is prone to human manipulation. So it is essential the protocol is followed by all the network participants and even its users
You forget that mathematics is applied by human beings altogether. While it can be faultless and perfect (unless proven otherwise, of course), humans are not. I would most certainly agree with you if you could somehow remove the human element from this sequence...
The weakest link in the chain