There were tons of scammers that had used stolen IDs just to get money and withdraw
Exactly, now imagine the stolen ID is yours.
I do not see anything that could go wrong way by providing ID.
Seeing from various perspectives is actually a problem in itself, this will involve trust in the service that requests your ID. Let's say you're trying to prove that you're an honest guy, but who are you dealing with?
We are never far away from the vicinity of criminal activity, all kinds of unexpected things can happen. Hiding ID doesn't mean we are perpetrators, it's just that we don't wanna be victims.