I seriously doubt there are that many people who verify the source code of software wallets like Electrum in detail. Their own developers do and a few hardcore coders who enjoy that sort of thing. Some bug bounty hunters and specialists probably search for vulnerabilities occasionally. And then there is the other side of the coin with black hat hackers trying to find vulnerabilities that they can exploit.
Asides from them, the most that general users do is verify the signatures of the builds before they install them and trust that everything will be ok.
I assume that the trusted open source wallet like Electrum are verified by the developers or those who have the sufficient knowledge about it. We, as an end user, don't have that much knowledge and we really don't need to do verifications about the source code.
If there was anything shady, it would have been highlighted and the code is constantly checked even by the competitors of the electrum wallet to find any vulnerability and defame them. That is one reason the open source wallet is more trusted however we should keep in mind that evil minded people are always there, testing the wallet to find and exploit the bug in the open source code.