Me too, the problem that a lot of people have is that they lack the skills and knowledge to verify the open source codes of these wallet softwares themselves. However, this is the reason why i always recommend that people go for well reviewed open source softwares, i.e. Electrum, so that even if you don't have the knowledge to verify the code yourself, you know that a large part of the community constantly does it. I know it is safer to do it yourself, but this way is surely better than using closed source softwares.
I seriously doubt there are that many people who verify the source code of software wallets like Electrum in detail. Their own developers do and a few hardcore coders who enjoy that sort of thing. Some bug bounty hunters and specialists probably search for vulnerabilities occasionally. And then there is the other side of the coin with black hat hackers trying to find vulnerabilities that they can exploit.
Asides from them, the most that general users do is verify the signatures of the builds before they install them and trust that everything will be ok.