An ordinary wallet user does not have the requisite knowledge or ability to review the code themselves, and thus they rely on the community doing it for them and publicly flagging up any bugs, vulnerabilities, suspicious or malicious code. And code which is not open source and therefore prevents other projects, companies, developers, etc., from using that code in their products means that none of these projects, companies, developers, etc., will bother looking at the code, probing the limits of the code, building on top of the code, and so forth.

"Code accessible" or "code verifiable" simply means fewer people will be looking at the code than they would if it were open source.