You are forgetting about an essential part of PGP which is called
Web of Trust, you don't just trust any key that signs a binary. You have to first spend time finding the correct key that you can trust then import that and verify the signature that it created.
The problem is that most people don't actually know how to use Web of Trust since most of the people they know do not reveal their emails, and those who do, probably do not have PGP key.