Private keys allow people to sign messages. Very much like a person can sign a cheque, contract or credit card receipt.

Public keys allow people to verify that a message has been signed with the corresponding private key. Very much like a bank, lawyer or credit card company can verify that you have signed aforementioned document by having a sample to compare it too.

Unlike handwritten signatures cryptographic signatures can't be easily faked.

So when Alice sends 5 coins to Bob it's just Alice signing a message with her private key saying that she sends 5 coins to Bob. While Bob doesn't have Alice's private key (otherwise he could pretend to be Alice), he has Alice's public key and so can verify that she can send these coins.

For a deeper, more general look at Public-key cryptography, see here:
https://en.wikipedia.org/wiki/Public-key_cryptography

Cryptocurrencies are just one of many use cases of this sort of cryptography.