It also removes the characters "+" and "/" when compared to the more familiar Base64.

A key is just a long number. It can be expressed in any base you like. You could use Base10 if you wanted, or even Base2, in which case there would be loads of zeroes. We use Base58 to make the number shorter, more easily readable, and reduce the chance of mistaken characters as above.

Bech32 addresses use a custom character set of 32 characters - all case insensitive letters (26) and numbers (10), minus "1", "b", "i" and "o" (outside of the first 3 characters "bc1", known as the human readable part and the separator). This has the advantage of being impossible to make mistakes regarding upper or lower case, as well as generating smaller QR codes. You can read more about this in BIP 173.