EDIT: When I first opened BGL Core a while ago, I played with it like a noob and created a few wallets besides "Main", can I delete those wallets?
Take a precaution doing this, if you mistakenly delete the wrong file and didn't have a proper backup, you might lose your coins.
Depending on your OS, go to the default data directory
#Windows
%APPDATA%\Bitcoin
#Linux
~/.bitcoin/
#Mac
~/Library/Application Support/Bitcoin/
Open the folder named "wallets", then delete the folder(wallet) you want to delete.
The file you store your private key should be encrypted. The same goes for your wallet.dat file. If hackers can get access to your private key, your coins will be stolen.
A very bad practice is keep your private key in Notepad file, unencrypted. You can use Keypass, read more in
[GUIDE] How to Create a Strong/Secure PasswordYea, storing a bare private key unencrypted is bad security/backup practice.
@gcf0z If your intention is to backup your wallet, I think you better to store the encrypted wallet.dat file. And also use a strong passphrase.