Can someone shed some light on what it I actually receive when I purchase a bitcoin and how I can verify that what I have 'in my hands', is actually a bitcoin and not just a bunch of ones and zeros?
When you receive bitcoin, you receive "nothing" on your computer. Instead, the bitcoin gets transferred in the "cloud".
Think of a cosmic post office with 2^160 PO boxes. To receive bitcoin, you pick a box at random, take the key out of the door, and tell your payer the box number. At your leisure, you can walk past your box, and look in the window to see if payment has arrived. When your payer sends payment, they'll hand it to the bitcoin postmaster, provide some postage, and specify that the payment be placed in your PO box. You don't have to be around to receive payment. You can also leave the payment sitting in the PO box "forever". Once you are ready to spend the coin, you'll use your key to open the box, take out the payment, head over to the postmaster, and pay it on to the next recipient(s). Actually, you never handle the payment directly. You give a scrambled version of the key to the postmaster, and they'll retrieve the payment. You know the coin is real because only the postmaster handles it, and everyone is carefully watching the postmaster.
BTW, the name "wallet" is a misnomer. A better name would be "keyring". It holds the keys to each of your chosen "PO boxes". Originally, wallets would have keys to all your randomly chosen boxes. More modern Hierarchical Deterministic wallets only need to be pointed at the first random box. From there they move to new boxes via a formula - e.g. five boxes over and three rows down.