How is the connection strength and the connectivity measured? Are there stats that one could look at before deciding which route to take to make sure that the payment goes through?
I'm not sure.
Apparently it can happen that a payment doesn't go through. Is there a reason it's made this way
LN is a bit like a
spoke-hub: instead of making a connection from anywhere to anywhere, everyone makes only a few connections.
and not like regular Bitcoin transactions, where everyone can send to every other person on the world?
That exists already

You can of course just use a normal on-chain Bitcoin transaction.
The problem with this is scalability: currently every Bitcoin node sees all transactions. That works fine for a few transactions per second. LN will be able to handle much larger transaction volumes, without storing all transaction data on the blockchain
for eternity.