Normally the transfer takes around 2 hours from Electrum to Coinbase.
It has been 5 hours now and the transfer from Electrum to Coinbase is still unconfirmed. It shows the transaction is pending in my coinbase so I didn't send it to the wrong wallet.
Should I be worried or just be patient?
About six hours ago, there was a huge spike in transactions on the network. 15+ MB in transactions have been added to the mempool since then. You can check the network stats
here.
Is the transaction RBF-enabled? If so, you can bump the fee up. It'll probably confirm
eventually as is, but the network is really congested right until you get above the 50 satoshi/byte level.