I have been receiving Bitcoin only in one wallet address.
So electrum uses all your UTXOs for your transaction, even if one of them is enough. This can make your transaction more expensive in fees.
Is the number of coin control inputs going to matter? is it going to affect the transaction speed or its just a matter of privacy?
The more inputs your transaction contains, the more fee you have to pay.
The solution is to go to coin control tab and select minimum number of UTXOs that can cover the sending amount.
Also note that you should consider consolidating your UTXOs when fees are low.