Approximately how many transactions and/or addresses are in that wallet?

It could be that your wallet simply has too many transactions and/or is requesting info for too many addresses at once... because of that the servers are throttling your connection... Once it is throttled, the "ElectrumX" server (which most Electrum Servers are running) will remember your IP and refuse your connection until a specified timeout period has passed.