So there is no way to ensure my spv wallet can connect? Seems kinda sub-optimal from a privacy perspective
You can either increase your connection limit so that you have more available connections than people are likely to connect to you, or you can use an external firewall software to manage this for you.
If you use an external firewall, you could have it set so that you allow 21 connections (30 connections total - 8 outgoing connections - 1 for your wallet) on the default port from external sources, and then have another firewall rule on a different port which forwards traffic to the Bitcoin node port (8333) internally (so as to bypass the earlier firewall rule). You then connect to that other port with your SPV wallet.