I been doing some research into the Long Polling issue, wouldn't be possible to just let the miner connect to the LP address directly instead of going through the proxy?
No. The LP request will return work, and if the work doesn't pass through the proxy then it can't be recorded in the database. And if it can't be recorded in the database, then work submissions against that work cannot be routed to their source pool, and the work the miner has done will have been for nothing.
Further, the LP specification implies that the X-Long-Polling header must return a host-relative URI. (Though some pools do not follow this.)