Well even now with IPv4 there are people with a shitload of IPs (for example botnet owners or ISPs), and that will only get worse. If bitcoin somehow relies on IPs being scarce, this is a pretty big (potential) hole in security.
Yea, the networking code is one of those things that no one has really touched since satoshi...though I don't think either port nor IPs can exploit bitcoin for sybil but I just don't know what kind of numbers would actually be required to pull it off (though it does get much, much easier with IPv6).