No, it wouldn't make much difference if the IRC network was compromised.

It is just used to get a list of hostnames to try to connect to, but you only need to connect to one real node in order to be able to receive the majority block chain.  It is just one of many methods used to find other nodes - the address messages broadcast on the bitcoin network are the primary means, and once you've been connected to the network the client keeps a local cache of the addresses it has seen.  The ones it sees on IRC are just added to the list.