Well, you need at least a couple that do listen, otherwise there would be no nodes to connect to from all the other nodes (with only outbound connections). But if there were only a few of such listening nodes, it would mean we (as a network) had a huge attack vector and it wouldn't be good for decentralization either.