The real limits are the bandwidth, computing and memory resources of participating nodes. For the sake of argument suppose a 1 TB block was released into the network right now and we'll also assume there was no block size limit of any kind. Many nodes would likely not be able to successfully download this block in under 10-30 minutes, so there is a very good chance that other miners will have generated two blocks before this block makes its way to them.

What does this mean? The miner generating a 1 TB block knows this would happen. So in terms of economic self interest he will generate the largest possible block that he is still confident that other miners will accept and process. A miner who receives a block will also consider whether to build on it based on whether they think other miners will be able to download it. In other words, if I receive a large block I may decide not to mine on it, because I believe that the majority of mining power will not mine on it - because it is either too large for them to download or because their rules against large blocks reject it.