The actual transaction data would still have to reach every node in the network.

The reason we aren't simply using 1GB blocks is that the network needs to synchronize when a new block is found. If the blocks are too large this cannot be done fast enough leading to a higher stale block rate.

Now, if a miner wants to verify that a block is valid, he needs to know the actual transaction data. Otherwise he could not check whether the transactions in the block are valid (i.e. no double spends, valid signatures). Therefore, even if the blocks only contain hashes of transactions, the actual transactions beeing added to the blockchain by that block would need to be synchronized among nodes. So effectively you do not gain anything in terms of efficiency with your approach.