Although you are right for most cases, I lost 1 mBTC once and was able to bruteforce it back because it was located on an address which was constructed by an algorithm.
It's comparable with HD wallets which generates all future addresses based on a seed. As long as you know the seed you are able to recover all the coins.