Yes, you do, and you are right, the coins are sent to that wallet address you have provided in the software configuration. But that is not all. Let's say you mine Ethereum. You can spend the coins you have receiving in your mining operation via the myetherwallet website wallet. And that's fine. You don't have to download the blockchain for that. Myetherwallet has to do that for you.
But what about if you mine some less known coin? There is no web wallet for that coin. You have to download the desktop wallet, and in most cases the desktop wallet would have to download the entire blockchain just to send the mining reward coins to an exchange. Unfortunately most of the less known coins have no lightweight wallets.