You don't lose it but it goes to another address which means you don't accumulate as much in one address. If you were below the payout threshold and abandon the address then it will never get paid anything. I highly recommend using one address as much as possible.