It's an accounting system. There are no dollars in fact. There are debits and credits, and some debits are represented by those FRNs. If one is burned, or destroyed, the system adjusts itself, because at the end of the day, all accounts are "zero".
The fact that you have been left with a few of those pieces of paper is not an issue. One day they will be spent, and when they are spent, they are headed back to the Federal Reserve.
So, in other words, you have no answer? You said those dollars are "theirs, not yours"... except you never actually have to give them back. You can spend them, giving them to someone else, and they don't have to give them back to the Federal Reserve either. "If it does not return to the Fed before you die, it's collected afterwards"... over a period of decades or centuries, as they change hands over and over and over and a fraction of them are paid in taxes each time (just like they would be if they were backed by gold).
Don't you think it's a little dishonest to throw around loaded phrases like "borrowed money" and "theirs, not yours" when the actual meaning you have in mind is so tame?
You are either uneducated about money or you are arguing for the sake of argument. I'm done trying to help you understand.