Yes you need to download the whole blockchain to find your transactions back.
All the transactions are kept in there and that's how the client knows how much money you have.