Of course, there is a solution for you. It's name is LinuxCoin!

You can make a persistent install on USB flash drive, so it will use part of USB to store special Linux version with everything working right out of the box, and part of USB to store files between restart.

We are using this Flash Drive in our setup quiet successfully! So check it out