True. The pitfall with "change" is how counter intuitive it is: if you give someone $100 to pay $50, you expect to get $50 back. With Bitcoin, you don't know you're giving more than you should, and if you've ever used a bank you don't expect to get any change. So of course, if you know what you're doing you're fine, but as a general advice sweeping all funds at once is safest.
That being said: I've sent change back to the same paper wallet in the past.