I believe the smaller is essentially dust, and if you have a bunch of those transactions when you go to send those coins you'll get hit pretty hard on transaction fees?
That's right. When you go to spend your coins you will pay up to 18x more miner fee if you are using the not recommended withdrawal limit. Also there is a fix 365 fee per withdrawal (which go to the miners too, not to us), and 365 every 100K better than 365 every 6K.