Consolidating means creating a transaction with all the outputs you're trying to consolidate to one output. So you'll have to pay the fee anyway.