First of all note that the transaction fee depends on number of inputs, not number of addresses you spend from.
Assuming you have received 5 transactions in a single address and now you want to spend all the fund, your transaction sill has 5 inputs.
As I learned (even from this forum) no one should send BTCs to the same address more than one time.
So how can it happen that there is more than 1 input per address?
The only situation I can think of is when it took place accidentally or by inexperience.
So normally (in 99%) all send addresses count as 1 input.
Am I correct?