I believe that formula for calculating TX fee goes (I am not 100% sure):

normal_fee := 0.0005                            
size := X * 148 + Y * 34 + 10 +- X
fee := normal_fee / (1 - size / 500kB)

Is this is correct or I messed something?