But, if TX fee is high. People will only send his/her bitcoin if it's important

This will of course level out to an optimum.
Besides, the last satoshi will be mined in 2140. With growing number of competing miners, the price of a bitcoin just has to increase versus the declining amount of freshly mined bitcoins.