OP,

That is partially correct, but the difficulty is adjusted based on the total hash rate every 10-12 days or whatever it is (it's actually per number of blocks, I'm sure someone remembers it, I don't).  Presumably you will get more people participating in mining the easier the difficulty.

I think we're getting somewhat close to the point where we stop seeing new miners and thus the difficulty will stop increasing, and your assumptions are correct.