Well, the problem is really that we haven't finished (or even started) deploying bloom filters. It means the RAM requirements are going up over time as Bitcoin usage scales.
There's no theoretical reason Bitcoin can't run well on your phone. But as Andreas said, we haven't finished all the programming for that yet.