Growth isn't an axis on the chart. The y-axis is expectations not growth. By the time of maturity Bitcoin could be a lot bigger than it was at the height of expectations, but have lower expectations.
This is a true statement in general, however simple arithmetic (ok, and some reasonable assumptions about use cases) implies that the price of Bitcoin would have to be at least 100x larger if it had, say, a billion users. So, if the price of Bitcoin
today was $5 million per BTC, then I would agree with you... the Bitcoin ecosystem could keep growing while the expectations (i.e., market price) would just keep going down (because expectations went unreasonably high, beyond the equilibrium value). However, at current prices, it's just impossible for Bitcoin to grow much more without also raising the market price.
I'll add that the new coin supply (25 BTC block reward every 10 min) makes it difficult for investors to cling to unrealistic expectations over the medium term. At ~3600 BTC/day, the current price of ~$350 / BTC requires ~$1.3 million of new capital (money, energy for hashing, etc) to enter the system. A stable price means that there's sufficient demand to absorb these new coins. A price based on "unrealistic expectations" cannot last because the new coin supply requires that market participants continuously inject new capital to back their unrealistic expectation---but eventually they run out of money and the price falls.
(E.g., if the BTC price was $5 million per coin like Chris mentioned, it would require $18 billion in new capital each day. Even if every current holder refused to sell for less than $5 million the price would still drop unless those holders in aggregate can inject $18 billion each day to absorb the mining supply.) Bitcoin's inflation schedule is quite ingenious actually because the
new supply forces the price back to a level commensurate with
new demand. It's not possible for the price to remain overly-hyped for too long.