Chelsea is the team that has spent the most money on transfers in recent years, and despite this, these failures are very bad. The patience of the fans has now run out. They rightfully want the team to win. Faith in Potter is shaken when there is no win for spending so much money. Money isn't everything, Chelsea show us that. Normally they should be in the league leadership race right now. However, they are in the ranking that will not even be able to go to the European cups.
Look at how long it took Arteta at Arsenal to start winning, they've been through tougher times even than Chelsea are at the moment. Arsenal were at the bottom of the table, but after that they are leading the league, I know that this will not work for everyone, but still you yourself said that money is not the most important thing. You can't just spend a lot of money buying new players and expect teams to start winning.
I think loyal fans will certainly be able to be patient with bad team results this season because after all, to have a good form of course it takes time which is not fast. Also, after the change of coach made Chelsea fail this season, they were unable to compete in the top four.
Unless, for example, after several seasons Chelsea still fail then of course not only Chelsea fans lose patience, but all aspects involved. So yes, there is still a lot of chance to get back on track with the title, if Chelsea can improve then next season could be a good season for Chelsea. Who knows!