As a gambler you should always be willing to go against what your fan conviction tells you. Otherwise you will be guaranteed to lose money in the long run, because bookies will always have a pricing edge on you.
When I see that my team is struggling, I would be the first to lay them, because that's my edge - following one particular team closely. It is incredibly unlikely for the team that you are backing to consistently outperform market expectations.
If you are unable to objectively assess how your team is going, then it's probably best to stay out of betting on that team altogether and move onto other games/sports.