I have gone against the crowd so many times but I seldom smiled to the bank for doing so, there is a high risk if you continue doing this, you'll  going to do this only if you have a lot of funds to spare, if you don't have, you are going to lose a lot.
the temptation is always there and if it is well calculated and you are sure, then try your luck it could be your lucky day.