It doesn't matter to other people though it kinda matter,because when you are in a relationship, having a big difference in your age can lead to more misunderstanding. Age won't matter I guess if both people in a relationship is mature enough to handle each other's immaturity.
Age really is not issue but most of the time it is the social stigma actually associated with marrying someone with a huge age interval. You will see people talking about it especially when the man is far older than the lady and is rich, they will term the lady as being a gold digger and a lot of defamatory names as well.