Yes, it matters, just like anything else matters. Both should be above age of consent and have enough life experience to know what they want in a partner, and they should be able to relate to eachother on a certain level. It's not always bad for someone who is 20 years apart, for example, to date eachother but it may be more difficult than two people of the similar age. If I had a kid who was 20, I would raise my eyebrows if a 60 year old was after him/her, but I would not mind if my kid was 40, for example, and a 60 year old wanted to date them. It's all relative and depends on the context.
However, I often think - why would a older person, like a 40 year old man, be going after a 20 year old woman when he can be with another woman of the same age who is more mature and more relatable? Obviously it's not because of the young woman's "personality", because what does she offer than an older woman cannot? Youth and looks. So the man is specifically going after the young one for her hotness. See, this is what I don't agree with. There is nothing wrong with liking someone for their youth and looks, just don't blow smoke up people's asses about "connecting" and liking their "personality".