No, society has been unfair to people with disabilities. Including both men and women.
Indeed, people living with disabilities are not treated fairly in most societies. In some places, they are seen as second-class citizens.
In my country, people living with disabilities would have a robust sexual life if they are financially stable. Our society places more emphasis on money than any other thing. These special people have to work hard to overcome societal stigma before they can attract and get emotional satisfaction.
I have had a relationship with someone that is challenged physically and I find them very caring, trustworthy, and loving. Due to the fear and uncertainty, society has put in them, they are always scared of losing their partners. But, I find them very jealous and timid because they always feel you are discriminating against them because of their challenge.