I don't know if it's just my office or everywhere. Being a women should not give them a privilege while the male counterparts are working harder for the same pay grade.
If this is the case on your work its not really fair. It would be best to talk to the management and have a complaint to address the issue and enlighten you with the situation. Dont you have a manager to observe your team at work? Because if yes then maybe they're tolerating it.
On the other side dont generalize women, they are working hard too. Thats just your stance because of your experience on your workplace but thats not how it is.