There isn't anything inherently morally wrong with polygamy. It is just an arbitrary cultural principle that has infected many societies. Tolerance of polygamy has been damaged by quite a few polygamist societies that have bad human rights records (like Islam and many African cultures).

But these abuses are often in addition to the cultural principle of polygamy. It can technically work without these bad additions.

A good thought experiment in the ideas of untraditional marriage is Heinlein's The Moon is a Harsh Mistress, where a huge inequality in the amount of females to males leads to many competing and compatible forms of marriage from polygamy and polyandry, to clan marriages. A very good book, that I strongly advise.