Simple solution is to calculate payout multiplier by winning chance on server end, don't calculate win chance by payout entered by players, there will be errors in your calculations. On client end currently you only let users enter payout X they want, you don't let them modify winning chance, right? Now either you reverse this (let them enter win chance and you give them payout X) or you need to be smart there, when they enter payout X and your function calculates winning chance for them, don't just stop there and once the winning chance is calculated, again calculate payout X from that winning chance within same function. This should get rid of all your troubles.
And now please don't tell me you don't know how to convert win % to payout X.
...and no this is not really a small issue, you should take it seriously!