I'm probably not 100% understanding how this works, but is it true that if the rate of people investing outpaces the amount gambled, the returns will dilute? If so, would it make sense to have some sort of on-the-fly tracking/scaling of a total allowable house investment base on a windowed rate of house gain? Something to keep a fix on the estimated rate of return. Just like there's a max-bet determined by the pool size, this would be keeping the other side of the coin intact.
The market sorts that itself. If ROI falls then at some point it'll drop below some people's threshold and they'll pull out their cash. Maximising capital is in dooglus' best interest - as it also maximises possible revenue. What you propose also potentially loses gambling customers - as when they finish a session of play and want to sling their BR into investment mode until the next day they sometimes won't be able to. And that then encourages them to withdraw - adding a delay/barrier against them resuming play later.
dooglus should just focus on getting as many people gambling as possible - and let investment sort itself out. If capital becomes too excessive then the easiest way to handle it is increase his own cut - if he's getting far more capital than he needs then clearly he isn't taking enough.