I've been searching a bit, there is no way to set a minimum worker difficulty?
Not on the website. However, I think the server honors difficulty requests sent by stratum as a command-line option of your miner.
Through my research I read that the server actually rejects the command line setting on bfgminer
Really curious if it can or can't be done, can anyone provide answer?
It can not. There is pretty much zero reason to set a custom difficulty. The server already pretty aggressively chooses an appropriate difficulty.