No, you can use a single worker without any problems. Workers are primarily for tracking purposes in case you wanted to know how well each gpu/card is performing or which one is down or any other reason you can think of.