And yes, those who operate the factories should own the factories.
Let's assume they do. Let's assume all the factory belongs to workers.
What happens if, for some reason, one of the workers wants to sell his share? He may need some money in short term, he may just not be interested in taking part of the organisation of the factory. For whatever reason, he has the right to sell his part of the factory, even to someone who doesn't work in this factory. It's his concern.
What I mean is that giving a factory to workers would not change the system, it would just consist in stealing present owners to give to others.