which seems to be quite ok for me, but then after some time again it goes back to HTTP error 500 and then exits after a few mis-tries.
What am I doing wrong?
How can I get it working constantly?
Is there any (*)nix based software that is dedicated for creating your own pool except for bitcoind?
With 160 machines all being CPU miners, you'll want to see what the askrates are on those boxes. bitcoind can only handle so many getwork requests and they currently all occur synchronously. For this specific workload, I would look into multiminer as it is able to take a getwork and break it down for CPU miners. Pushpool is an option but that just moves the getwork load down the chain slightly.
Bitcoind is appropriate for what you are doing, you just need to look at scalability.