Yeah, we figured it out, it is some problem with 100fdx duplex.
I got really strange symptoms. It should be caused by the "stupid" switch.
So one friend took the fault device to his place, there it is plugged right away to the modem.
Since then it runs like 100% without nearly 0 rejection.

So if you use a switch you need one where you can switch manually 100fdx duplex.

Or you can try to force it with a twisted cable, this worked, but after i took the router of it didn't helped anymore even when i put the router back.