I understand what you are saying. But considering how big of a market share AMD and Intel chips have, and if it was a widespread backdooring problem, I think we would have too many complaints not to understand that something is seriously wrong.
There have been bug reports regarding RDRAND, which were noticed in some minority of processors.
As you can see, there have been instances of faulty behavior, and I'm sure you can find more if you dig up the space. I think it's possible for some hardware to be backdoored. (but not all)