But it is not an audit of the silicon - it is just a test of an entropy of the data that it outputs.

You can design a hardware in a way that it will output a high entropy data, though according to your pattern.
And then, when you know the secret pattern - you can predict what it would generate...

Of course its just a theory, but theorizing is what we do here.
But in any case hardware random number generators are in general unauditable - you don't know what they do and how they do it.
You need to trust the manufacturer and most of them are big corps that should not be trusted, as Google itself has already proven with their SecuredRandom class implementation in Android - I'm old enough to know that you don't usually make such a serious "screw ups" by mistake.

If you build your own hardware and then test it for entropy - such a hardware you can trust.
But usually its just easier and cheaper to trust yourself with picking a random seed-string