In as much as those "secondary observers" are actual observers of their own, what is needed for everything to be consistent is that they too, in THEIR view of reality, see the first observer as consistent with that.
Do ALL humans communicate ALL of their detailed existence to ALL humans?
No, of course not, and that is not necessary.
In order for a perception to be consistent, only those things that are *observed* from the secondary observer need to be consistent. The rest doesn't matter. If I can't observe it, ever, I don't care, it doesn't exist in a certain way.
Take the following case:
There is me, Joe, and a light bulb (in my reality).
I see the lightbulb glowing green. (direct observation).
I see Joe saying "the light is green" (my observation of Joe)
This is a consistent world view, a consistent history of observations.
Nothing stops one, however, from considering now Joe's point of view (in Joe's reality).
Joe sees the lightbulb glowing red (direct observation)
Joe sees me saying "the light is red" (Joe's observation of me)
This is (another) consistent world view, a(nother) consistent history of observations.
In the consistent history view of things, both are not in contradiction. My observing a green light, and observing Joe say "green light" is not in contradiction with Joe observing a red light and observing me say "red light". Joe, as primary observer, simply has a different consistent history than me.
What would be inconsistent, is that I see a green light, and I see Joe saying "red light". Or Joe seeing a red light, and seeing me say "green light".
Joe only exists for me to the extend that I can observe him, and that I can observe other things that might observe Joe and so on. If all these observations are consistent, then that's my consistent view of reality. Which may be entirely different for another primary observer, but to which I have no access.