Obviously we share common ancestors, not to mention one male can get any number of females pregnant. You don't need 20 people to generate 10 people, actually just a couple suffices.
Okay, let's do the reverse.
Pretend that a thousand years ago there were only two people on the planet. They had two children, a boy and girl. Forty generations later, that equates to a trillion offsprings, of which there has never been a trillion people total on this planet.