Just defeats the whole purpose. What is the point in having a six-team competition? I don't think that the International Olympic Committee (IOC) will even approve this stupid plan. All of the Olympic sports have a well-defined qualification criteria, and they have geographic representation from all the continents. ICC's idea will be rejected outright (perhaps they want this, since they can use as an excuse).

Let's compare some of the sports in 2020 Tokyo Olympics:

Football: 16 participants (all except hosts need to play qualifying tournament), 12 for the women's tournament.
Baseball: 6 participants (all except hosts need to play qualifying tournament)
Basketball: 12 participants (all except hosts need to play qualifying tournament)
Field hockey: 12 participants (all except hosts need to play qualifying tournament)
Handball: 12 participants (all except hosts need to play qualifying tournament)
Rugby Sevens: 12 participants (all except hosts need to play qualifying tournament)
Softball: 6 participants (all except hosts need to play qualifying tournament)
Volleyball: 12 participants (all except hosts need to play qualifying tournament)

As you can see, only baseball and softball are the items with less than 12 participants. And in any case, all the participants need to take part in the qualifying tournament. The idea of a qualifying tournament will be unpalatable for the ICC.