In my opinion, the price wont go down just because of it, but it will grow slower than before, because money is limited and people have to choose.
Who would choose shitcoins though? These are basically cash grabs and pretty much everyone knows it. What usually happens is once when people are done making money with them, they dump and move their money (which had grown at this point) to Bitcoin. So maybe it could be said that they actually help Bitcoin growth? Lol.
It's true that legit ones could take money away from Bitcoin though, but they're very few and very far between. This isn't even necessarily a bad thing, because competition could also drive growth up.
Not all of altcoins are shit, but you can say that if youre too lazy to do some researches because most of altcoins are actually shit, but not all of them.