In theory the ignore list needs to be checked only for 20-40 posts or threads at a time so shouldn't be too slow.
However I have given up on trying to build and maintain a big list. I kept only a few exceptionally verbose users in it who make it painful to scroll through threads, but otherwise I just "tune out" (scroll through) the posts that I don't want to read instead of using the ignore function.
Not trying to put every shitposter on ignore makes it easier to report their posts, if you're into that kind of thing.