Every SHA round the hash function throws away information. That is there are more inputs to the round than there are uniquely determined outputs, each round is not reversible in the formal sense. If you wanted to reverse this process you would have to guess what this information is that has been thrown away. You cant just guess any arbitrary set of bits though because when you reverse back to the start of the rounds you have to generate the magic number or you will have an impossible starting value for the function and you cant set enough of the bits in the step just before to generate the magic number because there is only so  much thrown away each round that can be guessed at in reversal. The effect of this is that you need to be guessing at the right bits throughout all of the process and the complex mathematical relationship you have to solve becomes unpractical to solve with current known techniques.

Now what of the values of the magic numbers? well they could all be zero as far as I am concerned. You still have to do the same amount of work guessing the bits that are thrown away in every round to generate a string of zeros or a string of prime numbers or what have you. If you had complete freedom to set the magic numbers and the inputs you could make a sha function that hashes to any chosen value, you just have to reverse the function and then decree that whatever value you have in the magic numbers are the magic numbers so its important that these values are fixed and for the sake of not being able to determine that one hash value, not something that is completely arbitrary and unexplained.

Prime numbers seem as good as anything because it is highly unlikely that they are the garden of Eden state of some specific hash that the developers of the algorithm wanted to be able to generate. It is possible that there is some deeper mathematical relationship between the numbers and the hash algorithm itself but that relationship would also have to be shared with the input data because it is part of the same uniform mathematical process involved in a round.