The problem you are facing doesn't need the button to be removed, a better solution I can think of is to have a pop-up asking for confirmation before you can put a user on ignore only to make sure that you are doing it willingly or it has been triggered by mistake.
This is actually a good and worthy suggestion for this feature I think. We see this type of notification on different extension regularly. If this suggestion will be accepted, then It will be good to the newbie as well as the all members of the forum.