I think the strongest argument is this:
Nearly all IRV-advocates and voters using IRV wrongly believe that it somehow is tabulated optimally and has no spoilers. If everyone transparently understood the IRV spoiler scenarios, they would not react particularly strongly to the cases where it arises. However, IRV is almost always oversold with claims that are false. People believe that winners always have majority support or that spoilers can't happen. Thus, IRV sets up a situation to risk losing the public's trust.
A system that violates people's basic intuitions is a system people will be suspicious of.
Because IRV spoilers are both hard to explain, hard to look at the ballots and understand, and violate people's intuitions, it is a set-up for the destruction of trust in elections and in voting reform as a movement in general.
This is far worse than the effect of the spoiler itself.
The most important feature of a voting system is that it is easy for people to understand the results and trust that the system is working and thus feel trusting of the democratic process overall.