@Xikini
If you're going to ignore someone, isn't finding them ultra-annoying just the kind of most excellent justification/reason to do so?
Like that's the whole reason why we ignore people - to not let annoying people get on our nerves and to avoid engaging aggressively towards someone that doesn't necessarily deserve it (yet).
Of course it can be (and is) abused sometimes/by some people as well, and the only way you can know which side of that you're on is by comparing the ignored party's action with the punishment - namely in this case whatever the other part has done to annoy you, and the degree to which you've decided to ignore this person because of it, and then experiment a little (be a little open to change once in a while) and you'll learn as time goes by.
Side-question: have you taken an MBTI test, and did you score as IxFP?

Just wondering since your question/reasoning seems stereotypical IxFP.