Liryk, if you are guy from Poland, check out this video:
Toxic players are problem of most MMOs, not only OTs.
RIOT were trying with "Tribunal" in which players were judged by players.
Later, before each game was displayed information about how bad behavior affects the team-mate's morale and weakens the chances of victory.
Other games use a "system of honor" to promote players who behave well instead of punishing those who are behaving badly.
Other thinks that toxic players aren't really problem, they were, they will be, and if the game is good enough people will play it anyway.
I don't have any real solution for your case, but I suggest to try with "system of honor" and check how it will work.
In short term: players can give honor points to other players if they help them etc. Honor points gives some bonuses like unique looktype of mount, some cool addons to outfit etc., but it can't affect a real gameplay (in other case it will cause a corruption - buying points for gold etc.).