What's the Matter with Utilitarianism

To me utilitarianism is the least worst political philosophy, but it's not without problems. The one that I find most troubling being that Utilitarianism leaves no place for justice at a philosophical level. That's not to say that, for rule utilitarians at least, there's not space for something resembling justice in practice (in the long run we all benefit from having a just social contract). But justice is there simply because it helps make us all better off; not because it is right to put wrongs to right.

Consider the following hypothetical example:

Race A colonised the land of Race B. Much injustice was done. Eventually Race A formed a majority of the population. Years later the country starts to address its past. An act utilitarian would be guided in this process by the belief that we should undertake those acts that lead to the greatest wellbeing (so quite possibly no redress). A rule utilitarian might decide that compensation is part of a just social contract and even if the majority don't benefit from this in the short run, because in the long run we are all better off with such a social contract in place. But, while this might look like justice - it is not justice for justice's sake. And this seems troubling.