People who criticize "utilitarianism" are often actually criticizing something quite different, like coldness or expediency. John Stuart Mill himself wrote quite a lot about the distinction in his original work in 1863. The whole thing is worth reading: