Hard to explain, really. The character just really, really irritates me. I can't stand the way he looks, the way he delivers his lines, and the contrived nature of the character in general (he should have been one of the dignitaries seen in ROTJ, dammit).
He's just very annoying. Not in a Jar Jar way, mind you. Much more sublime... which might actually make him worse.