Do you like yourself when you are with him? When you were with him? Were you always so irritable, so ugly, so annoyed? Even though you couldn't explain it.
Nothing is wrong with you or the other person's. It's just nature's way of saying you don't belong to each other. Quit this madness and move on.