A-Feelings are just feelings and they're not good or bad, right or wrong. You can feel attracted to someone and not have to have a relationship with them. The same way I can really like chocolate cake, but not have it for dinner every night.
Behaviors do not HAVE to follow feelings, although we usually treat them as one and the same.
If there's a good reason not to be with him, then don't. If he's just different from your normal "type", why not give it a chance? The world is full of unlikely love stories.
PS-Do you have a Q that needs an A? Let me know at firstname.lastname@example.org
PPS-Sign up for my email list and I’ll share some great relationship tips with you!