Affair with a close friend and neighbour

I’ve been married for more than 8 years and I can say I am happily married. We are both in our mid 30’s and my husband is supportive. He completely adores me! I also love him very much. I love being with him. Actually, we have that kind of relationship everybody “envies” and considers very balanced.
About 3 months ago I started to have an affair with my neighbor. We both have children who are friends and we have always spent a lot of time together. It all started off innocently enough and over the years we flirted and started to become emotionally attached. We discussed and shared a lot of things in the past. He is an intelligent person and a good friend of my husband. Most of the time my husband use to travel for his work.
Approx 3 months back I visited his home for some work. His wife and kids were away from home. After having some normal discussion he touched me I don’t know what happened to me. In a moment of heat, I allowed him. Since then it happened a few times more. Now I feel very unsecure. He keeps asking after every few days. I deny but he says he loves me and needs me. He says he won’t get the same feeling with his wife.
He is also married and has no intention to leave his wife. I don’t intend to leave my husband and my kid, but this situation is getting a little out of control for me. At least in what concerns my stability or sanity. Sometimes I just don’t know exactly what I should do or what I really want. Go on with this relationship or end it? I’m getting very confused and the problem is that if I was not married to my husband, I would like to be married to this man – we also have a lot in common and that’s what strikes me – how many times in your life are you supposed to meet your “soul mate”? I thought I had met mine 8 years ago when I met my husband (and I still do)? What shall I do?