I have a MAMA's BOY in my life?

I am a 57 yr old woman. I have been speaking to a 52 yr old man over a yr and a half. I am married and hubby knows all about him. He actually came here twice and became good friends with us. We went to the beach, had him here, he does love me but thats not the issue. he is a 52 yr old virgin, and a mamas boy. When I first started talking to him, he mentioned in a sense that he wished his mama was dead, so I have always been a caring person, so I went into HELP mode and this is how we got to know one another. He's NEVER had a girlfriend, or interaction with one. He lives with mama. Mama and him were doing everything together, she worked, he stayed home. I got him to get a job at Walmart, but he quit after a month. He does do farm work for neighbors, but she basically supports him. he has a heart of gold and would make a WONDERFUL husband. Mom definitely has the hammerlocks on him. I have battled for a year to get them to have a more healthier relationship. She knows how he feels about me and told me if I was ever to divorce (my marriage was rocky) that he and her would have to have a LONG talk if he was going to relocate several states away. He dotes on her constantly. When he came here she had bought me a cute little item and in it was a note. A smile face and a sad face and I was to circle the smile if I had sex with her son OR the sad if we didn't!! Of course we didn't but what right does she have to even think of something like this, what business is it of hers?? I am totally stressed to the max. Shortly before I started to talk to this 'man', his dad had passed and was VERY depressed. I brought happiness to him just being his friend, but I have reached beyond my stress level of him being like he hardly knows me when we talk and she's right there, her going everywhere with him, getting in the car and not even asking if he can go, they hug and kiss morning and night. I just feel creepy about all this. I have tons of **** on my plate as it is, I have tried almost everyday to make him stronger. It took him 3 mos to tell her about me!!!!! He would hang up on me if she came close by. Her and I have talked several times and she likes me a lot. BUT even if I wanted him, I would have to move there, she will NOT let him go period!! I dont want him, I just dont want to see him waste his life on her and die a virgin. I thought the job would be great for him to get away from her and that house that he's in all the time. When he was here visiting she called one night and we were having dinner, and I said dont answer it, and he didnt. She needed to speak to him EVERY day, she was BS he didnt answer it. Its not like he is an only child, he has 2 sisters that live on the same property, they have a life, why cant he?? He says he is shy, shy men dont fly and drive several states to see a woman they like. he knows Im married, and he and my husband got along great, I basically had him come visit to get him the hell away from her!!!! Now I am stressed because it just irks me how she is to him, and no Im not jealous, I just feel terrible for this nice guy, he has had NO life. The other night he was at a friends, married couple, and as soon as it got dark, called mom to let her know he would be home soon, he said he did it out of 'common courtesy'. Ive tried everything in my power to help him. I cant anymore, I am totally spent with all this. I hate to sever ties totally, he is suppose to come to the beach with us this summer again (we had a great time last summer and he thoroughly enjoyed himself) but I no longer can handle sharing my friendship with the 2 of them, I feel like Im in a triangle. Am I being over dramatic, or should I just back away until he can learn to release moms apron strings, IF that day ever comes. Thank you for listening.