Abusive ex boyfriend, who's truly to blame?

It started five months into the relationship. Everything before that five month mark was 'great' between him and I. I had been living in a broken home (like he was/is), with low self-esteem, and no one to rely on but him. We were both very loving and doting during that five month period; however, that five month marker came and I left my broken home to live with my father. A domino effect came with learning self-esteem, happiness and a whole bunch of other good stuff. The relationship I had with this boy, I started to realize was negative. All he ever did was grab me sexually in public, 'kiss and tell' to his buddies, and beg for sex. All of which I was compliant with at the time, because I was so depressed. But now that things were changing for me, I wanted him to change with me. So I asked him to start treating me a little better and to respect me in public. It didn't work; after many broken promises and guilt-trips (for not doing what he asked), I sought out a better match for me with a friend of a friend. It never progressed to anything further with him than a phone call and serious consideration. My boyfriend knew about this and suppressed whatever he was feeling, and hardly protested my leaving. I concluded, that I was going to work on my current boyfriend and ditch the other kid. I apologized to him and made it up to him with love letters, and lots of sex. Everything was okay for a few weeks. But then the relationship spiraled into multiple affairs on his part, growing verbal and occasional physical abuse only to be broken by short spells of honey-moon phases for two years. And every time he did something, he'd always refer back to that one thing I did, dating back to a year and a half ago. So I'm wondering, is what I did the cause for his abuse?