Past bites man in ass

The oddest thing happened last night, and I'm not entirely sure what I should do with it, if anything.
First, background.
I have a friend I've known for 23 years. He and I have gone back and forth between friendship and some weird mutal sexual gratification, although always on the phone, never together.  We've shared lots of things, and talked all the time. This was, however, about 15 years ago. The way I remember the conversation, I wanted to take it further and he spazzed, for lack of a better term. I was under the impression he was grossed out or upset because I wasn't (and still am not) the body type he really wants.  He left and we didn't speak for about 8 years. We reconnected on facebook (as all people do eventually) and would message each other back and forth, but it was all fairly superficial. And, I should mention, he's married to someone who looks to be awesome.
Last night he was messaging me on fb, and suddenly gave me his phone number. We ended up talking for 5 hours. It was late, we ended talking about 3 am. We talked about everything. We talked about our friendship from earlier, we talked about why he walked away, we talked about his life now and we had an explicit conversation.
He said he hadn't talked to me because his wife was threatened by my relationship with him. Which didn't and actually still doesn't make sense to me. His wife is awesome. She's gorgeous, fit, successful and the best thing that has ever happened to him. And then he dropped the second bomb of the evening. He said that he didn't go through with any actions with me because he believed he would be punished by God. Oh, and he still thinks of me all the time.
Now, I've heard several reasons for people not wanting to sleep with me, but that is a new one. Surely there is no deity would prevent me from getting a piece?
The concept, however, that he believed and still believes that I'm beautiful flattened me. When he walked away from me I took it hard. I believed I wouldn't be loved because I'm way curvier than I'm supposed to be. I'm not housebound by any means, but I'm not fit either. It's become a self fulfilling prophecy. I've not found love, I've not trusted anyone with my feelings because I didn't want to be hurt again. I've not taken a chance on anything that would be realistic. I'm as alone as I feared I would be. And now, he comes back and says, Oops? Should've told you I was tripping and it had nothing to do with you?
I thought after last night he would sober up (he wasn't totaled, but obviously drinking) and send me a message apologizing and then drop out of sight. But, instead I get a message saying he had a great conversation and that it was worth it. And insisting he wants to find me a good woman since I went ahead and told him I'm bi. (Which wasn't a surprise to him, or anyone I've told).
But is the damage done? Where do I file this tidbit of info?