Matthew, 41 - who bears a haunting resemblance to his father - sank into depression after discovering his identity.
He has since been in contact with his dad in a series of letters to his California prison and Manson has replied - each time chillingly signing off with a swastika.
When 41-year-old DJ Matthew Roberts located his mother, he pressed her for more information until she finally revealed to him that she had been raped by Manson "in a drug-fueled orgy" after following him and The Family to San Francisco.
"What I'm worried about is that you think you're going to meet your birth mother or father and they're going to love you and welcome you with open arms. But he's not that kind of person."
Despite Manson's evil actions, Matthew confesses he now battles confused emotions towards his biological father.
He says: "If I did talk to Charlie on the phone, I would say, 'I truly understand what it's like to be you, more than anyone could ever imagine on so many levels'.
"He's my biological father - I can't help but have some kind of emotional connection. That's the hardest thing of all - feeling love for a monster who raped my mother.
"I don't want to love him, but I don't want to hate him either."