The man says he isn't keen on celebrating the life of a woman who once treated him poorly Virginia Chamlee is a Politics Writer at PEOPLE ... won't be attending the funeral of his former mother ...