Life wasn't always so easy for him