The "Last Week Tonight" host joked that the part of him that he likes "more" didn't want to see the "right thing" go down.