Downton Abbey’s Matthew Goode—who joined the show in season five—opted not to return for the latest film in the franchise.