One foundational rule of the internet has been the social contract between website owners and search giants such as Google. Owners would agree to let search-engine bots crawl and index their websites ...