What robots.txt needs is a one line way to ban all SEO bots and a one line way to ban all AI bots/scrapers. (Those that obey robots.txt at least.)
What robots.txt needs is a one line way to ban all SEO bots and a one line way to ban all AI bots/scrapers. (Those that obey robots.txt at least.)