What robots.txt needs is a one line way to ban all SEO bots and a one line way to ban all AI bots/scrapers. (Those that obey robots.txt at least.)
Brad Enslen
@bradenslen
What robots.txt needs is a one line way to ban all SEO bots and a one line way to ban all AI bots/scrapers. (Those that obey robots.txt at least.)