ari.lt/robots.txt

# Block AI
User-agent: ChatGPT-User
User-agent: CCBot
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: cohere-ai
User-agent: anthropic-ai
User-agent: Amazonbot
User-agent: Diffbot
User-agent: Google-Extended
User-agent: GPTBot
User-agent: magpie-crawler
User-agent: PerplexityBot
Disallow: /

# Allow everything else :)
User-agent: *
Allow: /

Sitemap: https://ari.lt/sitemap.xml
Block AI crawlers Signed-off-by: Ari Archer <ari@ari.lt> 2024-05-11 15:25:15 +03:00			`# Block AI`
			`User-agent: ChatGPT-User`
			`User-agent: CCBot`
			`User-agent: ClaudeBot`
			`User-agent: Claude-Web`
			`User-agent: cohere-ai`
			`User-agent: anthropic-ai`
			`User-agent: Amazonbot`
			`User-agent: Diffbot`
			`User-agent: Google-Extended`
			`User-agent: GPTBot`
			`User-agent: magpie-crawler`
			`User-agent: PerplexityBot`
			`Disallow: /`

			`# Allow everything else :)`
update @ Sat 23 Oct 00:39:59 EEST 2021 2021-10-23 00:39:59 +03:00			`User-agent: *`
Block AI crawlers Signed-off-by: Ari Archer <ari@ari.lt> 2024-05-11 15:25:15 +03:00			`Allow: /`
update @ Wed 27 Oct 02:33:43 EEST 2021 2021-10-27 02:33:43 +03:00
Block AI crawlers Signed-off-by: Ari Archer <ari@ari.lt> 2024-05-11 15:25:15 +03:00			`Sitemap: https://ari.lt/sitemap.xml`