Very Internet Person

Not sure I care, but Dark Visitors is a way to keep your robots.txt updated automatically against AI agents.

Not sure I care, but Dark Visitors is a way to keep your robots.txt updated automatically against AI agents. 

https://darkvisitors.com/

cloudflare has a tool to do it automatically ^-^ 

i think the real reason to do it is this problem which could end up costing u money or taking your site offline via ai ddos attack-like problems

Clarity-technologies.com COO and co-founder, Internet Explorer, VIP 🧭 Co-host of /music 🎵 

xSonos xOura

bradbarrish.me

cloudflare has a tool to do it automatically ^-^ 

i think the real reason to do it is this problem which could end up costing u money or taking your site offline via ai ddos attack-like problems https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/

i like internet (◕‿◕)♡ riotgoools.com | veryinter.net/person | nf.td/elle | alt: @elle | very internet person channel: /vip

Yeah, I read about that and you're right. It is somewhat of a compelling reason.

i don't have huge concerns about ai training on the internet but i make my websites for humans to experience so if crawlers cause them to be inaccessible, then it defeats the point ¯\_(ツ)_/¯ better to just block the bots