Dan Romero
@dwr.eth
Wonder if ChatGPT will be the last major model to be trained on the open web? robots.txt specifically disallowing crawling from LLMs unless getting paid for the data?
11 replies
0 recast
0 reaction
0xbyron
@byron
I'm curious what's the law around crawling sites that disregard robots.txt and post mirrors of content.
1 reply
0 recast
0 reaction
Dan Romero
@dwr.eth
We’re going to find out. The LinkedIn case said scraping is ok assuming you want to be indexed
1 reply
0 recast
0 reaction
