Recently, many artificial intelligence technology companies have been accusedin an improper manner, extracting a large amount of website content and videos as training for its artificial intelligence model, and the artificial intelligence technology startup Anthropic was accused ofClaudeBot crawler robot mines a large amount of website content, even in just 4 hours, a specific website was accessed more than 350 million times, and even the content of the iFixit website was accessed more than 100 million times in one day.
Freelancer, an Australian company that matches freelancers with job needs, announced earlier that its CEO Matt Barrie had his company's website scraped by the ClaudeBot crawler robot in just four hours. iFixit CEO Kyle Wiens further stated that the ClaudeBot crawler robot not only scraped the website content but also affected the server system.
Before this, many website operators began to add restriction descriptions in robots.txt on web pages to prevent crawler robots from accessing them. However, many crawler robots designed by artificial intelligence technology companies directly bypassed the relevant restrictions and accessed website content in large quantities, affecting the operating traffic load of many websites.
Anthropic subsequently emphasized respecting the restrictions set by each website in robots.txt, and emphasized that it would adjust the frequency of crawler robots crawling data in the future to reduce the access traffic load affecting different websites.
In addition to affecting website traffic load and even causing burdens such as increased operating costs, the improper use of copyrighted content to become the basis for training artificial intelligence models is also the reason why many websites are currently protesting against artificial intelligence technology companies. For example, OpenAI and other companies were previously accused of taking a large amount of content from News Corporation, Vox Media, Financial Times and Reddit without authorization, and subsequently reached a consensus through the signing of a cooperative licensing agreement.
In response to the massive crawling of content on its website by crawler robots, iFixit CEO Kyle Wiens expressed his willingness to negotiate a licensing agreement based on commercial cooperation with Anthropic so that both parties can reach a consensus on their needs.
