A new study shows that today's leading AI models often struggle to identify the most basic technical problems on a website. In a test where the models ChatGPT, Claude, Gemini and Grok were asked to analyze sites with deliberately introduced errors, they only succeeded in identifying the correct problem in 20 percent of cases.
The test covered six different scenarios that constitute critical obstacles to search engine optimization, such as websites blocked in robots.txt, pages with noindex tags, broken links in the top navigation and domains filled with spam. Despite the fact that some of these errors completely prevent websites from being indexed by search engines, the AI models tended to suggest general measures instead of pointing out the acute error. Of a total of 144 generated SEO recommendations, only seven addressed the actual root cause. Instead, measures such as more content, improved internal linking or implementation of schema markup were usually suggested.
The most surprising thing is not that AI makes mistakes, says SEO expert Magnus Bråth.
It's that it misses errors that should be trivial to detect and that prevent all other SEO until they are fixed. If a website has noindex on all pages, it doesn't matter how good the content is. You still won't be visible.
The reason for the results is described as the AI models generating answers based on how one generally works with search engine optimization, rather than performing an actual diagnosis of the unique website.
AI often answers the question of how to generally improve SEO, not what this particular website actually needs, explains Magnus Bråth.
That's a crucial difference. It provides a reasonable standard answer instead of a technical analysis.
At the same time, it is emphasized that AI still fulfills a function within the industry, primarily as a productivity tool. Experts use the technology daily to streamline their work, but the test indicates that AI is not yet ready to replace the technical expertise in prioritizing measures.
I use AI myself every day and it saves a lot of time. But it is still an assistant, not an SEO specialist. If you let AI alone control the priorities, there is an obvious risk that you optimize the wrong things while the real problems remain, concludes Magnus Bråth.