Search Engines as Leeches on the Web
The reason I include this article is not necessarily for its content, but when it was written. Jakob Nielsen wrote this in 2006, before Google was the monster it is today. He wrote this before the invention of the LLM and it is eery how similar things sound then to now:
There's no doubt that search engines provide a valuable service to users. The issue here is what search engines do to the companies they feed on — the companies that fund the creation of original information. Search engines mainly build their business on other websites' content. The traditional analysis has been that search engines amply return the favor by directing traffic to these sites. While there's still some truth to that, the scenario is changing. [Loc]
I struggle to wrap my around what this means about the era of 2006 - 2022. Does that mean search companies were choking out the web even then? Or does it mean that this article was hyperbole and thus that the impact of LLMs will likewise be hyperbole?
The “open web” of blogs and websites has survived a lot and today LLMs feel like an existential threat. But apparently search engines themselves felt like an open threat as well. And maybe they were? Maybe the web of 2006 was more lively than the one of 2020. I was sadly too young to know then.
In some ways, the author predicted the future, though I find it hard to believe sites ever relied purely on organic search to be found. Their imagined future came to pass, which gives more credence to the rest of their article:
I predict that liberation from search engines will be one of the biggest strategic issues for websites in the coming years. The question is: How can websites devote more of their budgets to keeping customers, rather than simply advertising for new visitors? Here are some ideas, ranging from the proven (newsletters) to the speculative (mobile services): [Loc]
The best sites have been diversifying away from Google for years, using a blend of email and social media (its own can of worms) to drive traffic. Even so, the advent of LLMs is hurting publishers, though not in an evenly distributed way. What I found most interesting is this quote:
Recently, however, people have begun using search engines as answer engines to directly access what they want — often without truly engaging with the websites that provide (and pay for) the services. [Loc]
This felt especially relevant to today’s world of chat bots, where even link attribution is a thing of the past. This suggests a new way of using websites. If chat bots and Google can answer your question, what is the point of a website? I would argue it needs to change. We need to think of our web properties as something more than knowledge. People need to come for your voice, the vibe, or the experience.
We need to think harder about the design of our websites and the type of our content. We need to adapt. I’m not quite sure how yet, but I do know for my own habits, I go places that are niche and have a voice. And I go directly there by URL. A couple examples are:
- semafor.com (really incredible and unbiased world news)
- xxlmag.com (super niche rap content)