Last week, OpenAI released its highly anticipated search product, ChatGPT Search, to take on Google. The industry has been bracing for this moment for months, prompting Google to inject AI-generated answers into its core product earlier this year, and producing some embarrassing hallucinations in the process. That mishap led many people to believe that OpenAI’s search engine would truly be a “Google killer.”
But after using ChatGPT Search as my default search engine (you can, too, with OpenAI’s extension) for roughly a day, I quickly switched back to Google. OpenAI’s search product was impressive in some ways and offered a glimpse of what an AI-search interface could one day look like. But for now, it’s still too impractical to use as my daily driver.
ChatGPT Search was occasionally useful for surfacing real-time answers to questions which I would have otherwise had to dig through many ads and SEO-optimized articles to find. Ultimately, it presents concise answers in a nice format: You get links to the information’s sources on the right side, with headlines and a short snippet that confirms that the AI-generated text you just read is correct.
However, it often just felt impractical for everyday use.
In its current form, ChatGPT Search is unreliable for what people use Google for the most: Short, navigational queries. Queries shorter than four words represent the bulk of searches on Google; these are often just a few keywords that get you to the right webpage. They’re the kind of searches most people are barely even conscious they’re making all day, and it’s what Google tends to do very well.
I’m talking about “Celtics score,” “cotton socks,” “library hours,” “San Francisco weather,” “cafes near me,” and other queries that make Google the doorstep to the internet for billions of people.
My test run with ChatGPT Search was quite frustrating at times, and it made me conscious of just how many keyword searches I perform in a day. I couldn’t reliably find information using short queries, and for the first time in years, I actually longed for Google Search.
Don’t get me wrong, Google has declined in quality for the last decade or so, largely because it’s been flooded with ads and SEO. Still, I kept opening Google in a separate window during my test because ChatGPT Search couldn’t get me a correct answer or webpage.
Who would win: ChatGPT Search or short queries?
I typed in “Nuggets score” to check how a live NBA game between the Denver Nuggets and the Minnesota Timberwolves was going. ChatGPT told me the Nuggets were winning even though they were actually losing, and showed a Timberwolves score that was 10 points lower than it really was, according to a Google result at the same time.
Another time, I tried “earnings today,” to check the companies reporting quarterly results that could affect stock prices on Friday. ChatGPT told me that Apple and Amazon were reporting their results on Friday, even though both companies had already reported a day earlier. In other words, it hallucinated and made up information.
In another test, I typed in a tech executive’s name to find their contact information. ChatGPT showed me a summary of the person’s Facebook profile, and hallucinated a link to their LinkedIn page, which produced an error message when I clicked it.
Another time, I typed in “baggy denim jeans,” hoping to shop. ChatGPT Search described to me what baggy denim jeans were in the first place (a definition I didn’t need), and recommended I go to Amazon.com for a nice pair.
I could go on, but you get the idea. Broken links, hallucinations and random answers defined my first day using ChatGPT Search.
Maybe a ‘Google killer’ someday, but not today
This was not an insignificant launch for OpenAI. Sam Altman praised the feature for being “really good,” even though he’s known for downplaying his startup’s AI capabilities. The reason this time is different may have something to do with search being one of the biggest businesses on the internet, and OpenAI’s version could be a real threat to its biggest competitor, Google.
To be fair, Google Search is a 25-year-old product and ChatGPT Search is brand new. In a blog post, OpenAI says it plans to improve the feature based on user feedback in the coming months, and it seems more than likely this could be a significant area of investment for the startup.
To its credit, ChatGPT Search is rather good at answering long, written-out research questions. Something like, “What American professional sports league has the most diversity?” isn’t a question you could easily answer with Google, but ChatGPT Search is pretty good at scraping multiple websites and getting you a decent answer in just a couple of seconds. (Perplexity is also pretty good at these questions, and its search product has been around for well over a year.)
Compared to the traditional version of ChatGPT, which already had web access, the search feature feels like a better interface for browsing the web. There are more clear links to the sources where ChatGPT gets its information now — for news stories, ChatGPT will be tapping the media companies that it’s been striking all those licensing deals with.
The problem is that most searches on Google are not such long questions. To really replace Google, OpenAI needs to improve these more practical, short searches people are already making throughout their day.
OpenAI is not shy about the fact that ChatGPT Search struggles with short queries.
“With ChatGPT search, we’ve observed that users tend to start asking questions in more natural ways than they have in the past with other search tools,” said OpenAI spokesperson Niko Felix in a statement emailed to TechCrunch. “At the same time—web navigational queries—which tend to be short, are quite common. We plan to improve the experience for these types of queries over time.”
That said, these short keyword queries have made Google indispensable, and until OpenAI gets them right, Google is still going to be the mainstay for many people.
There are a couple reasons why OpenAI might be struggling with these short queries. The first is that ChatGPT relies on Microsoft Bing, which is widely regarded as an inferior engine compared to Google. The second reason is that large language models may not be well suited to these short prompts. LLMs typically need fully written out questions to produce effective answers. Perhaps there needs to be some re-prompting — running short queries through an LLM as a longer question — before ChatGPT Search can do such searches well.
Though OpenAI has only now released its search product, Perplexity’s own AI search tool is already serving 100 million search queries a week. Perplexity has also been touted as a “Google killer,” but it runs into the same problems with short queries.
Aravind Srinivas, the CEO of Perplexity, discussed how people use his product differently compared to Google Search at TechCrunch Disrupt earlier this week: “The median number of words in a Google query is somewhere between two and three. In Perplexity, it’s around 10 to 11 words. So clearly, more of the usage in Perplexity is people coming and directly being able to ask a question. On the other hand, at Google, you’re typing in a few key words to instantly get to a certain link.”
I think the fact that people are not using these products for web navigation presents a bigger problem than OpenAI or Perplexity are letting on. It means that ChatGPT Search and Perplexity are not replacing Google Search for the task it’s best at: web navigation.
Instead, these AI products are filling a new niche, surfacing information that gets buried in traditional search. Don’t get me wrong, that’s valuable in its own right.
OpenAI and Perplexity both claim they will work on getting better at these short queries. Until then, I don’t think either of these products can fully replace Google. If OpenAI wants to replace the doorstep to the internet, it has to create a better one.