"I asked the AI" is not research

You can't reliably fact-check or research obscure topics by asking an AI. It may know less about the topic than you do.

I had two experiences this past week where I saw how misleading it can be to take AI at face value. First, I was looking for an old blog post on writer’s block I had written. I did a simple Google search, “clauswilke blog writer’s block,” and the Google AI returned the following:

The blog post I was looking for absolutely exists, and it is definitely indexed by Google, but based the specific search term I had originally entered Google couldn’t find it and the AI made up the story that there is no readily available information on the topic.

The second experience is more disconcerting, as it involves a person claiming that because DeepSeek AI couldn’t find a particular story it didn’t exist. The discussion was about whether a particular story about placement of high school students in advanced math classes had ever been published. The person wrote the following:

Just reading this post, what would be your impression? To me, it appears to say that the person got a hold of the paper, read it, and didn’t find the story that was supposed to be in there. Then, to be sure, they asked DeepSeek AI to read the paper also, and DeepSeek AI couldn’t find it either.

Neither is true.

First, as far as I can tell, the article doesn’t exist in online form anywhere. I did a thorough search, and I’m pretty good at finding things online. At a minimum, it’s not easily accessible. So unless somebody actually has the physical book they are unlikely to have read the article. And second, DeepSeek AI is unlikely to have seen the article in its training data, again because it’s not easily available. Therefore, we can reasonably assume that neither the person nor DeepSeek AI had ever seen the article and therefore had no basis on which to make any claims regarding its content. In fact, when I pressed the person on whether they had read the article they confirmed they hadn’t:

I don’t know whether this person is deliberately acting in bad faith or genuinely thinks that anything that can’t be found with a quick Google search doesn’t exist. The car example seems pretty silly to me. If I know the car’s license plate and location where it is parked I can go there and check the color. Just like if I know the book and page number I can obtain the physical copy, read it, and verify the story.

And that’s what I did. The volume containing the cited article is available on Amazon, so I bought it. I could have done inter-library loan at my university instead, but I didn’t want to wait. I put in a rush order with Amazon and the book arrived on my doorstep this afternoon. With baited breath I opened the book and read the relevant chapter.

But before we go into what I found, let’s first review the details of the claim that got us here. The context is a Substack post about how low-income students are frequently deprived of the ability to take advanced math classes, even if they are scoring well in math. In this post, we can find the following story, which admittedly is pretty insane:

The disputed claim was the last sentence in this quoted section, that this story had previously been published in a peer-reviewed journal.

And what did I find when I got my hands on a physical copy of the cited article? The story was indeed in there. While some specific details in the Substack post were not included in the peer-reviewed article and vice versa, these differences were not material to the fundamental question: Had this story previously been published in a peer-reviewed article? Yes, it had.

This entire adventure cost me $30.34¹ and two days of my time. Real research has costs, in terms of money, effort, and time. It isn’t easy. It’s not convenient. If you want to find out something nobody else knows you have to do something nobody else has done. Of course this doesn’t fully apply here. Other people have read the book. The specific article we’re talking about has 19 citations on Google Scholar. But the point stands. You have to go beyond a quick Google search or chat with your preferred AI if you actually want to figure out what’s happening in the world. While a lot is online, not everything is.

I am worried that we’re moving towards a state of affairs, in particular for the younger generations now growing up in the presence of AI, where people will not intuitively understand that there are knowable facts that are unknown to AIs. They will think that doing research is the same as asking a question of an AI that has “research” in its name. If during your formative years you regularly consult AI for all your questions, and AI seemingly always has the answer, then you never develop the research muscle that allows you to go deeper on your own.

To be clear, it is totally fine to use AI as a research tool. For example, if you ask the AI for relevant references, and you verify that any references it gives you are correct and appropriate, then that’s entirely above board. But if you leave out the verification step and just take everything the AI says at face value, you’re on shaky ground. I hope we can avoid the predicament of wide-spread over-reliance on AI for research and can continue to teach our students what actual research and fact-checking entails.

1. $16.35 for the book, $11.68 for priority shipping, and $2.31 in tax.