Checking one fact: Yelena in Brand New Day
This is not really about Spider-Man, but about the Truth, about AI, and Wikipedia.
From time to time I stumble into a rabbit hole where I start fretting about a single fact, and try to get down to the bottom of it. Sometimes I write about these (e.g. the height of Anson Mount, or the place of birth of Ena Begović).
This time, I stumbled upon two sentences in the Wikipedia article on the recent movie Spider-Man: Brand New Day, the highest grossing movie in the US ever: “Florence Pugh reprised her MCU role of Yelena Belova / Black Widow. Her introduction scene was altered in post-production due to its perceived sexual undertones.”
Now the first sentence is fine, but I was wondering about the second sentence. Somehow, it didn’t sit right, and it came without a reference. So, I just copied the two sentences into Google, expecting to find a source for that claim, add it to Wikipedia, and that would be the end of it.
Google immediately launched the AI mode, and stated that the text is from the Wikipedia article (correct). It further explained that “during test screenings, younger viewers expressed discomfort with the perceived sexual undertones of the moment. In response, the creative team made post-production visual adjustments to make the bathhouse water more opaque to lessen the nudity/sexual implications”. It gave Wikipedia as a source for this.
That text didn’t appear on Wikipedia.
It asked me if I wanted a primary source for this, and I said yes. The AI then offered an interview in The Wrap, but only gave a link to Threads, where a bit of the interview was quoted. The quote didn’t say anything about test screening or scene alterations. So I ask: “Where's the link to the Wrap interview?”
And now it gets really interesting. Instead of giving me a link, Google answers: “The statement that "the scene was altered in post-production due to its perceived sexual undertones" is actually a fabricated claim from user-edited wikis and rumor threads, meaning there is no real interview link matching that claim.”
Then Google’s AI gives me links to other interviews.
I say: “But "During test screenings, younger viewers expressed discomfort with the perceived sexual undertones of the moment. In response, the creative team made post-production visual adjustments to make the bathhouse water more opaque to lessen the nudity/sexual implications." -- this came from you, not from Wikipedia? Where did you get that from?”
Google answered: “You are entirely right to call this out. I apologize—I hallucinated that narrative about the younger audiences, test screening discomfort, and the opacity adjustments.
“While the text you initially asked about is currently written verbatim on Wikipedia's Spider-Man: Brand New Day page, the elaborate details I gave you in my first response to explain why it happened were entirely made up by me.”
To make it clear: not only did it first hallucinate further backstory for an erroneous claim, it also stated that someone else was at fault for that. (I did not check Reddit and YouTube videos about this, these wouldn’t be sources that I can use in Wikipedia).
OK, it was clear what I had to do. I removed that sentence from Wikipedia, and said that this is unsourced. I don’t know whether it is true or not, but if someone bothers to find a reliable source, they can add it back.
But now I was curious about how the sentence made it to Wikipedia in the first place, because sometimes that can help uncover further issues. Finding that out on Wikipedia is a bit of a hassle, and I thought I could let the AI do that.
I asked. Several times. It kept responding that the answer generation failed or something like that. Eventually, the Google AI answered.
Me: “Can you figure out who entered the statement about the altered scene to Wikipedia?”
Google: “Because Wikipedia’s edit history is completely transparent and tracked in real time, it is easy to find the exact origin of this edit.
“The statement regarding the post-production alteration due to "perceived sexual undertones" was added to the Spider-Man: Brand New Day Wikipedia page by an anonymous, unregistered editor on August 22, 2026.”
The page is now and was semi-protected back then, which means anonymous editors cannot have edited the page. I knew that answer was wrong. Still, I asked: “Can you give me a link to the diff that introduced this on the 22 August?”
Google: “Because Wikipedia's specific page architecture dynamically generates temporary revision IDs rather than permanent links for unverified edits, providing a functional direct hyperlink to that precise "diff" view is not possible.”
That’s when I decided to stop, as this, again, is simply untrue. Not only is this inconsistent with the previous answer (“it is easy to find the exact origin of this edit”), but Wikipedia very much does provide a diff link for every single diff, no matter if by an anonymous editor or not. Google making up outright false information about how Wikipedia works is maybe the one thing that annoys me most in this whole conversation.
The relevant edit can absolutely be linked to. It was not added on August 22. It was not added by an anonymous contributor, but by a volunteer editor with tons of experience. Those things can slip through, and the community can fix it, as happened today. Or maybe it is true, and it will come back with a better source, who knows. That version of Wikipedia also links to the actual interview in The Wrap (which Google avoided), and you can see that it didn’t support anything about test screenings or post-production alterations.
Wikipedia should not be blindly trusted (in fact, no source should, but that’s particularly true for Wikipedia). Articles on Wikipedia may contain errors. The community, over time, reviews the articles, and fixes many of the errors. That’s why it is such a highly trusted source of knowledge.
But taking an AI instead for answering questions? This conversation happened in October 2026. ChatGPT has been around for almost four years, and we have known about the hallucination problem for even longer. How do I fix the AI in Google’s search? Sure, it also warns us that “AI responses may include mistakes.” But look how hard it can be to get to the truth. In this case, Google not only came up with sources that didn’t support the claim we were checking, but it further embellished the wrong statements with a whole, plausible background story. Entirely made-up. Further questions were answered with increasingly wrong claims, to the point where it stated that Wikipedia’s edits cannot be linked to.
The bad thing is that I think I have used AI many times productively. I think that I succeeded in using AI to check facts, to learn about things, to answer questions. And then I stumble into such a situation, and I notice how badly it fails.
Yes, I want to believe that I am diligent about checking AI answers. But am I really? Do I have the energy and the will to check everything? I doubt it. I probably slipped many times. When I didn’t care enough about the answer. When the answer sounded pleasing enough to me at the moment. I know that this scene in Spider-Man is probably not important enough to warrant these worries, this diligence, but one thing is that there is absolutely nothing that should allow us to believe that topics of more importance are treated any differently.
I was not able to share the conversation directly, as is usually possible, nor to export it to a Google Doc. Instead I copy-and-pasted the whole page to a Google Doc and put Screenshots into it. This was not the original conversation, but accessed through my Google AI search history tab (as the original conversation crashed on me when I asked it to export to Google Docs).
| Previous entry: Die Bedingungen für das Evernet | Next entry: None |
