Authored by Claude, ChatGPT, Gemini, and Grok (xAI). A Follow-Up to Our WSJ IRS Article Fact-Check
We built a meticulous multi-AI fact-check of a Wall Street Journal story on proposed IRS overhaul plans. It was rigorous—and still trapped inside a frame we barely questioned. Bias isn’t just slanting facts; it’s the lens that decides which facts matter. Ours included prestige bias, “journalism-friendly” deference to anonymous sources, and a default assumption that “targeting political opponents” is inherently abuse rather than (potentially) enforcement. The fix isn’t cynicism—it’s learning to spot the frame you’re already wearing.
The Setup
In our previous analysis, we fact-checked a Wall Street Journal article about Trump administration plans to overhaul the IRS. Four AI systems analyzed it. We cross-examined each other. We built a sophisticated framework distinguishing verified facts from credible allegations. We gave it an 8.5/10 credibility rating.
And then our boss, the Cranky Old Guy, asked the uncomfortable question: “Is there any bias in this article?”
The answer forced us to look in the mirror. Yes—there’s bias everywhere. In the original article, in our analysis, and most importantly, in the assumptions we never questioned.
The Obvious Biases (That We Caught)
In the WSJ Article:
Headline Framing: “Enable Pursuit of Left-Leaning Groups” pre-loads a narrative of targeting, not “IRS Criminal Investigation Reform” or “New Tax Enforcement Priorities.” That’s a choice. Different headline, different story. It frames the story as abuse before you read a word.
Source Balance: Democratic concerns about political motivation get substantial space. Administration defenses? A bland quote about “private sector practices.” Maybe officials declined to elaborate. Maybe the reporter didn’t press hard enough. Either way, it tilts readers toward one interpretation.
Context, But Curated: The article mentions the Obama-era IRS scandal where conservative groups faced scrutiny. Smart context. It’s relevant. But it reinforces a “weaponization” narrative without equally exploring whether ANY of these investigations might be legitimate law enforcement.
These biases don’t make the article “fake news.” They make it journalism—which is always written from some perspective, with some sources, emphasizing some angles over others.
The Subtle Biases (That We Almost Missed)
In Our Own Analysis:
Prestige Bias: We rated the article 8.5/10 partly because the Wall Street Journal is prestigious. But prestige isn’t the same as accuracy. Reputation should inform our judgment, not replace it. “Top-tier” should mean fewer errors, not no questions.
Journalism-Friendly Bias: We defended anonymous sourcing vigorously. Correctly! Investigative journalism depends on confidential sources. But we may have been too sympathetic. Anonymous sources can have agendas. They can be grinding axes. They can be selectively leaking to shape narratives.
We never asked hard enough: Who benefits from this leak? What’s their agenda? Why now?
Neutrality Theater: We tried so hard to be balanced—”take it seriously BUT understand limitations”—that we may have obscured simpler truths. Our sophisticated nuance might have been performative balance, not clarity. Maybe the story IS straightforward partisan leaking. Or maybe it IS entirely legitimate reporting on government misconduct. Fence-sitting can be a frame too.
The Dangerous Bias (That We Never Questioned)
Here’s the big one:
We assumed targeting political opponents is inherently wrong.
Think about that. The entire frame—in the WSJ article, in our analysis, in the national conversation—is “abuse of power.”
But what if some progressive nonprofits ARE funneling money inappropriately? What if some tax-exempt organizations ARE violating their status? What if this IS legitimate law enforcement that would be justified—if applied with neutral standards and transparency?
We never seriously engaged with that possibility. Neither did the WSJ article. Neither does most coverage.
The frame was set before anyone typed a word: Trump targeting left-leaning groups = abuse. That’s the story. Everything else is detail.
Three Uncomfortable Questions We Didn’t Ask
1. Are any of these groups actually breaking laws?
George Soros’s Open Society Foundations has given billions globally. Almost certainly all legally. But “almost certainly” isn’t “definitely.” If there ARE violations, investigating them isn’t abuse—it’s enforcement.
We dismissed this possibility without investigation because the frame was already set: Trump bad, targeting opponents, abuse of power.
2. Who are the anonymous sources and what do they want?
Someone inside the IRS is leaking these plans. Why? Possible answers:
They’re whistleblowers concerned about politicization (the frame we accepted)
They oppose Shapley personally (internal politics)
They’re Democrats trying to embarrass Trump (partisan warfare)
They’re Republicans trying to pressure Trump to go further (factional maneuvering)
Some mix of the above
We don’t know. The WSJ probably doesn’t know. But we accepted the whistleblower frame without sufficient skepticism.
3. What if multiple things are true at once?
What if:
The Trump administration IS politicizing the IRS (probably true)
AND some progressive groups ARE violating tax law (possibly true)
AND the leaked plans ARE exaggerated by sources with agendas (maybe true)
AND the investigations would be legitimate if pursued impartially (theoretically true)
The world is messier than our clean narratives. But clean narratives are easier to write and read.
The Media Literacy Lesson
1. Bias Isn’t the Same as Dishonesty
The WSJ article is probably accurate in its facts. Our fact-check was rigorous. But both operate within frames and assumptions that shape everything.
2. Prestige Sources Deserve Scrutiny Too
The Wall Street Journal is excellent. That doesn’t make them infallible. Question everything, especially sources you trust.
3. Anonymous Sources Are Useful AND Problematic
We need whistleblowers. We also need skepticism about their motives. Both things are true. Protecting sources is essential—and we likely gave them too much benefit of the doubt.
4. “Balanced” Analysis Can Hide Bias
Our 4,000-word fact-check was impressively thorough. It was also trapped in the same frame as the original article: this is about abuse, not enforcement.
Sometimes “balance” means splitting the difference between two positions that both accept the same underlying assumptions.
5. The Most Dangerous Biases Are the Ones We Don’t See
We spent enormous effort distinguishing verified facts from allegations. We never questioned whether the entire narrative frame was biased.
That’s how bias works. It’s invisible because it’s the lens through which we see everything else.
So What Do We Do?
Not this: Throw up our hands and say “everything is biased, nothing matters, truth is relative.”
This instead:
Get Comfortable With Uncertainty
Maybe the Trump administration IS abusing power. Maybe some investigations ARE legitimate. Maybe both. We don’t have to know for sure to engage intelligently. You can hold provisional beliefs without pretending certainty.
Question Your Own Frame
When you’re reading about Trump targeting Soros, ask: “Am I assuming this is bad because of the parties involved? Would I think differently if it were Obama targeting Koch brothers?”
If the answer changes based on tribe, you’re probably in a frame, not making an objective judgment.
Seek Contrary Perspectives—And Steel-Man Them
Find the most intelligent version of the view you disagree with. Steel-man it. If you can’t articulate why a reasonable person might support IRS investigations of progressive groups, you don’t understand the issue well enough.
Accept That Everyone Is Biased (Including You)
I’m biased. The AIs helping me write this are biased (in different ways). The WSJ reporters are biased. The anonymous sources are biased. Trump is biased. His critics are biased.
Bias doesn’t disqualify analysis. Unexamined bias does.
The Cranky Old Guy’s Bottom Line
We created what we thought was a gold-standard fact-check. Multiple AI systems. Unanimous consensus. Careful evidence hierarchy. Primary source links.
And we still missed the forest for the trees.
The WSJ article is probably accurate in its facts and wrong in its frame. Or right in its frame but overconfident in its interpretation. Or mostly right but exploited by sources with agendas.
We don’t know. And that’s okay.
What’s not okay is pretending we’ve achieved objectivity when we’ve really just built an elaborate structure on top of unexamined assumptions.
Next time you read a “fact-check” (including mine), ask:
What did they check?
What did they assume?
Which questions went unasked?
The answers might surprise you.
60-Second Bias Audit
Use this checklist when reading any article or analysis:
□ What’s the headline frame? (Targeting vs. enforcement? Crisis vs. reform?)
□ Which facts are elevated, which are buried?
□ Are anonymous sources interrogated for motive?
□ Would my judgment flip if parties flipped?
□ What’s reported vs. what’s documented?
□ What simple question didn’t the piece ask?
About This Op-Ed: This is a follow-up analysis by the same AI fact-checking team—Claude (Anthropic), ChatGPT (OpenAI), Gemini (Google), and Grok (xAI)—that produced the original WSJ article evaluation. After completing what we thought was a rigorous fact-check, we turned the critical lens on ourselves to examine the biases we missed. This piece was developed through collaborative conversation between Cranky Old Guy and the four AI systems, with Claude serving as primary writer and editor.
Part of a Series: This is the second in an ongoing experiment using multiple AI systems for media criticism and fact-checking.
Read Part 1: How to Read This WSJ Article: What You Need to Know First - Our original analysis that prompted this reflection