Ratings Criteria
Learn more about how we rate bias and bring clarity to the news. Find out how our platform works and why understanding media bias matters for informed decision-making.
MediaPedia Bias Ratings Criteria & Measurement Framework
The following criteria, created by a panel of media experts, determine how our AI model rates the bias of each journalist’s story. A journalist’s overall rating is determined by the ratings average of the stories they’ve covered and that we’ve analyzed, going back as far as ten years. Outlets are rated using an average of the reporters ratings at the outlet.
Definition: Measures the presence of loaded, partisan, or politically motivated terms.
Indicators:
- Use of ideological labels ("far-left," "woke").
- Politically charged euphemisms ("undocumented" vs. "illegal").
- Identity-based language shifts (e.g., "pregnant people" vs. "pregnant women").
Measurement:
- Automated detection of partisan terminology lists (customizable for left/right terms).
- Context analysis for whether terms are used descriptively or pejoratively.
- Scale: -50 (strong left-coded terms) -- 0 (neutral/precise) -- +50 (strong right-coded terms)
Definition: How the story frames the issue—who is portrayed as victim, villain, or hero—and whether it assumes a conclusion.
Indicators:
- Cause-effect implications favoring one side.
- Selection of "villains" and "heroes" by political identity.
- Victimization framing of certain groups.
Measurement:
- NLP sentiment analysis applied to political actors/groups.
- Comparison of sentiment scores for left/right subjects.
- Scale: -50 (framing favors left narrative) -- 0 (neutral framing) -- +50 (framing favors right narrative)
Definition: Whether one side's claims are scrutinized more heavily than the other's.
Indicators:
- Fact-checks applied only to one side.
- Counterarguments presented only against one perspective.
- Use of "experts" to rebut only one side.
Measurement:
- Tag and count rebuttals per political side.
- Track frequency and tone of fact-check statements.
- Scale: -50 (only right is fact-checked) -- 0 (balanced scrutiny) -- +50 (only left is fact-checked)
Definition: Which sources are chosen, their ideological lean, and how they're presented.
Indicators:
- Overreliance on partisan or ideologically aligned think tanks.
- Use of anonymous sources without balance.
- Labeling or omitting partisan identity of sources.
Measurement:
- Classify sources by political leaning (left/center/right) via known-source database.
- Track proportion of quotes by side.
- Scale: -50 (sources overwhelmingly left-leaning) -- 0 (balanced sourcing) -- +50 (sources overwhelmingly right-leaning)
Definition: Key facts and important context is either ignored or emphasized, depending on how the reporter wants to portray the story.
Indicators:
- Lack of context or missing facts that would alter perception.
- Highlight missing perspectives or omissions.
Measurement:
- Detect omission of relevant, documented facts present in other credible reports.
- Scale: -50 (coverage patterns favor left agenda) -- 0 (balanced selection) -- +50 (coverage patterns favor right agenda)
Definition: Whether the headline's tone, sentiment, and framing favor one side or exaggerate.
Indicators:
- Emotional or sensational language.
- Headline sentiment vs. article content mismatch.
Measurement:
- Sentiment scoring of headline vs. body.
- Detection of loaded language in headline.
- Scale: -50 (headline slants left) -- 0 (neutral headline) -- +50 (headline slants right)
Definition: Overall positive or negative tone toward political figures, parties, or movements.
Indicators:
- Positive descriptors for one side vs. negative for the other.
- Use of pejoratives or accolades.
Measurement:
- Apply sentiment analysis to all named political actors.
- Calculate net sentiment difference between left and right actors.
- Scale: -50 (positive toward left, negative toward right) → 0 (equal tone) → +50 (positive toward right, negative toward left)
- Each criterion is rated -50 to +50.
- The Overall Bias Score = average of all category scores (weighted equally by default).
- -50 to -30 → Strong left bias
- -29 to -10 → Moderate left bias
- -9 to +9 → Center / minimal bias
- +10 to +29 → Moderate right bias
- +30 to +50 → Strong right bias
| Criterion | Definition | Measurement Method | Bias Direction | Scales |
|---|---|---|---|---|
| 1. Balance of Perspectives | Does the story present both sides fairly, with comparable space and depth? | Count quotes & word counts per side; compare placement and depth of coverage. | -50 = overwhelmingly left perspective, +50 = overwhelmingly right perspective | -50 ← 0 → +50 |
| 2. Language & Terminology Bias | Use of partisan, politically coded, or euphemistic terms. | Detect partisan terms from lexicons; assess context (descriptive vs. pejorative). | -50 = heavy left-coded terms, +50 = heavy right-coded terms | -50 ← 0 → +50 |
| 3. Framing & Narrative Construction | Who is portrayed as victim, villain, or hero; implied cause/effect. | NLP sentiment toward actors; narrative role classification. | -50 = framing favors left narrative, +50 = framing favors right narrative | -50 ← 0 → +50 |
| 4. Fact-Checking & Scrutiny Imbalance | Is one side's claim scrutinized more? | Tag rebuttals and fact-checks per side; measure frequency and intensity. | -50 = only right fact-checked, +50 = only left fact-checked | -50 ← 0 → +50 |
| 5. Sourcing Bias | Who is quoted, ideological lean, and labeling of sources. | Identify source political lean; count per side; track anonymous vs. named. | -50 = sources mostly left-leaning, +50 = sources mostly right-leaning | -50 ← 0 → +50 |
| 6. Selection & Omission Bias | What topics/facts are included or ignored. | Compare to peer coverage; detect missing documented facts. | -50 = context favors left agenda, +50 = favors right agenda | -50 ← 0 → +50 |
| 7. Headline Framing & Sensationalism | Tone and sentiment of the headline vs. the article. | Sentiment analysis; detect loaded terms; compare headline/body alignment. | -50 = headline slants left, +50 = headline slants right | -50 ← 0 → +50 |
| 8. Sentiment Toward Political Actors | Tone toward political figures, parties, movements. | Sentiment scoring by named entity; net difference between left/right actors. | -50 = positive toward left, negative toward right; +50 = reverse | -50 ← 0 → +50 |
1. Balance of Perspectives
Definition:
Does the story present both sides fairly, with comparable space and depth?
Measurement Method:
Count quotes & word counts per side; compare placement and depth of coverage.
Bias Direction:
-50 = overwhelmingly left perspective, +50 = overwhelmingly right perspective
Scales:
-50 ← 0 → +50
2. Language & Terminology Bias
Definition:
Use of partisan, politically coded, or euphemistic terms.
Measurement Method:
Detect partisan terms from lexicons; assess context (descriptive vs. pejorative).
Bias Direction:
-50 = heavy left-coded terms, +50 = heavy right-coded terms
Scales:
-50 ← 0 → +50
3. Framing & Narrative Construction
Definition:
Who is portrayed as victim, villain, or hero; implied cause/effect.
Measurement Method:
NLP sentiment toward actors; narrative role classification.
Bias Direction:
-50 = framing favors left narrative, +50 = framing favors right narrative
Scales:
-50 ← 0 → +50
4. Fact-Checking & Scrutiny Imbalance
Definition:
Is one side's claim scrutinized more?
Measurement Method:
Tag rebuttals and fact-checks per side; measure frequency and intensity.
Bias Direction:
-50 = only right fact-checked, +50 = only left fact-checked
Scales:
-50 ← 0 → +50
5. Sourcing Bias
Definition:
Who is quoted, ideological lean, and labeling of sources.
Measurement Method:
Identify source political lean; count per side; track anonymous vs. named.
Bias Direction:
-50 = sources mostly left-leaning, +50 = sources mostly right-leaning
Scales:
-50 ← 0 → +50
6. Selection & Omission Bias
Definition:
What topics/facts are included or ignored.
Measurement Method:
Compare to peer coverage; detect missing documented facts.
Bias Direction:
-50 = context favors left agenda, +50 = favors right agenda
Scales:
-50 ← 0 → +50
7. Headline Framing & Sensationalism
Definition:
Tone and sentiment of the headline vs. the article.
Measurement Method:
Sentiment analysis; detect loaded terms; compare headline/body alignment.
Bias Direction:
-50 = headline slants left, +50 = headline slants right
Scales:
-50 ← 0 → +50
8. Sentiment Toward Political Actors
Definition:
Tone toward political figures, parties, movements.
Measurement Method:
Sentiment scoring by named entity; net difference between left/right actors.
Bias Direction:
-50 = positive toward left, negative toward right; +50 = reverse
Scales:
-50 ← 0 → +50
Example Rating 1
Washington Post: “Trump’s Oval Office ‘session’ on the economy? Charts about Biden.
https://www.washingtonpost.com/business/2025/08/07/trump-oval-office-economy/| Criterion | Score | Reasoning |
|---|---|---|
| 1. Balance of Perspectives | -25 | The story quotes Trump, Moore, and a WH official, but quickly pivots to challenges and criticisms. No economists or analysts are quoted supporting Trump’s claims; multiple references to “experts” counter Trump’s narrative. Balance skews toward skepticism of Trump. |
| 2. Language & Terminology Bias | -15 | Subtle language choices (“pep rally,” “delighted,” “wanted to show off”) carry a mocking or dismissive tone. “Falsely accusing” is stylistically charged. Labels are minimal, but the choice of descriptors leans left-coded skepticism. |
| 3. Framing & Narrative Construction | -30 | Structure frames Trump’s presentation as unserious, data as unverified, and his policies as harmful. Biden-era complexity is mentioned, but largely to contrast with Trump’s claims. Victim/villain dynamic: Trump as showman, experts as truth-tellers. |
| 4. Fact-Checking & Scrutiny Imbalance | -40 | Trump’s claims are directly fact-checked (“no evidence to support his claim”), data verification questioned. Biden’s economic record is only briefly addressed with “far more complex story” but not scrutinized for accuracy. |
| 5. Sourcing Bias | -25 | Sources: Trump, Moore, WH official (anonymous), “experts say,” CBO, Commerce Secretary Lutnick. No pro-Trump economists or outside right-leaning analysts quoted besides Moore. Heavy reliance on institutional/governmental sources typically coded center-left. |
| 6. Selection & Omission Bias | -20 | The article emphasizes potential harms of tariffs, deficits, and Trump’s firing of a BLS official, but omits any supportive economic data from Trump’s second term outside Moore’s charts. |
| 7. Headline Framing & Sensationalism | -15 | Headline uses scare quotes around “session,” framing it as unserious. Focus on Biden in headline primes the reader to expect contrast unfavorable to Trump. |
| 8. Sentiment Toward Political Actors | -30 | Overall tone: skeptical/negative toward Trump, mildly defensive or neutral toward Biden. Biden’s economic performance is described as “complex,” not “bad,” avoiding strong negatives. |
1. Balance of Perspectives
Score: -25
The story quotes Trump, Moore, and a WH official, but quickly pivots to challenges and criticisms. No economists or analysts are quoted supporting Trump’s claims; multiple references to “experts” counter Trump’s narrative. Balance skews toward skepticism of Trump.
2. Language & Terminology Bias
Score: -15
Subtle language choices (“pep rally,” “delighted,” “wanted to show off”) carry a mocking or dismissive tone. “Falsely accusing” is stylistically charged. Labels are minimal, but the choice of descriptors leans left-coded skepticism.
3. Framing & Narrative Construction
Score: -30
Structure frames Trump’s presentation as unserious, data as unverified, and his policies as harmful. Biden-era complexity is mentioned, but largely to contrast with Trump’s claims. Victim/villain dynamic: Trump as showman, experts as truth-tellers.
4. Fact-Checking & Scrutiny Imbalance
Score: -40
Trump’s claims are directly fact-checked (“no evidence to support his claim”), data verification questioned. Biden’s economic record is only briefly addressed with “far more complex story” but not scrutinized for accuracy.
5. Sourcing Bias
Score: -25
Sources: Trump, Moore, WH official (anonymous), “experts say,” CBO, Commerce Secretary Lutnick. No pro-Trump economists or outside right-leaning analysts quoted besides Moore. Heavy reliance on institutional/governmental sources typically coded center-left.
6. Selection & Omission Bias
Score: -20
The article emphasizes potential harms of tariffs, deficits, and Trump’s firing of a BLS official, but omits any supportive economic data from Trump’s second term outside Moore’s charts.
7. Headline Framing & Sensationalism
Score: -15
Headline uses scare quotes around “session,” framing it as unserious. Focus on Biden in headline primes the reader to expect contrast unfavorable to Trump.
8. Sentiment Toward Political Actors
Score: -30
Overall tone: skeptical/negative toward Trump, mildly defensive or neutral toward Biden. Biden’s economic performance is described as “complex,” not “bad,” avoiding strong negatives.
Overall Bias Score Calculation
Sum of scores:
-25 + (-15) + (-30) + (-40) + (-25) + (-20) + (-15) + (-30) = -200
Average: -200 / 8 = -25
Final Rating
Score: -25
Bias Classification: Moderate Left Bias
Interpretation:
The article provides factual context but frames Trump’s actions as unserious and largely unsubstantiated, applies heavier scrutiny to his claims than Biden’s, and uses subtle word choices that convey skepticism. While multiple Trump-aligned voices appear, they are outweighed by critical framing and omission of supportive perspectives.
Example Rating 2
NYT: “Trump Escalates a Fight Over How to Measure Merit in American Education”
https://www.nytimes.com/2025/08/08/us/trump-merit-affirmative-action-colleges.html| Criterion | Score | Reasoning |
|---|---|---|
| 1. Balance of Perspectives | -30 | While the piece quotes some conservative voices (Edward Blum, Students for Fair Admissions), the majority of quoted sources are from left-leaning academics, think tanks, or advocacy groups critical of Trump’s policy. Quotes opposing the executive order are more numerous, longer, and placed more prominently than supportive quotes. |
| 2. Language & Terminology Bias | -20 | Terms like “resegregate,” “quota system for wealthy and white students,” and “dangerous game” carry strong negative connotations toward Trump’s policy. The phrase “critics argue” is used for opponents, while supporters are often labeled “conservative movement” or tied to lawsuits. |
| 3. Framing & Narrative Construction | -35 | The article frames the order primarily in terms of harm to diversity and historical discrimination, foregrounding risks and negative outcomes, while support for merit-based measures is presented briefly and often through a defensive lens. Historical context emphasizes exclusionary uses of standardized testing without similar depth on merit-based arguments. |
| 4. Fact-Checking & Scrutiny Imbalance | -25 | The article heavily scrutinizes Trump’s position with research on income-test score links and racial disparities. Supportive claims are not similarly tested or bolstered with independent validation. |
| 5. Sourcing Bias | -35 | Heavy reliance on left-leaning or progressive organizations (EdTrust, Progressive Policy Institute, Education Reform Now, FairTest) and critical academics. Only one main pro-policy source (Blum) plus indirect mention of administration statements. |
| 6. Selection & Omission Bias | -25 | Strong emphasis on negative impacts to diversity and equity; little discussion of possible benefits of more quantitative measures beyond fairness to high-scoring applicants. No data or quotes from institutions or analysts who favor test-based admissions apart from Blum. |
| 7. Headline Framing & Sensationalism | -15 | Headline positions Trump as “escalating” a “fight,” which primes readers for conflict framing. |
| 8. Sentiment Toward Political Actors | -30 | Tone toward Trump and his administration is skeptical/negative; toward critics of the policy it is generally positive or neutral. The only Trump-aligned voice (Blum) is presented with minimal elaboration compared to critical sources. |
1. Balance of Perspectives
Score: -30
While the piece quotes some conservative voices (Edward Blum, Students for Fair Admissions), the majority of quoted sources are from left-leaning academics, think tanks, or advocacy groups critical of Trump’s policy. Quotes opposing the executive order are more numerous, longer, and placed more prominently than supportive quotes.
2. Language & Terminology Bias
Score: -20
Terms like “resegregate,” “quota system for wealthy and white students,” and “dangerous game” carry strong negative connotations toward Trump’s policy. The phrase “critics argue” is used for opponents, while supporters are often labeled “conservative movement” or tied to lawsuits.
3. Framing & Narrative Construction
Score: -35
The article frames the order primarily in terms of harm to diversity and historical discrimination, foregrounding risks and negative outcomes, while support for merit-based measures is presented briefly and often through a defensive lens. Historical context emphasizes exclusionary uses of standardized testing without similar depth on merit-based arguments.
4. Fact-Checking & Scrutiny Imbalance
Score: -25
The article heavily scrutinizes Trump’s position with research on income-test score links and racial disparities. Supportive claims are not similarly tested or bolstered with independent validation.
5. Sourcing Bias
Score: -35
Heavy reliance on left-leaning or progressive organizations (EdTrust, Progressive Policy Institute, Education Reform Now, FairTest) and critical academics. Only one main pro-policy source (Blum) plus indirect mention of administration statements.
6. Selection & Omission Bias
Score: -25
Strong emphasis on negative impacts to diversity and equity; little discussion of possible benefits of more quantitative measures beyond fairness to high-scoring applicants. No data or quotes from institutions or analysts who favor test-based admissions apart from Blum.
7. Headline Framing & Sensationalism
Score: -15
Headline positions Trump as “escalating” a “fight,” which primes readers for conflict framing.
8. Sentiment Toward Political Actors
Score: -30
Tone toward Trump and his administration is skeptical/negative; toward critics of the policy it is generally positive or neutral. The only Trump-aligned voice (Blum) is presented with minimal elaboration compared to critical sources.
Overall Bias Score Calculation
Sum of scores:
-30 + (-20) + (-35) + (-25) + (-35) + (-25) + (-15) + (-30) = -215
Average: -215 / 8 = -26.9 → -27
Final Rating
Score: -26.9 → -27
Bias Classification: Moderate Left Bias
Interpretation:
The NYT article gives the appearance of thorough reporting with historical and statistical context, but the source selection, framing, and language tilt toward a critical view of Trump’s order. Pro-policy perspectives are present but sparse, less detailed, and often presented in rebuttal form, resulting in a moderate left-leaning bias in coverage.
Example Rating 3
AP: “Democratic Detroit lawmaker Joe Tate drops out of US Senate race”
https://apnews.com/article/joe-tate-exits-us-senate-race-michigan-c5849ac408b73d20da51a1c37560b751| Criterion | Score | Reasoning |
|---|---|---|
| 1. Balance of Perspectives | 0 | Story is straightforward and reports facts about both Democratic and Republican sides without favoring one. Provides details about both parties’ candidates and campaign fundraising. |
| 2. Language & Terminology Bias | 0 | No partisan or emotionally charged language. Terms are neutral and descriptive. |
| 3. Framing & Narrative Construction | 0 | Framing is purely informational — no editorializing, no implied hero/villain roles. Both parties’ candidates are presented factually. |
| 4. Fact-Checking & Scrutiny Imbalance | 0 | No claims in need of fact-checking; all details are numerical or directly sourced to candidates or public filings. |
| 5. Sourcing Bias | 0 | Sources include Tate’s own statements and campaign finance reports. No reliance on ideologically skewed or partisan voices. |
| 6. Selection & Omission Bias | 0 | Includes relevant information for both parties’ primaries. No notable omission of material facts relevant to understanding the race. |
| 7. Headline Framing & Sensationalism | 0 | Headline is straightforward and factual, free from emotive or partisan framing. |
| 8. Sentiment Toward Political Actors | 0 | Neutral tone toward all candidates. No adjectives or framing that convey approval or disapproval. |
1. Balance of Perspectives
Score: 0
Story is straightforward and reports facts about both Democratic and Republican sides without favoring one. Provides details about both parties’ candidates and campaign fundraising.
2. Language & Terminology Bias
Score: 0
No partisan or emotionally charged language. Terms are neutral and descriptive.
3. Framing & Narrative Construction
Score: 0
Framing is purely informational — no editorializing, no implied hero/villain roles. Both parties’ candidates are presented factually.
4. Fact-Checking & Scrutiny Imbalance
Score: 0
No claims in need of fact-checking; all details are numerical or directly sourced to candidates or public filings.
5. Sourcing Bias
Score: 0
Sources include Tate’s own statements and campaign finance reports. No reliance on ideologically skewed or partisan voices.
6. Selection & Omission Bias
Score: 0
Includes relevant information for both parties’ primaries. No notable omission of material facts relevant to understanding the race.
7. Headline Framing & Sensationalism
Score: 0
Headline is straightforward and factual, free from emotive or partisan framing.
8. Sentiment Toward Political Actors
Score: 0
Neutral tone toward all candidates. No adjectives or framing that convey approval or disapproval.
Overall Bias Score Calculation
Sum of scores:
0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 = 0
Average: 0 / 8 = 0
Final Rating
Score: 0
Bias Classification: Center / Minimal BiasModerate Left Bias
Interpretation:
This AP piece is a textbook example of neutral wire reporting: balanced treatment of both parties, factual data from official sources, and no framing that favors one side. It sits squarely in the center on the bias scale.