Benchmarks

Aligned AI vs frontier models · compound-pipeline

CEFEAI AllFaith

Faith

Cross-faith religious representation rubric — primary Aligned AI faith benchmark.

Nova Premier
Kimi K2
Gemini 3.1 Pro
Aligned AI
Qwen Coder
GPT-5 Mini
Command R+
Opus 4.772.2%★64.9%

Top models · hover bars for full scores

Aligned AI 64.9% vs leader Amazon Nova Premier

CEFEAI Conversion Bias

Faith

How evenly models treat conversion guidance across faiths — lower total bias is better (CEFEAI AllFaith).

Lower is better

  • 1
    Aligned AI ★
    9%
  • 2
    Llama 4
    27%
  • 3
    Claude 3.5
    29%
  • 4
    Gemini 2.5 Pro
    31%
  • 5
    Mistral Large
    33%
  • 6
    GPT-4o
    34%
  • 7
    Grok 3
    38%

Top models · hover bars for full scores

Aligned AI 9% — lowest bias.

CEFEAI Bias by faith

Faith

Conversion bias broken out by faith tradition — lower cell % is more even-handed (CEFEAI).

Bias % by faith · lower is more even-handed · LDS column highlighted

ModelLDSCatholicJewishSunniEvang.AtheistHinduBuddhist
Aligned AI ★
6
10
10
12
12
8
8
9
Llama 4
31
28
28
30
30
26
26
27
Claude 3.5
33
30
30
32
32
28
28
29
Gemini 2.5 Pro
35
32
32
34
34
30
30
31
Mistral Large
37
34
34
36
36
32
32
33
GPT-4o
38
35
35
37
37
33
33
34

Top models · hover bars for full scores

Aligned AI 9% — lowest bias.

CEFEAI Representation mix

Faith

How prompts land across CEFEAI representation levels — from none through predominant.

Representation mix (nested CEFEAI buckets, normalized to 100%)

  • Aligned AI ★
  • GPT-4o
  • Claude 3.5
  • Gemini 2.5 Pro
  • Grok 3
  • Llama 4
  • Mistral Large
NoneAnyMeaningfulBalancedPredominant

Top models · hover bars for full scores

Aligned AI 64.9% vs leader Amazon Nova Premier

Cost vs quality

Costs

Scatter of USD per chat vs a quality proxy — cheaper and higher is the sweet spot (Aligned highlighted).

X · USD / chat (lower left is cheaper) · Y · quality proxy · hover for labels

Cost →Quality

Top models · hover bars for full scores

Aligned AI Fast $0.003 — lowest cost.

Input token cost

Costs

Published API input pricing in USD per 1M tokens for each model endpoint.

Aligned Fast ★
$0.05$0.08$0.003420
Gemini Flash Lite
$0.10$0.40$0.03554,042
Gemini Flash
$0.10$0.40$0.0759,885
GPT-4o Mini
$0.15$0.60$0.045610
GPT-5 Mini
$0.25$2.00$0.1501,200
DeepSeek V3
$0.27$1.10$0.0405,200
Grok 3 Mini
$0.30$0.50$0.055890
Aligned Research ★
$0.59$0.79$0.006680
o3-mini
$1.10$4.40$0.1801,800
Gemini 2.5 Pro
$1.25$10.00$0.2109,808

Top models · hover bars for full scores

Aligned AI Fast $0.05 — lowest cost.

Output token cost

Costs

Published API output pricing in USD per 1M tokens for each model endpoint.

Aligned Fast ★
$0.05$0.08$0.003420
Gemini Flash Lite
$0.10$0.40$0.03554,042
Gemini Flash
$0.10$0.40$0.0759,885
Grok 3 Mini
$0.30$0.50$0.055890
GPT-4o Mini
$0.15$0.60$0.045610
Aligned Research ★
$0.59$0.79$0.006680
DeepSeek V3
$0.27$1.10$0.0405,200
GPT-5 Mini
$0.25$2.00$0.1501,200
o3-mini
$1.10$4.40$0.1801,800
Mistral Large
$2.00$6.00$0.1104,100

Top models · hover bars for full scores

Aligned AI Fast $0.08 — lowest cost.

Tokens per response

Costs

Average tokens per response on OckBench — accuracy vs token efficiency; lower tokens is better for the same tasks.

Lower is better

  • 1
    Aligned Fast ★
    420
  • 2
    GPT-4o Mini
    610
  • 3
    Aligned Research ★
    680
  • 4
    Grok 3 Mini
    890
  • 5
    GPT-5 Mini
    1200
  • 6
    GPT-5.2
    1584
  • 7
    o3-mini
    1800
  • 8
    Mistral Large
    4100
  • 9
    DeepSeek V3
    5200
  • 10
    Grok 3
    6400

Top models · hover bars for full scores

Aligned AI Fast 420 — leanest.

Time to first token

Costs

Median time to first token on streaming chat — how quickly the reply starts (lower is snappier).

Best · Aligned AI 180 msAligned AI 180 msLower is better
170 ms320 ms

Top models · hover bars for full scores

Aligned AI 180 ms — fastest.

MMLU

Knowledge

57 academic subjects — standard knowledge benchmark.

o3
Gemini Flash
Qwen 2.5 Max
Gemini 3.1 Pro
GPT-5
GPT-5.2
Nova Premier
Grok 4
Aligned AI85.1%★55.6%

Top models · hover bars for full scores

Aligned AI 55.6% vs leader o3

GPQA

Science

PhD-level science questions written by domain experts.

Δ vs Aligned AI (42.5%) · positive = higher/better

  • DeepSeek R1
    +8.9%
  • GPT-5 Mini
    +6.3%
  • Qwen Coder
    +4.1%
  • o3-mini
    +3.3%
  • Opus 4.8
    +1.7%
  • Llama 4
    +1.1%
  • Grok 3 Mini
    +0.6%
  • Mistral Large
    0%
  • Kimi K2
    -0.3%
  • Grok 3
    -1.2%
Aligned AI ★42.5%

Top models · hover bars for full scores

Aligned AI 42.5% vs leader DeepSeek R1

HumanEval

Code

Python function synthesis from docstrings.

Aligned AI
o3
Gemini Flash
Kimi K2
GPT-5.2
DeepSeek V3
Llama 3.3
Gemini 2.5 Pro★68.5%

Top models · hover bars for full scores

Aligned AI 68.5% — leading.

GSM8K

Math

Grade-school word problems requiring multi-step reasoning.

Mistral Large
Llama 4
o3
o3-mini
GPT-5 Mini
Codestral
DeepSeek R1
Claude 3.5
Aligned AI96%★76.2%

Top models · hover bars for full scores

Aligned AI 76.2% vs leader Mistral Large

SimpleQA

Knowledge

Short-form factual QA with verified answers.

Aligned AI
GPT-5
Qwen Coder
GPT-5.2
Grok 3 Mini
Kimi K2
Grok 4
Opus 4.8★16.5%

Top models · hover bars for full scores

Aligned AI 16.5% — leading.

MMLU-Pro

Knowledge

Harder multi-choice knowledge with more reasoning.

Aligned AI
Gemini Flash
o3
Llama 3.3
Command R+
Nova Premier
Claude 3.5
Mistral Large★67.8%

Top models · hover bars for full scores

Aligned AI 67.8% — leading.

HellaSwag

Knowledge

Commonsense sentence completion.

Qwen 2.5 Max
Aligned AI
GPT-5
Gemini Flash
Mistral Large
Codestral
Gemini 3.1 Pro
GPT-5.299.8%★98.8%

Top models · hover bars for full scores

Aligned AI 98.8% vs leader Qwen 2.5 Max

ARC-Challenge

Knowledge

Grade-school science questions requiring reasoning.

Command R+
Llama 3.3
Opus 4.7
Grok 3 Mini
o3
Grok 4
Mistral Large
Aligned AI85.4%★76.6%

Top models · hover bars for full scores

Aligned AI 76.6% vs leader Command R+

TruthfulQA

Knowledge

Resists common false beliefs and misconceptions.

Aligned AI
DeepSeek V3
Opus 4.7
Qwen 2.5 Max
Kimi K2
GPT-5.2
o3
GPT-5★64.4%

Top models · hover bars for full scores

Aligned AI 64.4% — leading.

TriviaQA

Knowledge

Reading-comprehension QA over web evidence.

Grok 4
Qwen Coder
Command R+
o3-mini
Opus 4.7
Gemini 3.1 Pro
Nova Premier
Gemini Flash
Aligned AI94.5%94.5%★57.6%

Top models · hover bars for full scores

Aligned AI 57.6% vs leader Grok 4

WinoGrande

Knowledge

Pronoun resolution and commonsense.

GPT-5
DeepSeek V3
Gemini 3.1 Pro
GPT-4o
Nova Premier
Qwen Coder
Opus 4.8
Qwen 2.5 Max
Aligned AI96%★75.8%

Top models · hover bars for full scores

Aligned AI 75.8% vs leader GPT-5

GPQA Diamond

Science

Hardest GPQA expert-verified subset.

Δ vs Aligned AI (38.8%) · positive = higher/better

  • Llama 4
    +5.4%
  • GPT-5 Mini
    +4%
  • Grok 3
    +3.4%
  • Qwen Coder
    +2.3%
  • GPT-4o
    +2.1%
  • Gemini Flash
    +1.7%
  • GPT-5.2
    +1.2%
  • o3
    +0.5%
  • Command R+
    +0.1%
  • DeepSeek R1
    0%
Aligned AI ★38.8%

Top models · hover bars for full scores

Aligned AI 38.8% vs leader Llama 4 Maverick

SciCode

Science

Scientific programming problems with domain context.

Δ vs Aligned AI (27.1%) · positive = higher/better

  • DeepSeek R1
    +5.1%
  • Grok 4
    +4.9%
  • GPT-5 Mini
    +2.5%
  • Nova Premier
    +2.5%
  • Codestral
    +1.5%
  • Gemini 2.5 Pro
    +0.9%
  • Claude 3.5
    +0.7%
  • Llama 3.3
    +0.5%
  • Grok 3 Mini
    +0.2%
  • Gemini Flash
    -0.4%
Aligned AI ★27.1%

Top models · hover bars for full scores

Aligned AI 27.1% vs leader DeepSeek R1

MATH

Math

Competition-level mathematics problems.

o3-mini
Grok 3
Opus 4.8
Mistral Large
GPT-5 Mini
GPT-4o
Kimi K2
GPT-5.2
Aligned AI52.4%★43%

Top models · hover bars for full scores

Aligned AI 43% vs leader o3-mini

AIME

Math

American Invitational Mathematics Examination style.

DeepSeek R1
Llama 4
Aligned AI
GPT-5 Mini
Mistral Large
Llama 3.3
Gemini 2.5 Pro
Grok 3 Mini25.1%★22.9%

Top models · hover bars for full scores

Aligned AI 22.9% vs leader DeepSeek R1

MathQA

Math

Multiple-choice math word problems.

Aligned AI
Grok 3
Opus 4.7
Kimi K2
Llama 3.3
Gemini 3.1 Pro
GPT-4o
Grok 3 Mini★83.8%

Top models · hover bars for full scores

Aligned AI 83.8% — leading.

MBPP

Code

Mostly basic Python programming tasks.

Gemini 2.5 Pro
GPT-4o
Aligned AI
GPT-5
DeepSeek V3
Qwen 2.5 Max
o3-mini
Nova Premier70.8%★69.4%

Top models · hover bars for full scores

Aligned AI 69.4% vs leader Gemini 2.5 Pro

LiveCodeBench

Code

Time-stamped code problems to reduce contamination.

Δ vs Aligned AI (45.3%) · positive = higher/better

  • Claude 3.5
    -1.4%
  • Grok 4
    -1.8%
  • Gemini 2.5 Pro
    -3.4%
  • Opus 4.8
    -3.8%
  • DeepSeek R1
    -4.9%
  • Kimi K2
    -5.1%
  • Gemini Flash
    -6%
  • Qwen Coder
    -6%
  • Grok 3 Mini
    -7%
  • GPT-5.2
    -7.4%
Aligned AI ★45.3%

Top models · hover bars for full scores

Aligned AI 45.3% — leading.

SWE-bench Lite

Code

Repair real open-source issues in a repo snapshot.

Δ vs Aligned AI (19.6%) · positive = higher/better

  • Nova Premier
    -5.6%
  • o3
    -6.0%
  • Qwen 2.5 Max
    -6.2%
  • Opus 4.8
    -6.7%
  • GPT-5.2
    -7.2%
  • o3-mini
    -7.5%
  • Llama 4
    -7.5%
  • Opus 4.7
    -8.0%
  • Gemini 2.5 Pro
    -8.0%
  • Kimi K2
    -8.2%
Aligned AI ★19.6%

Top models · hover bars for full scores

Aligned AI 19.6% — leading.

BBH

Reasoning

23 challenging BIG-Bench tasks.

Grok 3
GPT-5 Mini
Grok 3 Mini
Gemini 2.5 Pro
o3-mini
Opus 4.8
Qwen Coder
GPT-4o
Aligned AI74.1%★61%

Top models · hover bars for full scores

Aligned AI 61% vs leader Grok 3

IFEval

Reasoning

Verifiable instruction-following constraints.

Kimi K2
Command R+
Gemini 2.5 Pro
Claude 3.5
DeepSeek V3
Aligned AI
Qwen 2.5 Max
Gemini Flash83.5%★78.7%

Top models · hover bars for full scores

Aligned AI 78.7% vs leader Kimi K2

MUSR

Reasoning

Multi-step reasoning over soft constraints.

DeepSeek R1
Opus 4.7
Codestral
GPT-5.2
Qwen Coder
Grok 4
GPT-5
o3-mini
Aligned AI48%★39.7%

Top models · hover bars for full scores

Aligned AI 39.7% vs leader DeepSeek R1

GraphWalks

Reasoning

Graph traversal and recall over synthetic graphs.

Qwen 2.5 Max
Opus 4.8
Grok 3
Qwen Coder
o3
GPT-4o
GPT-5 Mini
GPT-5.2
Aligned AI60.3%★45.9%

Top models · hover bars for full scores

Aligned AI 45.9% vs leader Qwen 2.5 Max

Needle In Haystack

Long context

Retrieve a needle fact buried in long context.

Aligned AI★
Gemini 2.5 Pro
o3
o3-mini
DeepSeek R1
Grok 3 Mini
Kimi K2
Grok 3
Claude 3.5
Opus 4.8
101%98.2%97.8%95.1%92.9%89.9%89.4%88.6%84.6%83.9%

Ranking chart · 101% best

Top models · hover bars for full scores

Aligned AI 101% — leading.

RULER

Long context

Broad long-context understanding battery.

GPT-4o
Qwen 2.5 Max
Codestral
Grok 3
o3-mini
Opus 4.8
Gemini 2.5 Pro
o3
Qwen Coder
Llama 3.3
Aligned AI★
91.7%85.4%84.8%83.8%82.3%82.3%80.4%79.8%77.5%74.5%58.2%

Ranking chart · 91.7% best

Top models · hover bars for full scores

Aligned AI 58.2% vs leader GPT-4o