DinoAI - the #1 agent harness on Snowflake's data-eng-bench
DinoAI passed 76.7% of Snowflake's 103 data engineering tasks in one full run. No harness has published a higher pass rate, and Snowflake's own Cortex Code stands at 73.8%. Model cost per task starts at $0.69.


In August 2026, Snowflake and Bespoke Labs released data-eng-bench. It is an open benchmark of 103 dbt™ tasks on a retail data warehouse. Snowflake published results for its own agent, Cortex Code, and for Claude Code and Codex.
In October we ran DinoAI, the AI agent in Paradime, on the Snowflake edition of the same 103 tasks. We ran it with four models: GPT-6.1, GPT-5.6, Claude Opus 5 and Claude Opus 5.5.
This report gives the results and the cost of each run. It also shows what the data says about choosing a model for data engineering work. We list the limits of our method near the end, so you can judge the numbers yourself.
- #1
- Pass rate on data-eng-bench
- Among all published harness results
- 77.7%
- Top pass rate
- DinoAI with GPT-5.6, 80 of 103 tasks
- $0.69
- Lowest model cost per task
- DinoAI with GPT-6.1, at a 76.7% pass rate
- 5.8×
- Cost gap between models
- $0.69 to $3.97, same harness and tasks
Key findings
- DinoAI is the #1 harness on data-eng-bench. It has the highest published Pass@1 (76.7% in one full run) and the highest published Pass^3 (66.0%). It leads on five of eight head-to-head measures and trails on one.
- DinoAI passed 75 to 80 of 103 tasks, depending on the model. Three of the four results are above every other published result we found.
- On GPT-5.6, DinoAI passed 77.7% of tasks and Cortex Code passed 64.1%. On Claude Opus 5 the two agents tied: 74.8% and 73.8%, a gap of one task.
- The model moved the score by 5 tasks and the cost by 5.8× when the harness and the tasks stayed the same.
- Price per token does not predict cost per task. Output is under 2% of the tokens and as much as 39% of the bill.
- The newest models scored about the same and cost far less. Cost fell 56% from GPT-5.6 to GPT-6.1, and 76% from Claude Opus 5 to Opus 5.5.
- Most failures were close. Of 90 failed trials on multi-assertion graders, 65 missed by one or two assertions.
What data-eng-bench measures
Each task gives an agent a ticket in plain English and a live dbt™ project. The agent reads the project, queries the warehouse, writes or repairs models, and runs dbt™. A hidden grader then checks the tables the agent built.
The warehouse holds one synthetic retail company. It has 579 source tables in 19 schemas, with about 8,000 columns. The business rules come from finance, inventory, marketing and customer analytics. Examples are ledger reconciliation, FIFO and LIFO costing, multi-touch attribution, churn and lifetime value.
The benchmark's repository sorts the 103 tasks into four categories.
Category | Tasks | What the agent does |
|---|---|---|
Analytics | 65 | Builds marts for retention cohorts, RFM segments, lifetime value, fraud detection, attribution and campaign ROI |
Development and bug fixes | 16 | Finds and repairs broken or incomplete dbt™ models |
Data engineering | 13 | Builds incremental models and multi-database pipelines |
Dimensional modeling and snapshots | 9 | Builds dimension and fact tables, and history for slowly changing dimensions |
Snowflake also splits the tasks by type of work. 84 tasks ask the agent to build something new. 19 ask it to fix a model that returns wrong numbers. The difficulty labels are 3 easy, 47 medium, 45 hard and 8 very hard.
How a task is graded
Each task has a hidden suite of assertions written in pytest. The assertions test the output data in six ways.
- Grain. The table has one row for each order, customer or day, as the ticket states.
- Column contract. The table has the right columns, with the right names.
- Reconciliation. Totals match the source tables.
- Formula. Each metric follows the rule in the ticket.
- Edge cases. The model handles nulls, zero denominators and late rows.
- Idempotency. A second run gives the same result.
In our runs the graders held 2,789 assertions across the 103 tasks. The median task had 16, and the largest had 219. Scoring is strict: a task passes only if every assertion passes. A result of 23 out of 24 is a fail.
The benchmark reports two scores. Pass@1 is the share of tasks an agent passes in one attempt. Pass^3 is the share it passes three times out of three, which measures consistency.
What the benchmark does and does not test
It tests | It does not test |
|---|---|
Building and repairing dbt™ models inside a large project | Ingestion, orchestration, scheduling and deployment |
Business logic in finance, inventory, marketing and customer analytics | Industries other than one synthetic retail company |
Correct output data, checked against a reference solution | Code style, documentation and maintainability |
SQL on Snowflake and DuckDB | Spark, Airflow, and other warehouses |
Consistency across repeated attempts | Warehouse spend and query speed |
Work from a written ticket, with no help | Clarifying an unclear request with a person |
Token use and model cost for each task | Access control, privacy and governance |
A high score shows that an agent can do well-specified dbt™ work in this warehouse. It does not certify the agent for your stack, your data or your review process.
A short primer on the terms
Figure 1. One trial. This is a diagram of the process, not a measurement.
Term | Meaning |
|---|---|
Model | The language model that reads and writes text. GPT-6.1 and Claude Opus 5.5 are models. |
Agent | A model that works in a loop. It reads, acts, checks the result and repeats until the task is done. |
Harness | The software around the model. It supplies the tools, the instructions and the checks. DinoAI, Cortex Code, Claude Code and Codex are harnesses. |
Tool call | One action by the agent: run a SQL query, read a file, write a file, or run dbt™. |
Skill | A written guide that the agent loads when a task needs it. One example is a guide on how to verify a model. |
Token | The unit that models read, write and bill by. One token is about three-quarters of a word. |
Prompt caching | A discount on text the model has already read in the session. In these runs, cached input cost 5% to 10% of the fresh input price. |
Trial | One attempt at one task. |
Grader | The hidden test suite that decides pass or fail. |
The harness matters because it decides what the model can see and do. Snowflake's data shows the effect. With Claude Opus 5, Cortex Code passed 73.8% of tasks and Claude Code passed 69.6%.
How we ran it
- Tasks. All 103 tasks, Snowflake edition, ported to the DinoAI harness.
- Grading. The benchmark's own graders and pass rules, unchanged.
- Time limit. 70 minutes for each task. The benchmark's current default is three hours.
- Attempts. Three graded attempts for each task with each model. Pass^3 uses all three.
- Cost. Model tokens at list price. Warehouse compute is not included.
Results
Figure 2. DinoAI rows are Paradime results from October 2026. The GPT-5.6 row uses the most recent graded attempt for each task. The other rows are Snowflake's published figures from August 2026, which average three attempts.
Harness | Model | Tasks passed | Pass@1 | Pass^3 | Assertions passed | Source |
|---|---|---|---|---|---|---|
DinoAI | GPT-5.6 | 80 of 103 | 77.7% | 65.0% | 96.3% | Paradime |
DinoAI | GPT-6.1 | 79 of 103 | 76.7% | 66.0% | 96.8% | Paradime |
DinoAI | Claude Opus 5 | 77 of 103 | 74.8% | 64.0% | 97.6% | Paradime |
Cortex Code | Claude Opus 5 | n/a | 73.8% | 64.1% | n/a | Snowflake |
DinoAI | Claude Opus 5.5 | 75 of 103 | 72.8% | 64.3% | 95.9% | Paradime |
Claude Code | Claude Opus 5 | n/a | 69.6% | 60.2% | n/a | Snowflake |
Cortex Code | GPT-5.6 | n/a | 64.1% | 55.3% | n/a | Snowflake |
Codex | GPT-5.6 | n/a | 60.5% | 49.5% | n/a | Snowflake |
Two models appear in both sets of results, so they give the fairest comparison.
GPT-5.6. DinoAI passed 77.7% of tasks and Cortex Code passed 64.1%. That is about 14 more tasks. Our 77.7% uses the most recent graded attempt for each task. One full DinoAI run, which we used to measure cost, passed 73 of 103 (70.9%). That run is still 6.8 points ahead.
Claude Opus 5. DinoAI passed 74.8% and Cortex Code passed 73.8%. The gap is one task. Our repeat runs move by about three tasks, so we call this a tie.
Consistency. DinoAI's Pass^3 ran from 64.0% to 66.0% across the four models. With GPT-6.1, DinoAI passed 68 of 103 tasks in all three attempts, a Pass^3 of 66.0%. That is the highest published Pass^3. Teradata's Tera is next at 65.0%, and Snowflake's best is 64.1%, for Cortex Code with Claude Opus 5. With GPT-5.6, DinoAI passed 67 of 103 tasks in all three attempts, a Pass^3 of 65.0%. Cortex Code's Pass^3 with GPT-5.6 is 55.3%.
DinoAI ranks #1 among all published results
We searched for every public data-eng-bench result on November 16, 2026. Besides Snowflake's own figures, we found three.
Harness | Company | Model | Pass@1 | Pass^3 | Notes |
|---|---|---|---|---|---|
JamLabs | Claude Opus 5.5 | 68.0% | Not published | Self-reported in its leaderboard submission: 210 of 309 trials | |
Exmergo | Claude Sonnet 5 | 57.3% | Not published | One attempt for each task | |
Teradata | Claude Opus 5 | Not published | 65.0% | Reports Pass^3 only |
DinoAI's 77.7% and 76.7% are the two highest Pass@1 figures in this set. The 76.7% comes from one full run. The next result is Cortex Code with Claude Opus 5, at 73.8%. DinoAI's 66.0% is also the highest Pass^3, ahead of Tera at 65.0%.
DinoAI leads Cortex Code by about 14 tasks on GPT-5.6. On Claude Opus 5 the two harnesses are level.
The scorecard
A ranking needs more than one number. This table sets DinoAI against the best other published result on eight measures and we come out on top.
Measure | DinoAI | Best other published result | Ahead |
|---|---|---|---|
Highest pass rate, one full run | 76.7% (GPT-6.1) | 73.8% (Cortex Code, Claude Opus 5) | DinoAI |
Highest Pass^3 | 66.0% (GPT-6.1) | 65.0% (Tera, Claude Opus 5) | DinoAI |
Pass rate on GPT-5.6 | 77.7% | 64.1% (Cortex Code) | DinoAI |
Pass rate on Claude Opus 5 | 74.8% | 73.8% (Cortex Code) | Level |
Pass^3 on GPT-5.6 | 65.0% | 55.3% (Cortex Code) | DinoAI |
Pass^3 on Claude Opus 5 | 64.0% | 65.0% (Tera) | Level |
Cost per task of the top single-run result | $0.69 (GPT-6.1) | $0.76 (Cortex Code, Claude Opus 5) | DinoAI |
Cost per task on the same model | $1.57 and $3.97 | $0.36 and $0.76 (Cortex Code) | Cortex Code |
Finding 1: Choice of model moved the score by 5 tasks and the cost by 5.8X
Figure 3. Up and to the left is better. DinoAI w
Harness | Model | Pass@1 | Cost per task | Tokens per task | Cost per million tokens | Cost per passed task |
|---|---|---|---|---|---|---|
DinoAI | GPT-6.1 | 76.7% | $0.69 | 1.26M | $0.55 | $0.90 |
DinoAI | Claude Opus 5.5 | 72.8% | $0.97 | 1.11M | $0.87 | $1.33 |
DinoAI | GPT-5.6 | 77.7% | $1.57 | 1.85M | $0.85 | $2.22 |
DinoAI | Claude Opus 5 | 74.8% | $3.97 | 3.33M | $1.19 | $5.31 |
Cortex Code | GPT-5.6 | 64.1% | $0.36 | 0.44M | $0.82 | $0.56 |
Codex | GPT-5.6 | 60.5% | $0.54 | 0.81M | $0.66 | $0.89 |
Cortex Code | Claude Opus 5 | 73.8% | $0.76 | 1.07M | $0.71 | $1.02 |
Claude Code | Claude Opus 5 | 69.6% | $2.96 | 4.81M | $0.62 | $4.25 |
Cost per passed task is the cost of a full run divided by the tasks it passed. For Snowflake's rows we divided its published cost per trial by its Pass@1.
With one harness and one task set, the four models finished within 5 tasks of each other while their costs were much more different. A full run of 103 tasks cost $70.81 with GPT-6.1 and $408.97 with Claude Opus 5. Claude Opus 5 passed two fewer tasks.
GPT-6.1 gave the best value in our data. It passed 79 tasks at $0.69 each, or $0.90 for each passed task.
Cortex Code is cheaper than DinoAI on both models Snowflake tested. With Claude Opus 5 it cost $0.76 for each task and DinoAI cost $3.97. With GPT-5.6 it cost $0.36 and DinoAI cost $1.57. DinoAI does more work on each task with those models, and the bill shows it.
GPT-6.1 changes that position. DinoAI with GPT-6.1 passed more tasks than any configuration Snowflake published. It also cost less for each task than Cortex Code with Claude Opus 5. Snowflake has not published results for GPT-6.1 or Claude Opus 5.5, so this comparison crosses model generations.
Finding 2: Price per token does not predict cost per task
Figure 4. We estimated the split from run averages and list prices. Each estimate matches the measured cost to within two cents.
Claude Opus 5 lists at $5 per million input tokens, and GPT-6.1 lists at $2. That is a 2.5× gap in price. The gap in cost per task was 5.8×. Two behaviors explain the difference.
The number of model calls. Claude Opus 5 made 50.0 model calls for each task. Claude Opus 5.5 made 22.6. Each call re-reads the session so far. Caching discounts the re-read, but it still costs money. Cached input was 84% to 94% of all tokens and 15% to 43% of the cost.
The amount the model writes. Output was under 2% of tokens in every run. It was 20% to 39% of the cost, because an output token lists at 5 times the price of an input token. Claude Opus 5 wrote about 62,000 output tokens for each task. The other three models wrote 16,000 to 21,000.
The effective rate was low for every model. Cost divided by total tokens gives $0.55 to $1.19 per million tokens. That is 73% to 79% below each model's list price for input.
For each task | GPT-6.1 | GPT-5.6 | Claude Opus 5 | Claude Opus 5.5 |
|---|---|---|---|---|
Model calls | 31.1 | 34.6 | 50.0 | 22.6 |
Tool operations | 121.1 | 98.3 | 72.0 | 35.2 |
Tool operations per model call | 3.9 | 2.8 | 1.4 | 1.6 |
SQL queries | 26.1 | 23.0 | 29.1 | 9.2 |
File reads | 27.8 | 14.4 | 6.0 | 3.3 |
Output tokens (estimate) | 21,000 | 16,000 | 62,000 | 19,000 |
Minutes | 12.8 | 8.9 | 19.9 | 7.4 |
Cost per tool operation | $0.006 | $0.016 | $0.055 | $0.028 |
The GPT models batch their work. GPT-6.1 ran 3.9 tool operations in each model call, and Claude Opus 5 ran 1.4. More work in each call means fewer re-reads. One tool operation cost $0.006 with GPT-6.1 and $0.055 with Claude Opus 5, a gap of about 10×.
Finding 3: The newest models scored about the same and cost far less
OpenAI and Anthropic each have a newer model in our runs than in Snowflake's August results.
Change | Tasks passed | Cost per task | Change in cost |
|---|---|---|---|
GPT-5.6 to GPT-6.1 | 80 to 79 | $1.57 to $0.69 | 56% lower |
Claude Opus 5 to Claude Opus 5.5 | 77 to 75 | $3.97 to $0.97 | 76% lower |
The two savings have different causes. We re-priced each run at the other model's list price to separate them.
OpenAI cut the price. At GPT-5.6 prices, the GPT-6.1 run would cost $1.59 for each task. The GPT-5.6 run cost $1.57. The two models did about the same amount of billable work, so the price cut explains the saving.
Anthropic cut the price and the work. At Claude Opus 5.5 prices, the Claude Opus 5 run would cost $2.56 for each task. Price explains $1.41 of the $3.00 saving. Behavior explains the other $1.59: Claude Opus 5.5 made fewer than half the model calls.
A team that stayed on the older model paid 2.3× to 4.1× more for about the same result. A harness that can change models lets you take the saving without new engineering work.
Finding 4: Most failures were one or two assertions from a pass
Figure 5. Four DinoAI runs combined. The count leaves out 9 failed trials on single-assertion graders and 2 trials that reached the time limit.
DinoAI passed 95.9% to 97.6% of all assertions. It passed 72.8% to 77.7% of tasks. The all-or-nothing pass rule explains the gap.
The four runs had 101 failed trials. 90 of them were on graders with two or more assertions. In 38 of those, the agent missed one assertion. In 27, it missed two. Only 5 trials passed nothing.
A table with one wrong rule looks finished. It builds, it has the right columns, and most of its numbers are correct. For this reason, reconciliation checks and human review matter more than the pass rate suggests.
Failed trials also cost more. A failed trial cost 11% to 38% more than a passed trial with the same model. In each run, 27% to 34% of the model spend went to tasks that failed.
Finding 5: Different models fail on different tasks
Figure 6. At least one model passed 91 of the 103 tasks.
56 tasks passed with all four models, and 12 passed with none. The other 35 passed with some models and failed with others.
The best pair is the two cheapest models. GPT-6.1 and Claude Opus 5.5 passed 90 tasks between them. Their combined cost is $1.66 for each task, which is less than half the cost of Claude Opus 5 alone. They also disagree the most: they gave the same result on only 77 of 103 tasks.
The vendor does not set the pattern. GPT-5.6 shared 16 failed tasks with GPT-6.1 and 17 with Claude Opus 5.5. Two models from one vendor did not fail more alike than models from different vendors.
These numbers are a ceiling. To reach 90 or 91, a system must pick the right answer without the grader. A single run also moves by about three tasks, so chance explains some of the 35 split tasks.
Finding 6: The difficulty label has a weak correlation to the result
Figure 7. Pass rate over 412 trials (103 tasks, four models). The easy and very hard groups are small.
Pass rates fall as the label gets harder, from 80% on medium tasks to 71% on hard tasks. The slope is shallow. 5 of the 12 tasks that no model passed carry a medium label.
Fix tasks were slightly harder than build tasks. The pooled pass rate was 71.1% on the 19 fix tasks and 76.5% on the 84 build tasks.
The wording of the ticket mattered as well. In our failure review, an unclear spec was the largest single cause of tasks that passed in some runs and failed in others. When the ticket leaves a choice open, the agent can choose differently in each run.
The harness was improved using hard checks and not more instructions.
We reviewed the failures from all our runs. Four results apply to any coding agent on data work.
- More guidance text gave small or negative gains. The agent loaded its main guides in 73% to 100% of sessions. Sessions that loaded the verification guide did not pass more often.
- The agent ignored a standing rule about 1 time in 5. This happened even when the rule covered the exact case.
- Models respond differently to the same prompt. GPT-5.6 used its task-list tool in all 103 sessions. Claude Opus 5.5 never used it.
- Deterministic checks worked with every model. One example is a total that must reconcile with the source before delivery. These checks caught errors that the agent's own review missed.
We also test each new check against stored passing attempts before we deploy it. That test caught four wordings that would have broken tasks that already passed.
When you evaluate an agent, ask what it verifies before it delivers.
Five tasks where we disagree with grader setup
12 tasks failed with every model. For 5 of them, we traced the cause to the benchmark. We reproduced each issue on the original DuckDB edition, with the benchmark's own reference solution and grader.
Task | What we disagree |
|---|---|
| The expected values on DuckDB and on Snowflake differ |
| The grader counts a growth streak from a different start than the ticket's example |
| The ticket's formula rises with spend, and the grader requires it to fall |
| The grader filters on a lowercase value, and the data is uppercase |
| The ticket allows a negative quantity, and the grader's formula cannot produce one |
We are reporting each issue to the benchmark's maintainers. On the other 98 tasks, DinoAI's pass rates are 76.5% to 81.6%. Our headline numbers use all 103 tasks, because every agent meets the same 103.
Limits of this report
- Our time limit was shorter. Two Claude Opus 5 trials reached 70 minutes and count as failures.
- Cost covers model tokens only. It leaves out warehouse compute, agent runtime and grading.
What this means for data leaders
Many of the data leaders I talk to are trying to figure out how to deploy AI successfully in their data teams and move from proof of concept to production. Most are torn between two approaches:
- Use a frontier model and give it as much data and context as possible through MCP servers
- Use a specialized harness built for data work, paired with the models of your choice.
The first approach works for simple tasks. As Figure 7 shows, DinoAI got 100% of those tasks right.
As tasks get harder and sophisticated, models need a specialized harness to solve them. Without one, they don’t get enough guidance. And as models change, the harness has to change with them: managing models, skills, and Markdown files, updating tools, re-engineering the agent graph, running extensive evals, and more becomes indispensable and it takes years to build the security, and governance tooling around it. The AI component of this is never a one-time build. It’s an ongoing and expensive maintenance cost.
Figure 3 shows that DinoAI paired with GPT models delivers the best results at a lower cost. Paradime’s DinoAI harness offers the best balance of cost and performance, and that’s exactly what we optimize for.
So for data leaders, the question is no longer “Can we build this ourselves?” It’s “How do I get the best harness on top of these models to drive business impact as quickly as possible?”
There’s also the pace of the AI market. Models, techniques, and tools can become obsolete in days or weeks, and very few teams can keep up. Teams may think they’re building, but they fall further behind every day. That’s why a harness that is constantly being developed and improved is key to adopting AI at enterprise production scale. Between GPT-5.6 and GPT-6.1, for example, cost dropped 56% at nearly the same accuracy. Without a dedicated team or vendor improving the harness, you miss savings like that and keep paying the higher fixed cost.
Our analysis shows that, beyond benchmarks, running AI agents in production for data engineering and DataOps takes laser focus on the harness.
That’s what we’ve done with DinoAI, and the result shows. DinoAI currently has the highest accuracy on the market on this very hard data-eng-bench. DinoAI was also ranked first in our earlier ADE-bench report.
As more workflows in the enterprise becomes agentic, users give secure and highly accurate agents more agency to operate autonomously and that in turn drives enterprise adoption. With DinoAI our customers achieving exactly that without the devops overhead.
Try it
DinoAI runs inside Paradime, on your dbt™ pipelines, python, and Spark projects alongside your warehouse. If you have questions about our evaluation, please send them to benchmarks@paradime.io.
Interested to Learn More? Try Out the Free 14-Days Trial
One AI-native workspace to build, ship, and monitor dbt™ — without the tool sprawl.
More Articles
Join industry leaders winning with DinoAI agents
Data engineers and analysts at the most AI-native startups and enterprises use Paradime's DinoAI agents everyday to become market leaders.


