Benchmarking AI-Generated SQL Against Expert Queries on Real Datasets

Introduction

AI assistants can now generate SQL from natural-language requirements in seconds.

A developer can ask:

Find the top 10 customers by total order value
for the previous 90 days.

and receive a query almost immediately.

That is useful, but generating syntactically valid SQL is not the same as generating correct SQL.

A query can run successfully and still return the wrong rows, use an inefficient execution plan, mishandle NULL values, calculate aggregates incorrectly, or make assumptions about the database schema that are not valid.

For teams using AI for database development, the more useful question is:

How closely does AI-generated SQL match a query written and reviewed by an experienced database developer when both are tested against the same real-world dataset?

That question can be measured.

This article presents a practical benchmark methodology for comparing AI-generated SQL with expert-written SQL, with examples relevant to .NET applications, SQL Server, PostgreSQL, and other relational database systems.

Why SQL Benchmarking Is Different

Source-code generation can often be validated through compilation and unit tests.

SQL is different.

Consider:

SELECT *
FROM Orders
WHERE OrderDate >= DATEADD(DAY, -30, GETDATE());

The query may execute successfully.

But suppose the requirement says:

"Orders from the previous 30 complete calendar days."

The query may not represent the requirement exactly.

This creates several dimensions of SQL quality:

SQL Quality
    |
    +--> Syntax
    +--> Correctness
    +--> Result Accuracy
    +--> Performance
    +--> Maintainability
    +--> Security

A benchmark should measure more than whether the query runs.

AI-Generated SQL vs Expert SQL

The comparison should use two queries solving exactly the same problem.

For example:

Requirement
      |
      +----------+
      |          |
      v          v
AI Query     Expert Query
      |          |
      v          v
Same Dataset
      |
      v
Same Validation

The expert query should not automatically be considered perfect.

Expert-written SQL can contain bugs too.

The goal is to establish a strong reference implementation and compare both approaches against the business requirement and actual data.

Build a Realistic Dataset

A benchmark becomes much more useful when the dataset resembles an actual application.

For an e-commerce system, you might have:

Customers
Products
Orders
OrderItems
Payments
Addresses
Categories

For example:

CREATE TABLE Customers
(
    Id INT PRIMARY KEY,
    Name NVARCHAR(200),
    CreatedAt DATETIME2 NOT NULL
);

CREATE TABLE Orders
(
    Id BIGINT PRIMARY KEY,
    CustomerId INT NOT NULL,
    OrderDate DATETIME2 NOT NULL,
    Status VARCHAR(30) NOT NULL,
    TotalAmount DECIMAL(18, 2) NOT NULL,

    CONSTRAINT FK_Orders_Customers
        FOREIGN KEY (CustomerId)
        REFERENCES Customers(Id)
);

The dataset should contain realistic relationships and edge cases.

Include Difficult Data

A benchmark using perfectly clean data can produce misleading results.

Include cases such as:

NULL values
Duplicate business values
Multiple orders per customer
Customers without orders
Orders without optional fields
Large monetary values
Different dates
Cancelled orders
Partial payments

For example:

Customer A
  |
  +-- Order 1
  +-- Order 2
  +-- Order 3

Customer B
  |
  +-- No orders

Customer C
  |
  +-- Cancelled order

These cases expose incorrect joins and filtering assumptions.

Define the Business Requirement First

Do not begin the benchmark with SQL.

Begin with a natural-language requirement.

For example:

Return the top 10 customers by completed-order
revenue during the previous 90 days.

Cancelled orders must not be included.

Customers with no qualifying orders should not
appear in the result.

This becomes the benchmark specification.

Both the AI and expert developer should receive the same requirement.

Example Expert Query

An expert might write:

SELECT TOP (10)
    c.Id,
    c.Name,
    SUM(o.TotalAmount) AS TotalRevenue
FROM Customers AS c
INNER JOIN Orders AS o
    ON o.CustomerId = c.Id
WHERE o.Status = 'Completed'
  AND o.OrderDate >= DATEADD(DAY, -90, SYSUTCDATETIME())
GROUP BY
    c.Id,
    c.Name
ORDER BY
    TotalRevenue DESC;

The exact query depends on the database engine and business definition.

The important point is that the query reflects explicit business rules.

Example AI-Generated Query

An AI system might produce:

SELECT TOP 10
    c.Id,
    c.Name,
    SUM(o.TotalAmount) AS TotalRevenue
FROM Customers c
JOIN Orders o
    ON c.Id = o.CustomerId
WHERE o.OrderDate >= DATEADD(DAY, -90, GETDATE())
GROUP BY c.Id, c.Name
ORDER BY TotalRevenue DESC;

At first glance, this looks reasonable.

But it misses:

Status = 'Completed'

It also uses:

GETDATE()

instead of:

SYSUTCDATETIME()

if the application's business timestamps are stored in UTC.

The query runs.

The query can still be wrong.

This is exactly why execution success cannot be the primary benchmark.

Result Correctness

The strongest way to evaluate SQL correctness is to compare results against a trusted reference.

Conceptually:

Business Requirement
        |
        v
Expected Result
        |
        +---- AI Query
        |
        +---- Expert Query

Then compare:

Rows
Columns
Values
Ordering
Duplicates
NULL handling
Aggregates

For deterministic queries, the result sets should match.

Result Set Comparison

A simple comparison process can be:

AI Result
   |
   v
Normalize
   |
   v
Compare

Expert Result
   |
   v
Normalize
   |
   v
Compare

Normalization may include sorting rows when order is not part of the requirement.

For example:

Expected:
Customer 1 | 5000
Customer 2 | 4500

AI:
Customer 2 | 4500
Customer 1 | 5000

These results may be equivalent if ordering was not specified.

The benchmark must distinguish semantic differences from irrelevant presentation differences.

Exact Match Is Not Always Enough

Some queries produce floating-point or calculated values where exact string comparison is inappropriate.

For example:

Expected: 125.50
Actual:   125.5000

These may be numerically equivalent.

Likewise, database engines can represent some expressions differently while producing the same logical result.

Therefore, result comparison should be based on the data type and business meaning.

Test Multiple Query Categories

A good benchmark should include different SQL workloads.

Basic Filtering

SELECT *
FROM Orders
WHERE Status = 'Completed';

Aggregation

SELECT
    CustomerId,
    SUM(TotalAmount) AS Revenue
FROM Orders
GROUP BY CustomerId;

Multi-Table Joins

SELECT
    o.Id,
    c.Name,
    o.TotalAmount
FROM Orders o
JOIN Customers c
    ON c.Id = o.CustomerId;

Subqueries

SELECT *
FROM Customers
WHERE Id IN
(
    SELECT CustomerId
    FROM Orders
    WHERE TotalAmount > 1000
);

Window Functions

SELECT
    CustomerId,
    OrderDate,
    TotalAmount,
    ROW_NUMBER() OVER
    (
        PARTITION BY CustomerId
        ORDER BY OrderDate DESC
    ) AS OrderRank
FROM Orders;

Common Table Expressions

WITH CustomerTotals AS
(
    SELECT
        CustomerId,
        SUM(TotalAmount) AS TotalRevenue
    FROM Orders
    GROUP BY CustomerId
)
SELECT *
FROM CustomerTotals
WHERE TotalRevenue > 10000;

Date-Based Analysis

Date logic is especially important because time zones and boundaries can easily produce incorrect results.

Include Negative Cases

A strong benchmark should include requirements designed to expose common SQL mistakes.

For example:

Find customers who have never placed an order.

An inexperienced query may use:

SELECT c.*
FROM Customers c
JOIN Orders o
    ON o.CustomerId = c.Id
WHERE o.Id IS NULL;

This is logically incorrect because an inner join removes customers without orders.

The correct pattern is:

SELECT c.*
FROM Customers c
LEFT JOIN Orders o
    ON o.CustomerId = c.Id
WHERE o.Id IS NULL;

The benchmark should contain these cases because they distinguish syntactic SQL generation from genuine query reasoning.

NULL Handling

NULL is another useful benchmark category.

Consider:

SELECT *
FROM Customers
WHERE Email <> '[email protected]';

A developer might expect this to return customers whose email is different.

But rows where Email is NULL do not satisfy the comparison in the same way as ordinary values.

A requirement such as:

"Return customers whose email is not the specified address,
including customers without an email."

requires explicit handling.

For example:

SELECT *
FROM Customers
WHERE Email <> '[email protected]'
   OR Email IS NULL;

These cases should be part of realistic AI-SQL testing.

Aggregate Accuracy

Aggregation errors can be subtle.

Suppose:

Order
 |
 +-- Item A
 +-- Item B

If the query joins Orders to OrderItems, the order-level amount may accidentally be repeated for each item.

For example:

SELECT
    SUM(o.TotalAmount)
FROM Orders o
JOIN OrderItems i
    ON i.OrderId = o.Id;

If an order contains five items, its amount can contribute five times to the aggregate.

This is a classic query-design problem that a benchmark should deliberately test.

Performance Benchmarking

Correct results are necessary, but performance also matters.

A query that returns the right result in two seconds may be acceptable for an administrative report but unacceptable for an API endpoint handling hundreds of requests per second.

Measure:

Execution Time
CPU
Logical Reads
Physical Reads
Rows Processed
Memory Grant
Execution Plan

For SQL Server, execution statistics can help identify expensive queries.

For example:

SET STATISTICS IO ON;
SET STATISTICS TIME ON;

SELECT
    CustomerId,
    SUM(TotalAmount)
FROM Orders
GROUP BY CustomerId;

The exact diagnostic commands vary by database engine.

Use the Same Database State

Performance comparisons must use the same conditions.

Keep consistent:

Database Engine
Dataset
Indexes
Statistics
Hardware
Configuration
Query Parameters

Otherwise:

AI Query

and:

Expert Query

may not be comparable.

Warm and Cold Cache

Database caching can affect benchmark results.

A query executed immediately after another query may benefit from cached data.

Therefore, distinguish:

Cold-ish execution

from:

Warm execution

Do not overstate the meaning of a single timing result.

Run multiple iterations and report the methodology.

Parameterization

AI-generated SQL should also be checked for safe parameter handling.

A dangerous application pattern is:

var sql =
    $"SELECT * FROM Customers WHERE Name = '{name}'";

A safer approach is:

var sql =
    "SELECT * FROM Customers WHERE Name = @Name";

var command = new SqlCommand(sql, connection);

command.Parameters.AddWithValue("@Name", name);

In EF Core, LINQ queries are normally translated and parameterized by the provider:

var customers = await db.Customers
    .Where(x => x.Name == name)
    .ToListAsync();

The benchmark should therefore evaluate not only whether AI produces valid SQL, but whether the generated SQL is safe to integrate into application code.

SQL Injection Testing

Include requirements where user input is involved.

For example:

Search customers by name.

An AI-generated query should not encourage string concatenation.

Evaluate:

Parameterization
Input Handling
Dynamic SQL
Identifier Handling

This is especially important when AI-generated SQL is copied directly into application code.

Testing With .NET and EF Core

A common .NET workflow is:

User Request
     |
     v
Application Service
     |
     v
EF Core
     |
     v
SQL
     |
     v
Database

An AI coding assistant may generate LINQ rather than raw SQL.

That should also be benchmarked.

For example:

var result = await db.Orders
    .Where(x => x.Status == OrderStatus.Completed)
    .GroupBy(x => x.CustomerId)
    .Select(x => new
    {
        CustomerId = x.Key,
        Revenue = x.Sum(o => o.TotalAmount)
    })
    .OrderByDescending(x => x.Revenue)
    .Take(10)
    .ToListAsync();

The benchmark should evaluate both:

Generated LINQ

and:

Generated SQL

because a query can be logically correct while producing an inefficient SQL translation.

Compare Execution Plans

Two queries can return identical results while using very different execution plans.

For example:

Query A
Index Seek
     |
     v
Small Read

Query B
Table Scan
     |
     v
Large Read

The results are identical.

The operational cost is not.

Therefore, an advanced benchmark should compare execution plans where the database engine makes that information available.

Look for:

Table Scans
Index Scans
Index Seeks
Large Sorts
Hash Operations
Excessive Reads
Large Memory Grants

Do not automatically classify every scan as bad.

A scan can be perfectly reasonable when most rows need to be processed.

The benchmark should interpret plans in context.

Maintainability

SQL quality is not only about performance.

Compare:

SELECT
    c.Id,
    c.Name,
    SUM(o.TotalAmount) AS Revenue
FROM Customers c
JOIN Orders o
    ON o.CustomerId = c.Id
WHERE o.Status = 'Completed'
GROUP BY c.Id, c.Name;

with an unnecessarily complicated equivalent query.

The simpler query may be easier to maintain.

Evaluate:

Readability
Naming
Complexity
Duplication
Comments
Consistency

A query that only the original author understands creates maintenance cost.

Portability

If an application supports multiple database engines, AI-generated SQL may accidentally depend on one vendor's syntax.

For example:

SELECT TOP 10 ...

is appropriate for SQL Server.

Another database may use:

LIMIT 10

or a different mechanism.

The benchmark should therefore specify the target database.

Otherwise, an AI model may generate technically valid SQL for the wrong platform.

Test Schema Understanding

Give the AI only the schema information that a developer would reasonably have.

For example:

Customers(Id, Name, CreatedAt)

Orders(Id, CustomerId, OrderDate, Status, TotalAmount)

OrderItems(Id, OrderId, ProductId, Quantity, UnitPrice)

Then ask:

Calculate revenue by customer.

This tests whether the model can reason from the available schema rather than relying on hidden assumptions.

Test Ambiguous Requirements

Real requirements are often incomplete.

For example:

"Show the most valuable customers."

What does "valuable" mean?

Possible interpretations:

Highest order value
Highest lifetime revenue
Highest recent revenue
Highest number of orders

A good AI assistant should ask for clarification when the requirement is genuinely ambiguous.

This is an important benchmark category because blindly generating SQL can create confidently incorrect results.

Benchmark Dataset Size

A benchmark should include more than one dataset size.

For example:

Small
10,000 rows

Medium
1,000,000 rows

Large
10,000,000+ rows

The exact sizes depend on the available infrastructure.

The purpose is to test how query behavior changes as data volume increases.

A query that performs well on 10,000 rows may behave very differently on millions of rows.

Benchmark Matrix

A useful benchmark matrix could look like:

Query Type	Small Dataset	Medium Dataset	Large Dataset
Filtering	Test	Test	Test
Aggregation	Test	Test	Test
Join	Test	Test	Test
Subquery	Test	Test	Test
Window function	Test	Test	Test
Date analysis	Test	Test	Test
Pagination	Test	Test	Test
Complex reporting	Test	Test	Test

This makes the evaluation systematic.

Common Mistakes

Mistake 1: Checking Only Syntax

A query can execute successfully and still be logically wrong.

Mistake 2: Using Only Tiny Datasets

Small datasets hide performance problems.

Mistake 3: Comparing Different Database States

The benchmark becomes unreliable.

Mistake 4: Ignoring NULL Values

Real databases contain missing data.

Mistake 5: Ignoring Duplicate Rows

Joins can unintentionally multiply rows.

Mistake 6: Measuring Only Execution Time

Correctness must come first.

Mistake 7: Trusting Expert SQL Blindly

Reference queries should also be validated.

Mistake 8: Ignoring Security

Generated dynamic SQL can introduce injection vulnerabilities.

Troubleshooting

Problem	What to Check
AI query returns different rows	Compare filters, joins, and date boundaries
Row count is too high	Look for join multiplication
Aggregates are incorrect	Check grouping and duplicated relationships
Query is slow	Inspect indexes and execution plan
NULL rows disappear	Review three-valued SQL logic
Query works on one database but not another	Check vendor-specific syntax
EF Core query performs poorly	Inspect generated SQL
Results differ between runs	Check nondeterministic ordering and time-based filters
AI generates unsafe SQL	Require parameterization and review dynamic SQL

Best Practices

Start With a Precise Requirement

Business meaning should be defined before SQL generation.

Give the AI the Actual Schema

Do not expect correct SQL from incomplete schema information.

Validate Against Trusted Results

Use a reference implementation or independently verified expected results.

Test Edge Cases

Include:

NULL
Empty
Duplicate
Missing Relationship
Boundary Date
Large Value

Measure Performance Separately

Correctness and performance are different dimensions.

Use Realistic Data Volumes

Performance claims from tiny datasets are often misleading.

Check Generated SQL

When AI generates LINQ, inspect the SQL produced by the ORM where performance matters.

Review Security

Parameterization should be mandatory for user-controlled values.

Record the Full Benchmark Environment

Include database version, hardware, indexes, dataset size, model, and prompt.

Advantages of AI-Generated SQL

Faster Initial Query Development

Natural-language requirements can quickly produce a starting query.

Useful for Exploration

Developers can use AI to explore alternative query approaches.

Helpful for Learning

Generated explanations can help developers understand unfamiliar SQL concepts.

Good for Boilerplate

Simple filtering, grouping, and projection queries can often be generated quickly.

Supports .NET Development

AI can generate both LINQ and SQL, making it useful during EF Core development.

Disadvantages and Limitations

Incorrect Logic Can Look Correct

A syntactically valid query can still violate the business requirement.

Performance Is Difficult to Predict

The model cannot always know how the database will execute the query.

Schema Assumptions

AI may invent or misunderstand relationships when schema information is incomplete.

Vendor-Specific Differences

SQL syntax differs across database engines.

Security Risks

Generated dynamic SQL can introduce vulnerabilities if integrated without review.

Human Validation Remains Necessary

Production SQL should be tested against real requirements and representative data.

A Practical Benchmark Workflow

A strong AI-SQL evaluation can follow this process:

Business Requirement
        |
        v
Schema Definition
        |
        +----------------+
        |                |
        v                v
    AI Query        Expert Query
        |                |
        +--------+-------+
                 |
                 v
          Syntax Validation
                 |
                 v
          Result Validation
                 |
                 v
        Edge-Case Validation
                 |
                 v
       Performance Benchmark
                 |
                 v
       Security Review
                 |
                 v
          Final Comparison

This keeps the benchmark focused on actual engineering value.

Example Final Scorecard

For internal experimentation, a team could score:

Category	Weight
Result correctness	30
Edge-case correctness	15
Performance	20
Security	15
Maintainability	10
Database compatibility	10
Total	100

The weights should be adjusted to match the team's workload.

A reporting system may prioritize performance.

A financial application may place more emphasis on correctness and security.

What a Good Result Looks Like

The purpose of the benchmark is not to prove that AI is better than experts.

A useful result might instead show:

AI
Good at:
- Basic queries
- Boilerplate
- Query alternatives
- Simple aggregations

Needs review for:
- Complex joins
- Business-specific rules
- Performance-sensitive queries
- Security-sensitive SQL

That is a practical engineering conclusion.

The benchmark becomes valuable when it identifies where AI can safely accelerate SQL development and where expert review remains essential.

Conclusion

AI-generated SQL can save developers time, but a query that runs successfully is not necessarily a correct or production-ready query. The most useful way to evaluate AI SQL generation is to compare it with independently reviewed expert queries using the same requirements, schema, database, indexes, and datasets. The benchmark should measure result correctness first, followed by edge-case behavior, performance, security, maintainability, and database compatibility. For .NET developers, the evaluation should also include AI-generated LINQ and the SQL produced by Entity Framework Core because the generated application code and generated database query are both part of the final result. AI is particularly useful for creating a strong first draft, exploring query alternatives, and handling routine SQL, but complex business logic, performance-sensitive workloads, and security-critical database operations still require careful human validation.