Introduction
AI assistants can now generate SQL from natural-language requirements in seconds.
A developer can ask:
Find the top 10 customers by total order value
for the previous 90 days.
and receive a query almost immediately.
That is useful, but generating syntactically valid SQL is not the same as generating correct SQL.
A query can run successfully and still return the wrong rows, use an inefficient execution plan, mishandle NULL values, calculate aggregates incorrectly, or make assumptions about the database schema that are not valid.
For teams using AI for database development, the more useful question is:
How closely does AI-generated SQL match a query written and reviewed by an experienced database developer when both are tested against the same real-world dataset?
That question can be measured.
This article presents a practical benchmark methodology for comparing AI-generated SQL with expert-written SQL, with examples relevant to .NET applications, SQL Server, PostgreSQL, and other relational database systems.
Why SQL Benchmarking Is Different
Source-code generation can often be validated through compilation and unit tests.
SQL is different.
Consider:
SELECT *
FROM Orders
WHERE OrderDate >= DATEADD(DAY, -30, GETDATE());
The query may execute successfully.
But suppose the requirement says:
"Orders from the previous 30 complete calendar days."
The query may not represent the requirement exactly.
This creates several dimensions of SQL quality:
SQL Quality
|
+--> Syntax
+--> Correctness
+--> Result Accuracy
+--> Performance
+--> Maintainability
+--> Security
A benchmark should measure more than whether the query runs.
AI-Generated SQL vs Expert SQL
The comparison should use two queries solving exactly the same problem.
For example:
Requirement
|
+----------+
| |
v v
AI Query Expert Query
| |
v v
Same Dataset
|
v
Same Validation
The expert query should not automatically be considered perfect.
Expert-written SQL can contain bugs too.
The goal is to establish a strong reference implementation and compare both approaches against the business requirement and actual data.
Build a Realistic Dataset
A benchmark becomes much more useful when the dataset resembles an actual application.
For an e-commerce system, you might have:
Customers
Products
Orders
OrderItems
Payments
Addresses
Categories
For example:
CREATE TABLE Customers
(
Id INT PRIMARY KEY,
Name NVARCHAR(200),
CreatedAt DATETIME2 NOT NULL
);
CREATE TABLE Orders
(
Id BIGINT PRIMARY KEY,
CustomerId INT NOT NULL,
OrderDate DATETIME2 NOT NULL,
Status VARCHAR(30) NOT NULL,
TotalAmount DECIMAL(18, 2) NOT NULL,
CONSTRAINT FK_Orders_Customers
FOREIGN KEY (CustomerId)
REFERENCES Customers(Id)
);
The dataset should contain realistic relationships and edge cases.
Include Difficult Data
A benchmark using perfectly clean data can produce misleading results.
Include cases such as:
NULL values
Duplicate business values
Multiple orders per customer
Customers without orders
Orders without optional fields
Large monetary values
Different dates
Cancelled orders
Partial payments
For example:
Customer A
|
+-- Order 1
+-- Order 2
+-- Order 3
Customer B
|
+-- No orders
Customer C
|
+-- Cancelled order
These cases expose incorrect joins and filtering assumptions.
Define the Business Requirement First
Do not begin the benchmark with SQL.
Begin with a natural-language requirement.
For example:
Return the top 10 customers by completed-order
revenue during the previous 90 days.
Cancelled orders must not be included.
Customers with no qualifying orders should not
appear in the result.
This becomes the benchmark specification.
Both the AI and expert developer should receive the same requirement.
Example Expert Query
An expert might write:
SELECT TOP (10)
c.Id,
c.Name,
SUM(o.TotalAmount) AS TotalRevenue
FROM Customers AS c
INNER JOIN Orders AS o
ON o.CustomerId = c.Id
WHERE o.Status = 'Completed'
AND o.OrderDate >= DATEADD(DAY, -90, SYSUTCDATETIME())
GROUP BY
c.Id,
c.Name
ORDER BY
TotalRevenue DESC;
The exact query depends on the database engine and business definition.
The important point is that the query reflects explicit business rules.
Example AI-Generated Query
An AI system might produce:
SELECT TOP 10
c.Id,
c.Name,
SUM(o.TotalAmount) AS TotalRevenue
FROM Customers c
JOIN Orders o
ON c.Id = o.CustomerId
WHERE o.OrderDate >= DATEADD(DAY, -90, GETDATE())
GROUP BY c.Id, c.Name
ORDER BY TotalRevenue DESC;
At first glance, this looks reasonable.
But it misses:
Status = 'Completed'
It also uses:
GETDATE()
instead of:
SYSUTCDATETIME()
if the application's business timestamps are stored in UTC.
The query runs.
The query can still be wrong.
This is exactly why execution success cannot be the primary benchmark.
Result Correctness
The strongest way to evaluate SQL correctness is to compare results against a trusted reference.
Conceptually:
Business Requirement
|
v
Expected Result
|
+---- AI Query
|
+---- Expert Query
Then compare:
Rows
Columns
Values
Ordering
Duplicates
NULL handling
Aggregates
For deterministic queries, the result sets should match.
Result Set Comparison
A simple comparison process can be:
AI Result
|
v
Normalize
|
v
Compare
Expert Result
|
v
Normalize
|
v
Compare
Normalization may include sorting rows when order is not part of the requirement.
For example:
Expected:
Customer 1 | 5000
Customer 2 | 4500
AI:
Customer 2 | 4500
Customer 1 | 5000
These results may be equivalent if ordering was not specified.
The benchmark must distinguish semantic differences from irrelevant presentation differences.
Exact Match Is Not Always Enough
Some queries produce floating-point or calculated values where exact string comparison is inappropriate.
For example:
Expected: 125.50
Actual: 125.5000
These may be numerically equivalent.
Likewise, database engines can represent some expressions differently while producing the same logical result.
Therefore, result comparison should be based on the data type and business meaning.
Test Multiple Query Categories
A good benchmark should include different SQL workloads.
Basic Filtering
SELECT *
FROM Orders
WHERE Status = 'Completed';
Aggregation
SELECT
CustomerId,
SUM(TotalAmount) AS Revenue
FROM Orders
GROUP BY CustomerId;
Multi-Table Joins
SELECT
o.Id,
c.Name,
o.TotalAmount
FROM Orders o
JOIN Customers c
ON c.Id = o.CustomerId;
Subqueries
SELECT *
FROM Customers
WHERE Id IN
(
SELECT CustomerId
FROM Orders
WHERE TotalAmount > 1000
);
Window Functions
SELECT
CustomerId,
OrderDate,
TotalAmount,
ROW_NUMBER() OVER
(
PARTITION BY CustomerId
ORDER BY OrderDate DESC
) AS OrderRank
FROM Orders;
Common Table Expressions
WITH CustomerTotals AS
(
SELECT
CustomerId,
SUM(TotalAmount) AS TotalRevenue
FROM Orders
GROUP BY CustomerId
)
SELECT *
FROM CustomerTotals
WHERE TotalRevenue > 10000;
Date-Based Analysis
Date logic is especially important because time zones and boundaries can easily produce incorrect results.
Include Negative Cases
A strong benchmark should include requirements designed to expose common SQL mistakes.
For example:
Find customers who have never placed an order.
An inexperienced query may use:
SELECT c.*
FROM Customers c
JOIN Orders o
ON o.CustomerId = c.Id
WHERE o.Id IS NULL;
This is logically incorrect because an inner join removes customers without orders.
The correct pattern is:
SELECT c.*
FROM Customers c
LEFT JOIN Orders o
ON o.CustomerId = c.Id
WHERE o.Id IS NULL;
The benchmark should contain these cases because they distinguish syntactic SQL generation from genuine query reasoning.
NULL Handling
NULL is another useful benchmark category.
Consider:
SELECT *
FROM Customers
WHERE Email <> '[email protected]';
A developer might expect this to return customers whose email is different.
But rows where Email is NULL do not satisfy the comparison in the same way as ordinary values.
A requirement such as:
"Return customers whose email is not the specified address,
including customers without an email."
requires explicit handling.
For example:
SELECT *
FROM Customers
WHERE Email <> '[email protected]'
OR Email IS NULL;
These cases should be part of realistic AI-SQL testing.
Aggregate Accuracy
Aggregation errors can be subtle.
Suppose:
Order
|
+-- Item A
+-- Item B
If the query joins Orders to OrderItems, the order-level amount may accidentally be repeated for each item.
For example:
SELECT
SUM(o.TotalAmount)
FROM Orders o
JOIN OrderItems i
ON i.OrderId = o.Id;
If an order contains five items, its amount can contribute five times to the aggregate.
This is a classic query-design problem that a benchmark should deliberately test.
Performance Benchmarking
Correct results are necessary, but performance also matters.
A query that returns the right result in two seconds may be acceptable for an administrative report but unacceptable for an API endpoint handling hundreds of requests per second.
Measure:
Execution Time
CPU
Logical Reads
Physical Reads
Rows Processed
Memory Grant
Execution Plan
For SQL Server, execution statistics can help identify expensive queries.
For example:
SET STATISTICS IO ON;
SET STATISTICS TIME ON;
SELECT
CustomerId,
SUM(TotalAmount)
FROM Orders
GROUP BY CustomerId;
The exact diagnostic commands vary by database engine.
Use the Same Database State
Performance comparisons must use the same conditions.
Keep consistent:
Database Engine
Dataset
Indexes
Statistics
Hardware
Configuration
Query Parameters
Otherwise:
AI Query
and:
Expert Query
may not be comparable.
Warm and Cold Cache
Database caching can affect benchmark results.
A query executed immediately after another query may benefit from cached data.
Therefore, distinguish:
Cold-ish execution
from:
Warm execution
Do not overstate the meaning of a single timing result.
Run multiple iterations and report the methodology.
Parameterization
AI-generated SQL should also be checked for safe parameter handling.
A dangerous application pattern is:
var sql =
$"SELECT * FROM Customers WHERE Name = '{name}'";
A safer approach is:
var sql =
"SELECT * FROM Customers WHERE Name = @Name";
var command = new SqlCommand(sql, connection);
command.Parameters.AddWithValue("@Name", name);
In EF Core, LINQ queries are normally translated and parameterized by the provider:
var customers = await db.Customers
.Where(x => x.Name == name)
.ToListAsync();
The benchmark should therefore evaluate not only whether AI produces valid SQL, but whether the generated SQL is safe to integrate into application code.
SQL Injection Testing
Include requirements where user input is involved.
For example:
Search customers by name.
An AI-generated query should not encourage string concatenation.
Evaluate:
Parameterization
Input Handling
Dynamic SQL
Identifier Handling
This is especially important when AI-generated SQL is copied directly into application code.
Testing With .NET and EF Core
A common .NET workflow is:
User Request
|
v
Application Service
|
v
EF Core
|
v
SQL
|
v
Database
An AI coding assistant may generate LINQ rather than raw SQL.
That should also be benchmarked.
For example:
var result = await db.Orders
.Where(x => x.Status == OrderStatus.Completed)
.GroupBy(x => x.CustomerId)
.Select(x => new
{
CustomerId = x.Key,
Revenue = x.Sum(o => o.TotalAmount)
})
.OrderByDescending(x => x.Revenue)
.Take(10)
.ToListAsync();
The benchmark should evaluate both:
Generated LINQ
and:
Generated SQL
because a query can be logically correct while producing an inefficient SQL translation.
Compare Execution Plans
Two queries can return identical results while using very different execution plans.
For example:
Query A
Index Seek
|
v
Small Read
Query B
Table Scan
|
v
Large Read
The results are identical.
The operational cost is not.
Therefore, an advanced benchmark should compare execution plans where the database engine makes that information available.
Look for:
Table Scans
Index Scans
Index Seeks
Large Sorts
Hash Operations
Excessive Reads
Large Memory Grants
Do not automatically classify every scan as bad.
A scan can be perfectly reasonable when most rows need to be processed.
The benchmark should interpret plans in context.
Maintainability
SQL quality is not only about performance.
Compare:
SELECT
c.Id,
c.Name,
SUM(o.TotalAmount) AS Revenue
FROM Customers c
JOIN Orders o
ON o.CustomerId = c.Id
WHERE o.Status = 'Completed'
GROUP BY c.Id, c.Name;
with an unnecessarily complicated equivalent query.
The simpler query may be easier to maintain.
Evaluate:
Readability
Naming
Complexity
Duplication
Comments
Consistency
A query that only the original author understands creates maintenance cost.
Portability
If an application supports multiple database engines, AI-generated SQL may accidentally depend on one vendor's syntax.
For example:
SELECT TOP 10 ...
is appropriate for SQL Server.
Another database may use:
LIMIT 10
or a different mechanism.
The benchmark should therefore specify the target database.
Otherwise, an AI model may generate technically valid SQL for the wrong platform.
Test Schema Understanding
Give the AI only the schema information that a developer would reasonably have.
For example:
Customers(Id, Name, CreatedAt)
Orders(Id, CustomerId, OrderDate, Status, TotalAmount)
OrderItems(Id, OrderId, ProductId, Quantity, UnitPrice)
Then ask:
Calculate revenue by customer.
This tests whether the model can reason from the available schema rather than relying on hidden assumptions.
Test Ambiguous Requirements
Real requirements are often incomplete.
For example:
"Show the most valuable customers."
What does "valuable" mean?
Possible interpretations:
Highest order value
Highest lifetime revenue
Highest recent revenue
Highest number of orders
A good AI assistant should ask for clarification when the requirement is genuinely ambiguous.
This is an important benchmark category because blindly generating SQL can create confidently incorrect results.
Benchmark Dataset Size
A benchmark should include more than one dataset size.
For example:
Small
10,000 rows
Medium
1,000,000 rows
Large
10,000,000+ rows
The exact sizes depend on the available infrastructure.
The purpose is to test how query behavior changes as data volume increases.
A query that performs well on 10,000 rows may behave very differently on millions of rows.
Benchmark Matrix
A useful benchmark matrix could look like:
| Query Type | Small Dataset | Medium Dataset | Large Dataset |
|---|
| Filtering | Test | Test | Test |
| Aggregation | Test | Test | Test |
| Join | Test | Test | Test |
| Subquery | Test | Test | Test |
| Window function | Test | Test | Test |
| Date analysis | Test | Test | Test |
| Pagination | Test | Test | Test |
| Complex reporting | Test | Test | Test |
This makes the evaluation systematic.
Common Mistakes
Mistake 1: Checking Only Syntax
A query can execute successfully and still be logically wrong.
Mistake 2: Using Only Tiny Datasets
Small datasets hide performance problems.
Mistake 3: Comparing Different Database States
The benchmark becomes unreliable.
Mistake 4: Ignoring NULL Values
Real databases contain missing data.
Mistake 5: Ignoring Duplicate Rows
Joins can unintentionally multiply rows.
Mistake 6: Measuring Only Execution Time
Correctness must come first.
Mistake 7: Trusting Expert SQL Blindly
Reference queries should also be validated.
Mistake 8: Ignoring Security
Generated dynamic SQL can introduce injection vulnerabilities.
Troubleshooting
| Problem | What to Check |
|---|
| AI query returns different rows | Compare filters, joins, and date boundaries |
| Row count is too high | Look for join multiplication |
| Aggregates are incorrect | Check grouping and duplicated relationships |
| Query is slow | Inspect indexes and execution plan |
| NULL rows disappear | Review three-valued SQL logic |
| Query works on one database but not another | Check vendor-specific syntax |
| EF Core query performs poorly | Inspect generated SQL |
| Results differ between runs | Check nondeterministic ordering and time-based filters |
| AI generates unsafe SQL | Require parameterization and review dynamic SQL |
Best Practices
Start With a Precise Requirement
Business meaning should be defined before SQL generation.
Give the AI the Actual Schema
Do not expect correct SQL from incomplete schema information.
Validate Against Trusted Results
Use a reference implementation or independently verified expected results.
Test Edge Cases
Include:
NULL
Empty
Duplicate
Missing Relationship
Boundary Date
Large Value
Measure Performance Separately
Correctness and performance are different dimensions.
Use Realistic Data Volumes
Performance claims from tiny datasets are often misleading.
Check Generated SQL
When AI generates LINQ, inspect the SQL produced by the ORM where performance matters.
Review Security
Parameterization should be mandatory for user-controlled values.
Record the Full Benchmark Environment
Include database version, hardware, indexes, dataset size, model, and prompt.
Advantages of AI-Generated SQL
Faster Initial Query Development
Natural-language requirements can quickly produce a starting query.
Useful for Exploration
Developers can use AI to explore alternative query approaches.
Helpful for Learning
Generated explanations can help developers understand unfamiliar SQL concepts.
Good for Boilerplate
Simple filtering, grouping, and projection queries can often be generated quickly.
Supports .NET Development
AI can generate both LINQ and SQL, making it useful during EF Core development.
Disadvantages and Limitations
Incorrect Logic Can Look Correct
A syntactically valid query can still violate the business requirement.
Performance Is Difficult to Predict
The model cannot always know how the database will execute the query.
Schema Assumptions
AI may invent or misunderstand relationships when schema information is incomplete.
Vendor-Specific Differences
SQL syntax differs across database engines.
Security Risks
Generated dynamic SQL can introduce vulnerabilities if integrated without review.
Human Validation Remains Necessary
Production SQL should be tested against real requirements and representative data.
A Practical Benchmark Workflow
A strong AI-SQL evaluation can follow this process:
Business Requirement
|
v
Schema Definition
|
+----------------+
| |
v v
AI Query Expert Query
| |
+--------+-------+
|
v
Syntax Validation
|
v
Result Validation
|
v
Edge-Case Validation
|
v
Performance Benchmark
|
v
Security Review
|
v
Final Comparison
This keeps the benchmark focused on actual engineering value.
Example Final Scorecard
For internal experimentation, a team could score:
| Category | Weight |
|---|
| Result correctness | 30 |
| Edge-case correctness | 15 |
| Performance | 20 |
| Security | 15 |
| Maintainability | 10 |
| Database compatibility | 10 |
| Total | 100 |
The weights should be adjusted to match the team's workload.
A reporting system may prioritize performance.
A financial application may place more emphasis on correctness and security.
What a Good Result Looks Like
The purpose of the benchmark is not to prove that AI is better than experts.
A useful result might instead show:
AI
Good at:
- Basic queries
- Boilerplate
- Query alternatives
- Simple aggregations
Needs review for:
- Complex joins
- Business-specific rules
- Performance-sensitive queries
- Security-sensitive SQL
That is a practical engineering conclusion.
The benchmark becomes valuable when it identifies where AI can safely accelerate SQL development and where expert review remains essential.
Conclusion
AI-generated SQL can save developers time, but a query that runs successfully is not necessarily a correct or production-ready query. The most useful way to evaluate AI SQL generation is to compare it with independently reviewed expert queries using the same requirements, schema, database, indexes, and datasets. The benchmark should measure result correctness first, followed by edge-case behavior, performance, security, maintainability, and database compatibility. For .NET developers, the evaluation should also include AI-generated LINQ and the SQL produced by Entity Framework Core because the generated application code and generated database query are both part of the final result. AI is particularly useful for creating a strong first draft, exploring query alternatives, and handling routine SQL, but complex business logic, performance-sensitive workloads, and security-critical database operations still require careful human validation.