Google BigQuery Data Agent Observability: Tracking Latency, Cost, and Queries

Niharika Gupta

AI agents that work with data can make analytics applications easier to use, but they also introduce a new monitoring problem.

A traditional application may have a predictable sequence of API calls:

Request
  |
  v
Application
  |
  v
Database Query
  |
  v
Response

A data agent can behave differently:

User Question
      |
      v
AI Agent
      |
      +----> Interpret request
      |
      +----> Select data source
      |
      +----> Generate SQL
      |
      +----> Run BigQuery query
      |
      +----> Inspect result
      |
      +----> Generate another query
      |
      v
Final Answer

A single user question can therefore result in multiple model operations, multiple BigQuery queries, different query sizes, and different execution times.

This makes observability important.

For an AI-powered BigQuery application, developers should be able to answer questions such as:

How long did the agent take to respond?
How many BigQuery queries did it execute?
How much data did those queries process?
Which queries were slow?
Which requests generated unexpected query costs?
Did the agent repeatedly query the same data?
Where did latency occur: model processing, SQL generation, BigQuery execution, or application code?

Without these metrics, optimizing an AI data application becomes difficult.

What Is a BigQuery Data Agent?

A data agent is an AI-powered application that can understand natural-language requests and interact with data systems.

For example, a user might ask:

"Which products had the highest sales last quarter?"

Instead of requiring the user to write SQL, the agent can interpret the request and generate a query:

SELECT
    product_name,
    SUM(revenue) AS total_revenue
FROM sales
WHERE order_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
GROUP BY product_name
ORDER BY total_revenue DESC
LIMIT 10;

The agent then executes the query and converts the result into a human-readable response.

The important difference is that the application is now controlling a data workflow dynamically.

That workflow needs observability at every important stage.

Why Traditional Application Monitoring Is Not Enough

Traditional application monitoring often focuses on:

Request latency
Error rate
CPU
Memory
Database latency

These metrics are still useful, but they do not provide the full picture for an AI data agent.

Consider two requests:

Request A
1 AI operation
1 BigQuery query
500 MB processed
2 seconds

Request B
4 AI operations
7 BigQuery queries
120 GB processed
18 seconds

Both may return a successful HTTP response.

A conventional success-rate dashboard would consider both requests healthy.

From a cost and performance perspective, they are very different.

Agent observability therefore needs to connect application-level requests with model activity and BigQuery execution.

The Observability Model

A useful architecture is:

User Request
     |
     v
Agent Application
     |
     +---- Agent latency
     |
     +---- Model operations
     |
     +---- Tool calls
              |
              v
          BigQuery
              |
              +---- Query latency
              +---- Bytes processed
              +---- Slot usage
              +---- Query errors

The goal is to correlate these events.

For example:

Request ID: 8f42
Agent duration: 9.8 seconds
Model calls: 3
BigQuery queries: 2

Query 1
Bytes processed: 42 MB
Duration: 1.2 seconds

Query 2
Bytes processed: 8.4 GB
Duration: 6.7 seconds

Now the team can identify where most of the request time and data processing occurred.

Track End-to-End Agent Latency

The first metric should be the total time required to answer a request.

A simple application-level measurement can look like:

import time

start = time.perf_counter()

response = agent.run(user_question)

duration = time.perf_counter() - start

logger.info(
    "agent_request_completed",
    extra={
        "duration_seconds": duration
    }
)

This metric gives you the overall latency.

However, it should not be the only latency measurement.

If an agent takes 10 seconds, developers need to know why.

Break the request into stages.

Total latency
    |
    +-- Agent planning
    +-- Model calls
    +-- SQL generation
    +-- BigQuery execution
    +-- Result processing
    +-- Final response generation

This makes optimization much more practical.

Track BigQuery Query Latency

Every query executed by the agent should be associated with the originating request.

For example:

query_start = time.perf_counter()

job = client.query(sql)
result = job.result()

query_duration = time.perf_counter() - query_start

logger.info(
    "bigquery_query_completed",
    extra={
        "request_id": request_id,
        "query_duration_seconds": query_duration
    }
)

This allows the application to distinguish agent latency from database latency.

Suppose:

Total request: 12 seconds
BigQuery:      9 seconds
Model work:    2 seconds
Application:   1 second

The obvious optimization target is BigQuery.

Without stage-level metrics, developers might incorrectly spend time optimizing the model layer.

Track Bytes Processed

Query duration alone does not tell you the cost of a query.

BigQuery workloads should also be monitored for the amount of data processed.

A query that takes two seconds and processes several gigabytes may require more attention than a query that takes three seconds and processes a few megabytes.

For an agent application, track information such as:

Request ID
Query ID
Bytes processed
Bytes billed
Duration
Status

The BigQuery job metadata provides useful information for this type of monitoring.

A logging record could look conceptually like:

{
  "request_id": "8f42",
  "query_id": "job_123",
  "duration_seconds": 2.4,
  "bytes_processed": 52428800,
  "status": "success"
}

This creates a connection between the user request and the underlying data operation.

Track Query Count Per Agent Request

Query count is another useful metric.

An agent that normally executes one or two queries may suddenly start executing ten queries for the same type of request.

That can indicate:

Poor prompt or tool design
Inefficient SQL generation
Missing schema information
Repeated tool calls
Incorrect agent reasoning
Poor result handling

Track the relationship:

User Request
     |
     +-- Query 1
     +-- Query 2
     +-- Query 3

Then calculate:

queries_per_request

A useful dashboard might show:

Average queries/request: 2.3
P95 queries/request:     5
Maximum:                 14

The maximum value is particularly useful during debugging.

Monitor Generated SQL

The SQL generated by an AI agent should be observable, but logging needs to be handled carefully.

For internal debugging, storing the generated SQL can help identify inefficient queries.

For example:

SELECT *
FROM customer_transactions
WHERE customer_id IN (
    SELECT customer_id
    FROM customers
);

A monitoring system can then identify patterns such as:

SELECT *
Missing filters
Large table scans
Unnecessary joins
Repeated queries
Missing partition filters

However, SQL logs may contain sensitive information depending on how queries are generated.

Do not automatically log user data simply because the SQL is useful for debugging.

Use appropriate redaction and access controls.

Track Query Errors

A successful agent response should not be the only success signal.

An agent may recover from a failed query by generating another SQL statement.

For example:

Query 1
  |
  +-- Error
       |
       v
Agent retries
       |
       v
Query 2
       |
       v
Success

From the user's perspective, the request succeeded.

From an engineering perspective, the first failure is important.

Track:

query_errors
query_retries
agent_recoveries
final_request_status

This can reveal hidden reliability problems.

If 20 percent of successful agent requests require query retries, the system may appear healthy while carrying unnecessary latency and cost.

Measure Query Cost by Request

Cost analysis becomes more useful when it is associated with the user request.

For example:

Request A
2 queries
80 MB processed

Request B
5 queries
12 GB processed

Request C
1 query
25 MB processed

The application can aggregate this information by request, user, feature, or workload type.

This allows questions such as:

Which agent feature consumes the most BigQuery resources?
Which prompts generate the largest scans?
Which users or workloads generate unusual query volume?

The goal is not necessarily to charge users.

The goal is to understand resource consumption.

Use Request Correlation IDs

Every agent request should have a correlation ID.

For example:

import uuid

request_id = str(uuid.uuid4())

That ID should be propagated through the workflow.

Request ID
    |
    +-- Agent execution
    |
    +-- Model call
    |
    +-- SQL generation
    |
    +-- BigQuery job
    |
    +-- Final response

This is one of the most useful design decisions for troubleshooting.

Without correlation, developers may see hundreds of BigQuery jobs and have no easy way to determine which user request generated which query.

With correlation, the relationship becomes searchable.

Build a Useful Observability Dashboard

A dashboard should focus on operational questions rather than simply displaying every available metric.

Useful top-level metrics include:

Metric	Why it matters
Agent request count	Shows workload volume
Success rate	Measures reliability
P50 latency	Shows typical response time
P95 latency	Shows slower user experiences
Queries per request	Detects inefficient agent behavior
Bytes processed	Indicates data-processing volume
Query error rate	Detects SQL or data-access problems
Retry rate	Reveals hidden failures
Cost per request	Helps control AI data workload cost

A dashboard can then provide drill-down views.

For example:

Agent Requests
      |
      v
Slow Requests
      |
      v
BigQuery Queries
      |
      v
Query Details

This is more useful than a dashboard that only shows average latency.

P50 and P95 Latency Matter

Average latency can hide slow requests.

Suppose:

Request 1: 1 sec
Request 2: 1 sec
Request 3: 1 sec
Request 4: 1 sec
Request 5: 20 sec

The average is influenced by the outlier.

P50 represents the typical middle request, while P95 shows the experience of slower requests.

For an AI data agent, monitor both.

For example:

P50: 2.1 seconds
P95: 11.8 seconds

A high P95 may indicate:

Large BigQuery scans
Complex SQL
Multiple agent retries
Slow model responses
Repeated tool calls

This makes percentile-based monitoring more useful than average latency alone.

Detecting Agent Loops

AI agents can sometimes repeat an operation.

For example:

Generate SQL
     |
     v
Query
     |
     v
Result
     |
     v
Generate SQL again
     |
     v
Query again
     |
     v
Repeat

A query-count metric can expose this behavior.

Set operational thresholds.

For example:

MAX_QUERY_COUNT = 5

if query_count >= MAX_QUERY_COUNT:
    logger.warning(
        "agent_query_limit_reached",
        extra={"request_id": request_id}
    )

The exact threshold should depend on the application.

The important part is having a safety boundary.

An agent should not be allowed to execute an unlimited number of database operations for one user request.

Monitor Schema and Query Changes

Agent behavior can change when the underlying data schema changes.

For example, a column may be renamed:

customer_region

to:

region

An agent trained or configured around the old schema may generate invalid SQL.

Monitor query errors by:

table
column
error type
agent version
application version

This can reveal whether a new deployment or schema migration changed agent reliability.

Version the Agent Alongside Observability Data

Agent behavior is influenced by more than source code.

Track versions such as:

Agent version
Prompt version
Model version
Tool version
Schema version
Application version

Then associate those versions with requests.

For example:

{
  "request_id": "8f42",
  "agent_version": "2026.11",
  "prompt_version": "v7",
  "model": "configured-model",
  "schema_version": "v4"
}

This makes it possible to investigate questions such as:

Did query volume increase after the new prompt?
Did latency change after the agent version changed?
Did SQL errors increase after the schema migration?

Without version information, these comparisons become much harder.

Common Mistakes

Measuring only API latency

End-to-end latency does not explain where time was spent.

Track model, tool, and BigQuery stages separately.

Ignoring query count

An agent that executes many small queries may appear acceptable while consuming unnecessary resources.

Logging everything

More logs do not automatically mean better observability.

Avoid collecting sensitive user data or unnecessary query results.

Tracking only failed requests

Successful requests can still be inefficient.

Monitor retries, query counts, bytes processed, and latency.

Using only average latency

P95 and other percentile measurements are important for identifying slow requests.

Not connecting BigQuery jobs to agent requests

Without correlation IDs, troubleshooting becomes much harder.

Best Practices

Define an observability contract

Before building dashboards, define the fields that every agent request should produce.

For example:

request_id
agent_version
prompt_version
request_duration
model_call_count
query_count
query_duration
bytes_processed
query_errors
retry_count
status

Use structured logging

Structured logs make it easier to filter and aggregate data.

For example:

logger.info(
    "agent_request_completed",
    extra={
        "request_id": request_id,
        "query_count": query_count,
        "duration_seconds": duration,
        "status": "success"
    }
)

Set query safety limits

Prevent runaway agent behavior with application-level limits.

For example:

Maximum queries per request
Maximum execution time
Maximum bytes processed
Maximum retry count

The exact values should be based on the workload.

Monitor cost trends

Do not wait for a billing surprise.

Track data processing and workload growth over time.

Test observability during development

Generate intentionally slow queries, failed queries, and repeated tool calls during testing.

Then verify that the monitoring system captures them correctly.

Advantages and Disadvantages

Advantages	Disadvantages
Makes agent behavior measurable	Requires additional instrumentation
Connects AI activity with database usage	Logging can increase storage costs
Helps identify expensive queries	Sensitive data requires careful handling
Improves troubleshooting	Agent behavior can be difficult to classify
Supports performance optimization	Metrics need consistent correlation IDs
Helps detect unexpected query loops	Dashboards require ongoing maintenance

Production Checklist

Before putting a BigQuery data agent into production, verify:

Every request has a correlation ID.
End-to-end latency is measured.
Model and tool operations can be separated.
BigQuery query count is tracked.
Query duration is recorded.
Bytes processed and relevant job metadata are monitored.
Query errors and retries are visible.
Agent versions are recorded.
Prompt or configuration versions are traceable.
Sensitive query information is protected.
Query and retry limits are enforced.
P50 and P95 latency are monitored.
Cost trends are reviewed regularly.
Dashboards can identify individual slow requests.

Summary

A BigQuery data agent should not be monitored like a simple API.

A single user request can trigger multiple model operations, SQL generation steps, BigQuery jobs, retries, and result-processing operations. Looking only at HTTP response time hides much of the behavior that affects reliability and cost.

A useful observability strategy connects the entire request lifecycle through a correlation ID. Track agent latency, model activity, BigQuery query count, query duration, bytes processed, errors, retries, and workload cost.

The most useful dashboards should answer operational questions quickly: which requests are slow, which queries consume the most resources, how often the agent retries, and whether a recent agent or schema change affected behavior.

When these measurements are built into the application from the beginning, AI data workloads become easier to troubleshoot, optimize, and control as usage grows.

AI agentsBigQueryagent observabilityquery latencyquery costdata analyticsGoogle Cloud

Comments

Join the conversation! Your thoughts help the community grow.