Is Your LangGraph Agent Actually Moving the Needle, or Just Adding Complexity?

Evaluation Using a News Aggregator System

Every engineering team that has shipped a LangGraph agent in the last year has, at some point, stared at a tracing dashboard and asked the same quiet question: is this thing actually worth it?

Agents are seductive. They promise to replace brittle pipelines with "reasoning." They let you draw beautiful state-machine diagrams. They give you something to demo. But they also introduce non-determinism, latency, cost multipliers, failure modes you can't unit-test, and a debugging surface area that grows faster than your feature set.

The question isn't "can we build an agent?" — it's "does the agent produce measurably better business outcomes than the dumbest thing that could possibly work?"

This article gives you a concrete evaluation framework, then walks through it end-to-end on a real-time news aggregator system built in LangGraph. You'll see the naive pipeline, the agent upgrade, the metrics, the places where we killed the agent, and the decision checklist I now use before approving any agentic feature.

Part 1 — The Evaluation Framework

Before writing a single node, run your proposed agent through these five tests. If it fails more than one, you probably don't need an agent.

Test 1: The Dumb Baseline Test

Build the simplest possible non-agentic version first. A sequential pipeline. Rule-based routing. A single LLM call. Measure its business metrics. This is your bar. If the agent can't beat it by a margin that justifies its complexity, ship the pipeline.

Test 2: The Complexity Tax Audit

Quantify what the agent costs beyond tokens:

• Latency: p50 and p95 end-to-end

• Cost per request: sum of all LLM calls, tool calls, retries

• Failure modes: list every way the graph can get stuck (infinite loops, tool errors, malformed state)

• Observability cost: can you reproduce a bad run from a trace?

• Maintenance cost: how many engineers understand the graph?

Test 3: The Failure Mode Inventory

For every node, ask: what happens when this node is wrong? In a pipeline, a wrong node produces a wrong output. In an agent, a wrong node can send the graph into a loop, burn $0.40 in tokens, and return a confidently wrong answer. Catalog these.

Test 4: The Business Outcome Link

Map every agent capability to a business KPI. "Better summarization" is not a KPI. "10% increase in daily-active reading time" is. If you can't draw the line, you're building for engineers, not users.

Test 5: The Reversibility Test

Can you A/B test this? Can you roll it back in under 5 minutes? Can you disable a single node without taking down the graph? If no, you're building a monolith with extra steps.

Part 2 — The Use Case: Real-Time News Aggregator

The product

We're building Pulse, a personalized news briefing app. Users get:

1. A rolling feed of top stories, deduplicated across 40+ sources

2. Per-user personalization (tech, finance, sports weights)

3. Breaking-news detection with sub-60-second alerts

4. A daily 5-bullet morning briefing, written in the user's voice preference

Why this is a great test case

A news aggregator looks simple but has four decision points where a pipeline struggles:

Decision point	Pipeline approach	Where it breaks
Source routing	Static rules per source	New sources need code deploys
Deduplication	URL + title similarity	Same story, different angles → duplicates
Breaking news	Keyword triggers	Misses nuanced breaking stories
Personalization	User-tag matching	Doesn't understand why a story matters to a user

These are exactly the places where an LLM with routing and state pays off — if we can prove it.

Part 3 — The Dumb Baseline (v0)

Before any agent, we shipped this:

v0 metrics (10k DAU, 4 weeks):

• Feed CTR: 14.2%

• Duplicate rate (user-reported): 8.3%

• Breaking-news alert latency (p50): 4m 20s

• Morning briefing open rate: 31%

• Cost per active user per day: $0.004

• p95 feed latency: 380ms

This is our bar. Everything below must beat it.

Part 4 — The LangGraph Agent (v1)

Architecture

The key agentic move: the is_breaking router decides, per article, whether to take the fast path (one small LLM call) or the full pipeline. That's the only real "reasoning" — and it's the only place where a static pipeline genuinely fails.

The code

from typing import TypedDict, Literal, Annotated
from langgraph.graph import StateGraph, START, END
from langchain_openai import ChatOpenAI
import operator

# ---------- State ----------
class ArticleState(TypedDict):
    article: dict
    is_breaking: bool
    category: str
    cluster_id: str | None
    user_relevance: dict          # per-user score
    summary: str
    path_taken: Annotated[list, operator.add]
    error: str | None

# ---------- Nodes ----------
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

def is_breaking_node(state: ArticleState) -> dict:
    """Cheap classifier: is this a breaking-news event?"""
    resp = llm.invoke(
        f"Is this article reporting a breaking event in the last 30 min? "
        f"Answer YES or NO only.\n\nTitle: {state['article']['title']}\n"
        f"Lead: {state['article'].get('lead','')[:300]}"
    )
    return {"is_breaking": "YES" in resp.content.upper(),
            "path_taken": ["is_breaking"]}

def fast_path(state: ArticleState) -> dict:
    """Minimal summary for alerts — keep latency under 30s."""
    resp = llm.invoke(
        f"One-sentence breaking-news alert, neutral tone:\n{state['article']['title']}"
    )
    return {"summary": resp.content, "path_taken": ["fast_path"]}

def classify_dedupe(state: ArticleState) -> dict:
    """Semantic category + cluster against recent articles."""
    # In prod: vector lookup against last 24h embeddings
    resp = llm.invoke(
        f"Category (one of: politics, tech, finance, sports, world, other) "
        f"and a 6-word cluster label for dedup.\n\n{state['article']['title']}"
    )
    category, cluster = resp.content.split("\n")[0], resp.content.split("\n")[1]
    return {"category": category, "cluster_id": cluster,
            "path_taken": ["classify_dedupe"]}

def personalize(state: ArticleState) -> dict:
    """Per-user relevance — cached by (user_id, cluster_id)."""
    user = state["article"]["_user"]
    # Cache lookup omitted for brevity
    resp = llm.invoke(
        f"User cares about: {user['interests']}. "
        f"Rate relevance 0-10 and give one reason why.\n"
        f"Article: {state['article']['title']}"
    )
    return {"user_relevance": {"score": int(resp.content[:2]),
                               "reason": resp.content[3:]},
            "path_taken": ["personalize"]}

def summarize(state: ArticleState) -> dict:
    resp = llm.invoke(
        f"3-bullet summary, {state['article'].get('_voice','neutral')} voice:\n"
        f"{state['article']['content'][:2000]}"
    )
    return {"summary": resp.content, "path_taken": ["summarize"]}

# ---------- Router ----------
def route_breaking(state: ArticleState) -> Literal["fast_path", "classify_dedupe"]:
    return "fast_path" if state["is_breaking"] else "classify_dedupe"

# ---------- Graph ----------
g = StateGraph(ArticleState)
g.add_node("is_breaking", is_breaking_node)
g.add_node("fast_path", fast_path)
g.add_node("classify_dedupe", classify_dedupe)
g.add_node("personalize", personalize)
g.add_node("summarize", summarize)

g.add_edge(START, "is_breaking")
g.add_conditional_edges("is_breaking", route_breaking)
g.add_edge("fast_path", END)
g.add_edge("classify_dedupe", "personalize")
g.add_edge("personalize", "summarize")
g.add_edge("summarize", END)

graph = g.compile()

Two things worth calling out:

1. The graph is deliberately shallow. Four nodes max on any path. Every time I've seen teams build 12-node LangGraphs, the debugging cost exceeds the benefit.

2. The "agent" is really just a router. That's the honest version of what most production agents are. If your graph has no conditional edges, it's a pipeline wearing a trench coat

Part 5 — What We Actually Measured

We ran v0 and v1 in parallel for 6 weeks on 50/50 traffic split (10k users each). Here's what happened:

Metric	v0 (pipeline)	v1 (LangGraph)	Δ
Feed CTR	14.2%	16.8%	+18%
Duplicate rate	8.3%	2.1%	−75%
Breaking-news alert p50 latency	4m 20s	22s	−92%
Morning briefing open rate	31%	39%	+26%
Briefing "save for later" rate	4.1%	7.8%	+90%
Cost / active user / day	$0.004	$0.021	+425%
p95 feed latency	380ms	1,420ms	+274%
Incidents (graph stuck / timeout)	0	14	—
Engineer-hours to ship a new source	6h	1.5h	−75%

The honest read

Wins that justify the agent

• Breaking-news latency dropped from minutes to seconds. This is the one place where the router genuinely replaces something a pipeline can't do well (semantic "is this breaking?" judgment).

• Deduplication improved dramatically because semantic clustering caught same-story-different-angle cases.

• Briefing quality drove real engagement lifts.

Costs we had to own

• 5× cost per user. We offset this by caching personalize aggressively (hit rate ~72%) and using gpt-4o-mini everywhere.

• p95 latency tripled. We fixed this by streaming summaries and moving personalize to a background job for non-breaking paths.

• 14 incidents in 6 weeks. Most were the graph getting stuck when the LLM returned malformed output. We added a retry-with-schema node and a hard 3-attempt circuit breaker.

The part we killed

Here's the bit most agent write-ups leave out: we reverted two nodes back to the pipeline after 3 weeks.

• classify was moved back to the fine-tuned BERT classifier. The LLM was ~3% more accurate but 8× slower and 12× more expensive. For a category label, that trade was absurd.

• The initial dedupe pre-filter went back to Jaccard + embedding similarity. We only invoke the LLM for "maybe duplicates" (similarity 0.55–0.75). This cut LLM dedup calls by 88%.

The final graph is hybrid: LLM where it clearly wins (breaking judgment, semantic dedup tie-breaks, summarization, personalization), deterministic code everywhere else. That's the right answer 90% of the time.

Part 6 — The Decision Checklist

Before you approve your next LangGraph feature, run this. Every box needs a real answer, not a hope.

Do you actually need an agent?

• I built the dumb pipeline first and measured it

• The agent beats the pipeline on a business KPI, not just a vibe

• The graph has at least one conditional edge that a rule can't replace

• I can name the specific failure mode the pipeline had that the agent fixes

Can you afford the agent?

• Cost per request at p95 is within 5× the pipeline

• p95 latency is within your UX budget

• You have tracing on every node (LangSmith, Phoenix, or equivalent)

• You can replay any production run from its trace

• You have a circuit breaker and a max-iteration limit

Can you operate it?

• You can A/B test it against the pipeline

• You can roll back in < 5 minutes

• You can disable any single node without breaking the graph

• An on-call engineer who didn't write it can debug it

Is the graph honest?

• It's ≤ 6 nodes deep on any path

• Every LLM call has a structured output schema

• Every LLM call has a retry-with-fallback

• You're not using an agent where a function call would do

If you can't check at least 10 of these, you're not shipping an agent — you're shipping a science project.

Part 7 — Lessons from 6 Months of Running This

A few things I wish someone had told me before we started:

1. The router is the whole game. The quality of your agent is determined by the quality of its routing decisions. Spend 70% of your eval effort on the router. In our case, is_breaking accuracy of 94% was the single biggest lever on breaking-news latency.

2. Cache like your burn rate depends on it — because it does. The personalize node was going to bankrupt us until we cached by (user_id, cluster_id) with a 2-hour TTL. Same cluster ≈ same relevance. 72% hit rate, negligible staleness.

3. Treat LLM outputs as untrusted input. Every node should validate its own output against a schema and have a fallback. The 14 incidents we had? All of them were "LLM returned something the next node couldn't parse."

4. Measure the pipeline you replaced, forever. We still run the v0 pipeline in shadow mode on 5% of traffic. It's our canary for when the agent starts drifting. You should do the same.

5. The best agent is the smallest agent. Our first draft had 9 nodes. We cut it to 5 by moving three back to deterministic code. The result was faster, cheaper, more reliable, and easier to reason about. Complexity is a liability, not a feature.

Conclusion

LangGraph is a genuinely useful tool, but it's not a product strategy. The question is never "should we use an agent?" — it's "does this specific agent, on this specific task, produce a measurable business outcome that justifies its complexity tax?"

For our news aggregator, the answer was yes, but only in three places: breaking-news routing, semantic deduplication tie-breaks, and personalized summarization. Everywhere else, the pipeline won.

Ship the agent where it earns its keep. Kill it everywhere else. And whatever you do, measure the dumb baseline first — because the dumb baseline is almost always better than you think, and the agent has to beat it, not just look impressive in a demo.