Evaluation Using a News Aggregator System

Every engineering team that has shipped a LangGraph agent in the last year has, at some point, stared at a tracing dashboard and asked the same quiet question: is this thing actually worth it?

Agents are seductive. They promise to replace brittle pipelines with "reasoning." They let you draw beautiful state-machine diagrams. They give you something to demo. But they also introduce non-determinism, latency, cost multipliers, failure modes you can't unit-test, and a debugging surface area that grows faster than your feature set.

The question isn't "can we build an agent?" — it's "does the agent produce measurably better business outcomes than the dumbest thing that could possibly work?"

This article gives you a concrete evaluation framework, then walks through it end-to-end on a real-time news aggregator system built in LangGraph. You'll see the naive pipeline, the agent upgrade, the metrics, the places where we killed the agent, and the decision checklist I now use before approving any agentic feature.

Part 1 — The Evaluation Framework

Before writing a single node, run your proposed agent through these five tests. If it fails more than one, you probably don't need an agent.

Test 1: The Dumb Baseline Test

Build the simplest possible non-agentic version first. A sequential pipeline. Rule-based routing. A single LLM call. Measure its business metrics. This is your bar. If the agent can't beat it by a margin that justifies its complexity, ship the pipeline.

Test 2: The Complexity Tax Audit

Quantify what the agent costs beyond tokens:

Test 3: The Failure Mode Inventory

For every node, ask: what happens when this node is wrong? In a pipeline, a wrong node produces a wrong output. In an agent, a wrong node can send the graph into a loop, burn $0.40 in tokens, and return a confidently wrong answer. Catalog these.

Test 4: The Business Outcome Link

Map every agent capability to a business KPI. "Better summarization" is not a KPI. "10% increase in daily-active reading time" is. If you can't draw the line, you're building for engineers, not users.

Test 5: The Reversibility Test

Can you A/B test this? Can you roll it back in under 5 minutes? Can you disable a single node without taking down the graph? If no, you're building a monolith with extra steps.

Part 2 — The Use Case: Real-Time News Aggregator

The product

We're building Pulse, a personalized news briefing app. Users get:

  1. A rolling feed of top stories, deduplicated across 40+ sources

  2. Per-user personalization (tech, finance, sports weights)

  3. Breaking-news detection with sub-60-second alerts

  4. A daily 5-bullet morning briefing, written in the user's voice preference

Why this is a great test case

A news aggregator looks simple but has four decision points where a pipeline struggles:

Decision pointPipeline approachWhere it breaks
Source routingStatic rules per sourceNew sources need code deploys
DeduplicationURL + title similaritySame story, different angles → duplicates
Breaking newsKeyword triggersMisses nuanced breaking stories
PersonalizationUser-tag matchingDoesn't understand why a story matters to a user

These are exactly the places where an LLM with routing and state pays off — if we can prove it.

Part 3 — The Dumb Baseline (v0)

Before any agent, we shipped this:

46

v0 metrics (10k DAU, 4 weeks):

This is our bar. Everything below must beat it.

Part 4 — The LangGraph Agent (v1)

Architecture

45

The key agentic move: the is_breaking router decides, per article, whether to take the fast path (one small LLM call) or the full pipeline. That's the only real "reasoning" — and it's the only place where a static pipeline genuinely fails.

The code

from typing import TypedDict, Literal, Annotated
from langgraph.graph import StateGraph, START, END
from langchain_openai import ChatOpenAI
import operator

# ---------- State ----------
class ArticleState(TypedDict):
    article: dict
    is_breaking: bool
    category: str
    cluster_id: str | None
    user_relevance: dict          # per-user score
    summary: str
    path_taken: Annotated[list, operator.add]
    error: str | None

# ---------- Nodes ----------
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

def is_breaking_node(state: ArticleState) -> dict:
    """Cheap classifier: is this a breaking-news event?"""
    resp = llm.invoke(
        f"Is this article reporting a breaking event in the last 30 min? "
        f"Answer YES or NO only.\n\nTitle: {state['article']['title']}\n"
        f"Lead: {state['article'].get('lead','')[:300]}"
    )
    return {"is_breaking": "YES" in resp.content.upper(),
            "path_taken": ["is_breaking"]}

def fast_path(state: ArticleState) -> dict:
    """Minimal summary for alerts — keep latency under 30s."""
    resp = llm.invoke(
        f"One-sentence breaking-news alert, neutral tone:\n{state['article']['title']}"
    )
    return {"summary": resp.content, "path_taken": ["fast_path"]}

def classify_dedupe(state: ArticleState) -> dict:
    """Semantic category + cluster against recent articles."""
    # In prod: vector lookup against last 24h embeddings
    resp = llm.invoke(
        f"Category (one of: politics, tech, finance, sports, world, other) "
        f"and a 6-word cluster label for dedup.\n\n{state['article']['title']}"
    )
    category, cluster = resp.content.split("\n")[0], resp.content.split("\n")[1]
    return {"category": category, "cluster_id": cluster,
            "path_taken": ["classify_dedupe"]}

def personalize(state: ArticleState) -> dict:
    """Per-user relevance — cached by (user_id, cluster_id)."""
    user = state["article"]["_user"]
    # Cache lookup omitted for brevity
    resp = llm.invoke(
        f"User cares about: {user['interests']}. "
        f"Rate relevance 0-10 and give one reason why.\n"
        f"Article: {state['article']['title']}"
    )
    return {"user_relevance": {"score": int(resp.content[:2]),
                               "reason": resp.content[3:]},
            "path_taken": ["personalize"]}

def summarize(state: ArticleState) -> dict:
    resp = llm.invoke(
        f"3-bullet summary, {state['article'].get('_voice','neutral')} voice:\n"
        f"{state['article']['content'][:2000]}"
    )
    return {"summary": resp.content, "path_taken": ["summarize"]}

# ---------- Router ----------
def route_breaking(state: ArticleState) -> Literal["fast_path", "classify_dedupe"]:
    return "fast_path" if state["is_breaking"] else "classify_dedupe"

# ---------- Graph ----------
g = StateGraph(ArticleState)
g.add_node("is_breaking", is_breaking_node)
g.add_node("fast_path", fast_path)
g.add_node("classify_dedupe", classify_dedupe)
g.add_node("personalize", personalize)
g.add_node("summarize", summarize)

g.add_edge(START, "is_breaking")
g.add_conditional_edges("is_breaking", route_breaking)
g.add_edge("fast_path", END)
g.add_edge("classify_dedupe", "personalize")
g.add_edge("personalize", "summarize")
g.add_edge("summarize", END)

graph = g.compile()

Two things worth calling out:

  1. The graph is deliberately shallow. Four nodes max on any path. Every time I've seen teams build 12-node LangGraphs, the debugging cost exceeds the benefit.

  2. The "agent" is really just a router. That's the honest version of what most production agents are. If your graph has no conditional edges, it's a pipeline wearing a trench coat

47

Part 5 — What We Actually Measured

We ran v0 and v1 in parallel for 6 weeks on 50/50 traffic split (10k users each). Here's what happened:

Metricv0 (pipeline)v1 (LangGraph)Δ
Feed CTR14.2%16.8%+18%
Duplicate rate8.3%2.1%−75%
Breaking-news alert p50 latency4m 20s22s−92%
Morning briefing open rate31%39%+26%
Briefing "save for later" rate4.1%7.8%+90%
Cost / active user / day$0.004$0.021+425%
p95 feed latency380ms1,420ms+274%
Incidents (graph stuck / timeout)014—
Engineer-hours to ship a new source6h1.5h−75%

The honest read

Wins that justify the agent

Costs we had to own

The part we killed

Here's the bit most agent write-ups leave out: we reverted two nodes back to the pipeline after 3 weeks.

The final graph is hybrid: LLM where it clearly wins (breaking judgment, semantic dedup tie-breaks, summarization, personalization), deterministic code everywhere else. That's the right answer 90% of the time.

Part 6 — The Decision Checklist

Before you approve your next LangGraph feature, run this. Every box needs a real answer, not a hope.

Do you actually need an agent?

Can you afford the agent?

Can you operate it?

Is the graph honest?

If you can't check at least 10 of these, you're not shipping an agent — you're shipping a science project.

Part 7 — Lessons from 6 Months of Running This

A few things I wish someone had told me before we started:

1. The router is the whole game. The quality of your agent is determined by the quality of its routing decisions. Spend 70% of your eval effort on the router. In our case, is_breaking accuracy of 94% was the single biggest lever on breaking-news latency.

2. Cache like your burn rate depends on it — because it does. The personalize node was going to bankrupt us until we cached by (user_id, cluster_id) with a 2-hour TTL. Same cluster ≈ same relevance. 72% hit rate, negligible staleness.

3. Treat LLM outputs as untrusted input. Every node should validate its own output against a schema and have a fallback. The 14 incidents we had? All of them were "LLM returned something the next node couldn't parse."

4. Measure the pipeline you replaced, forever. We still run the v0 pipeline in shadow mode on 5% of traffic. It's our canary for when the agent starts drifting. You should do the same.

5. The best agent is the smallest agent. Our first draft had 9 nodes. We cut it to 5 by moving three back to deterministic code. The result was faster, cheaper, more reliable, and easier to reason about. Complexity is a liability, not a feature.

Conclusion

LangGraph is a genuinely useful tool, but it's not a product strategy. The question is never "should we use an agent?" — it's "does this specific agent, on this specific task, produce a measurable business outcome that justifies its complexity tax?"

For our news aggregator, the answer was yes, but only in three places: breaking-news routing, semantic deduplication tie-breaks, and personalized summarization. Everywhere else, the pipeline won.

Ship the agent where it earns its keep. Kill it everywhere else. And whatever you do, measure the dumb baseline first — because the dumb baseline is almost always better than you think, and the agent has to beat it, not just look impressive in a demo.