Search built for a model, not a browser
Technology
A general search API returns a page of blue links. An agent needs the content behind them: boilerplate stripped, near duplicates collapsed, trimmed to a token budget, ready to drop into a prompt. Tavily does the ranking and the extraction in one call.
We run it in the Quicknode Builder's Guide, where it feeds the research step of a LangGraph pipeline that writes structured Web3 tool profiles into Contentful. That build is where our experience with it comes from, and what this page describes.
Grounding
What a search API for agents has to do
- The fetch and the parse are the actual work
Getting a ranked list of URLs is the easy half. An agent needs page bodies with navigation and boilerplate removed, near identical sources collapsed, and the result cut to fit a context window. Doing that yourself means a crawler, a parser, and a proxy pool you now maintain.
- Retrieved web text is untrusted input
Everything the search step returns is a stranger's writing arriving inside your prompt. Instructions hidden in a page are a live injection path into whatever the agent does next. Treat search output as quoted data with a boundary around it, never as context carrying your own authority.
- You pay per search, and agents search often
Billing is per API call, and an agentic pipeline issues several per unit of work: one to find a source, another to verify it, another after a retry. Cost scales with how many nodes reach for the web, not with how many people use the thing.
In production
The pipeline this page describes
The reason search sits in a pipeline instead of inside a prompt is that a step can be inspected. You can see what came back, cache it, score it, and rerun that one node when it returns nothing useful, without paying for the four nodes after it.
In the Builder's Guide, the search node runs first of five, and it decides whether the four after it have anything true to work with. Cataloguing that a person did by hand, hours at a time, now runs in minutes. Start a conversation.