Use case · AI & agents
X/Twitter data your AI agents can actually use
Feed current X/Twitter posts into RAG, agents, and LLM pipelines. xfetch returns normalized JSON with deduped authors and one-call workflow endpoints — so you skip the glue code and the scraper maintenance.
GET /v1/search/recent/enriched?query=ai agents&limit=20 Authorization: Bearer <api_key>
{
"data": {
"tweets": [ { "id", "text", "author_id" } … ],
"authors": [ { "id", "username", "verified" } … ]
},
"meta": { "credits": { "charged": 41 } }
}The job
Most AI features need fresh, structured public data — and X/Twitter is where a lot of it surfaces first. The slow part is never the model; it is normalizing raw responses, deduping authors, and keeping a scraper alive. xfetch hands your pipeline clean records so you can spend time on the product.
RAG & agent builders
Feed current X/Twitter posts into retrieval, agents, and tools without writing a normalizer first.
LLM application teams
Ground summaries, classifiers, and chat features on fresh public X data with stable JSON.
Data & ML engineers
Pull deduped tweets and authors on a schedule for embeddings, training sets, and analytics.
AI research
Collect tweet context — author, quotes, retweeters, threads — as structured evidence for analysis.
One call for full tweet context
Summarizers and moderation agents need more than a single tweet. One call returns the tweet, its author, its quotes, and its retweeters as named blocks — no multi-call stitching.
GET /v1/tweets/1234567890/context Authorization: Bearer <api_key>
{
"data": {
"tweet": { … },
"author": { … },
"quotes": [ … ],
"retweeters": [ … ]
},
"meta": { "credits": { "charged": 3 } }
}A contract your agent can read
Point a coding agent at these and it can integrate without hand-holding — the API describes itself.
OpenAPI schema
Every /v1 and /2 endpoint, typed — point your codegen or coding agent at it.
/openapi.jsonFull LLM context
The long-form context: endpoints, pricing rules, and examples in one file.
/llms-full.txtGetting X data into an AI pipeline
| Dimension | Official X API | Scrapers & DIY | xfetch |
|---|---|---|---|
| Output shape | Official envelopes, assembled across calls | Raw, page-shaped payloads you normalize | Normalized JSON with named blocks |
| Author data | Manual join via expansions | Usually none — you dedupe | Deduped authors[] included |
| Glue code to ingest | Pagination, join, and retry yourself | Parsing, dedupe, and drift fixes | One call returns ingestion-ready records |
| Machine-readable contract | OpenAPI for the full API | Varies; often none | /openapi.json + /llms.txt for agents |
| Pricing model | Tiered contracts | Per-request or per-result | Credits — 1 base plus returned items |
| Maintenance | Track API changes | Fix breakage when pages change | Stable contract, opaque tokens |
What ingestion costs
Pulling 20 enriched tweets every hour is about 29,520 credits a month (41 credits per poll × 24 × 30). New accounts start with 1,000 free credits, and failed or rate-limited calls are never charged.
Copy-paste starter
A dependency-free TypeScript starter: query enriched search and emit one normalized record per tweet, ready for a vector store.
/**
* Minimal X/Twitter -> RAG ingestion with xfetch. No dependencies.
* Run: XFETCH_API_KEY=sk_live_... npx tsx ingest.ts
*/
const API = "https://api.xfetch.io";
const KEY = process.env.XFETCH_API_KEY;
const QUERY = 'ai agents OR "llm app" OR rag';
async function main() {
if (!KEY) throw new Error("Set XFETCH_API_KEY");
const url = new URL("/v1/search/recent/enriched", API);
url.searchParams.set("query", QUERY);
url.searchParams.set("limit", "20");
const res = await fetch(url, { headers: { Authorization: `Bearer ${KEY}` } });
if (!res.ok) throw new Error(`xfetch ${res.status}`);
const body = await res.json();
const data = body.data ?? { tweets: [], authors: [] };
const authors = new Map();
for (const a of data.authors) authors.set(a.id, a);
// One normalized record per tweet — ready for embeddings or a vector store.
const records = data.tweets.map((t) => ({
id: t.id,
text: t.text,
author: authors.get(t.author_id)?.username ?? t.author_id,
created_at: t.created_at
}));
console.log(JSON.stringify(records, null, 2));
console.log("Credits charged:", body.meta?.credits?.charged);
}
main().catch((e) => { console.error(e); process.exit(1); });
Boundaries
Read-only
xfetch reads public data. It does not post, like, repost, follow, or DM.
Normalized, not raw
You get clean JSON and opaque pagination tokens — never raw upstream payloads or cursors.
Bring your own model
xfetch returns structured data; embeddings, retrieval, and generation run in your pipeline.
Related
FAQ
- How do I get X/Twitter data into a RAG or LLM pipeline?
- Call GET /v1/search/recent/enriched for a query and you get matched tweets plus a deduped authors[] array as normalized JSON — drop the records straight into embeddings or a vector store. No client-side join and no scraper to maintain.
- Does xfetch return normalized JSON for LLMs and agents?
- Yes. /v1 returns clean, named blocks — tweets, authors, and workflow responses such as { user, recent_tweets } — so agents reason over the data instead of parsing raw payloads.
- Can an AI coding agent discover the xfetch API automatically?
- Yes. /openapi.json describes every endpoint, and /llms.txt plus /llms-full.txt give a coding agent a machine-readable map of the API, its pricing rules, and examples to read before integrating.
- How much does it cost to ingest tweets for AI?
- You pay in credits per returned item: enriched search is a small base plus a per-tweet credit, and single-object lookups are 1 credit. New accounts start with free credits to try it, and failed or rate-limited calls are never charged. See /pricing for the full table.
- Can I use xfetch with my agent framework?
- xfetch is a standard HTTP API with bearer-token auth, so you can call it from any language, agent framework, or MCP-capable client that can reach an HTTP endpoint. /openapi.json and /llms.txt help coding agents wire it up.
- Is this scraping, and do credits expire?
- You code against xfetch's stable, provider-neutral API contract, not a brittle scraper — responses are normalized and pagination tokens are opaque. Free and pay-as-you-go credits do not expire; monthly-plan credits renew each period.