Web data for AI agents

The Apify MCP server lets AI apps use Apify tools. Your AI assistant can search for a tool, run it and read the results. Your agent can use our live tools on Apify. Tools marked Coming soon will be added after release.

This page explains web scraping for AI agents in simple words. You will learn what MCP is, how to connect it, and how to keep costs safe.

• Claude
• OpenAI
• Cursor
• LangChain
• CrewAI
• Dify
• Langflow
• Python
• TypeScript

Works with any app that supports MCP or an HTTP API. The names and logos belong to their owners. We are not partners of these companies.

Why AI agents need fresh web data

An AI agent is an AI model that can use tools. It does not only write text. It can search, open pages, call APIs and save files.

A large language model (LLM) learns from old data. It does not know today's prices, new jobs or this week's news. It can also make up facts. Fresh data helps the agent find recent facts. The agent can still make mistakes, so check important results.

With live data, your agent can:

• Read the newest reviews of a product before it writes a summary.
• Find new job ads at 50 companies and sort them by role.
• Turn a docs website into Markdown and answer questions about it.
• Check a list of websites for contact details or tech stack.
• Get a YouTube transcript and turn it into notes.

Our tools return structured data. Structured data means clear fields, like title, price and date. LLMs work much better with clean fields than with raw HTML.

What is MCP

MCP means Model Context Protocol. It is an open standard. It lets AI apps connect to outside tools in one common way.

Think of MCP as a USB port for AI. One plug works with many devices.

Without MCP, every AI app needs its own plugin for every service. With MCP, one server works in many AI apps.

Connect the Apify MCP server

Apify runs a hosted MCP server at https://mcp.apify.com. "Hosted" means you do not install anything. You add the URL to your AI app.

Option 1: hosted server with OAuth (easiest)

1. Open your AI app's settings for MCP servers or connectors.
2. Add a new remote server with the URL https://mcp.apify.com.
3. Your browser opens. Sign in to Apify and approve access.
4. Done. The agent can now search and run Actors.

OAuth is a safe sign-in method. You approve access in the browser. You do not paste a token into a file.

Option 2: hosted server with an API token

Some apps ask for a header instead. Use your Apify API token:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com",
      "headers": {
        "Authorization": "Bearer YOUR_APIFY_TOKEN"
      }
    }
  }
}

Find your token in Apify Console under Settings → API & Integrations. Keep it private.

Option 3: run the server on your computer

You can also run the server locally with Node.js (version 18 or newer):

{
  "mcpServers": {
    "apify": {
      "command": "npx",
      "args": ["-y", "@apify/actors-mcp-server"],
      "env": { "APIFY_TOKEN": "YOUR_APIFY_TOKEN" }
    }
  }
}

Add our tools directly

By default, the server gives your agent tools to search the Apify Store, read Actor details and call an Actor. Your agent can find our tools by itself.

You can also load exact tools with the tools setting in the URL:

https://mcp.apify.com?tools=cliqtomedia/youtube-transcript-scraper,cliqtomedia/website-contacts-scraper

Then the agent sees these tools right away, with their input fields. This is faster and more reliable.

Read the full guide in the Apify MCP docs.

Claude, ChatGPT and Cursor

Many popular AI apps can work with MCP. Features change fast, so check your app's own help pages.

For coding agents like Claude Code, Cursor and GitHub Copilot, Apify also offers a plugin and agent skills. See the Apify docs for the latest setup.

Example prompts

Try prompts like these after you connect:

• "Use the cliqtomedia Greenhouse jobs tool. Get all open jobs from these 5 boards. Show only data jobs."
• "Turn this docs site into Markdown with the cliqtomedia website-to-markdown tool. Limit it to 30 pages."
• "Check these 200 emails with the cliqtomedia email verifier. Give me only the deliverable ones."
• "Get the last 100 reviews of this app from Google Play. Group the complaints by topic."

LangChain and other frameworks

You can also use our tools in your own AI app. A framework is a code library for building AI apps.

A short LangChain example:

# pip install langchain-apify
# set APIFY_TOKEN in your environment
from langchain_apify import ApifyActorsTool

tool = ApifyActorsTool("cliqtomedia/youtube-transcript-scraper")
result = tool.invoke({"run_input": {"videoUrls": ["jNQXAC9IVRw"]}})
print(result)

You can give this tool to a LangChain agent. The agent decides when to call it. See also Integrations.

LLM-ready output: Markdown and JSON

LLM-ready data is data an AI model can read easily. It has no menus, ads or cookie banners. It is short, clean and clear.

Our tools give two main formats:

• Markdown. Plain text with simple marks for headings, lists and tables. Great for reading, summaries and RAG.
• JSON. Data with named fields. Great for tables, filters and function calls.

What is RAG web data

RAG means retrieval-augmented generation. The AI first finds useful text. Then it uses this text to answer. RAG web data is web text that you collect for this.

A simple RAG flow:

PDF to Markdown for LLMs & RAG Chunks - No Login (coming soon) will return ready chunks with page numbers and token counts. A token is a small piece of a word. LLMs count text in tokens.

Example output

Here is a short example row from Fast Webpage to Markdown API for AI Agents (coming soon; the live Website to Markdown for RAG: Crawl & Changed Pages returns the same kind of Markdown):

{
  "url": "https://example.com/docs/start",
  "title": "Getting started",
  "statusCode": 200,
  "wordCount": 412,
  "markdown": "# Getting started\n\nInstall the app with one command..."
}

Field names differ a little in each tool. Each tool page shows its output fields.

Tools that work well with agents

These tools have short inputs and clean output. They are a good start for AI agents and MCP.

Read the web as text

AI & documentsComing soon

Fast Webpage to Markdown API for AI Agents

Give a link and get the page as clean Markdown in a few seconds.

AI & documentsLive on Apify

Website to Markdown for RAG: Crawl & Changed Pages

Give a start URL and a page limit.

AI & documentsLive on Apify

RAG Web Search: Google Results as Clean Markdown

Type a question or topic and get the top Google pages as clean text.

AI & documentsComing soon

PDF to Markdown for LLMs & RAG Chunks - No Login

Paste a PDF link and get Markdown with real headings, lists and tables.

AI & documentsLive on Apify

PDF & Image to Text OCR for AI Agents - No Login

Give a link to a PDF or an image.

AI & documentsLive on Apify

News Article Extractor: RSS, URLs & Clean Text

Paste news links or a news site.

Video and audio to text

Video & audioLive on Apify

YouTube Subtitles & Transcript Scraper API

Get the public captions of a YouTube video as full text and timed parts.

Video & audioComing soon

Video URL to Text: Transcribe TikTok, Reels, X, YouTube

Paste a video link and get the spoken words as text.

Video & audioLive on Apify

Audio & Podcast Transcriber (Speech to Text)

Turn audio files or podcast episodes into text.

Research and checks

Leads & companiesLive on Apify

Website Contact Scraper

Add a list of company websites.

Leads & companiesLive on Apify

Website Tech Stack Detector: CMS, Versions & CDN, No Login

Paste a list of websites. The tool checks each live page against 7,600+ technology fingerprints. You get one row per site with CMS, shop…

Leads & companiesLive on Apify

Bulk Email Verifier: MX, Disposable, Role & Typo Check

Paste email addresses or pick a dataset from a lead scraper.

Search & SEOLive on Apify

Google Autocomplete Scraper

Get the phrases Google suggests when people start typing.

Social mediaLive on Apify

Bluesky Scraper — Posts, Profiles, Feeds & Threads, No Login

Get public Bluesky posts, profiles, threads and feeds as clean rows.

Reviews & appsLive on Apify

Google Play Reviews Scraper

Export written Google Play reviews for your apps.

See all of them in AI-ready content & documents and in the full catalog.

Keep costs under control

Agents can start many runs fast. A loop or a big list can cost more than you expect. Use these safety steps.

Read more on our Pricing page.

Questions

Last updated: 16 October 2026