# Web data for AI agents

> Use our tools from AI agents. Connect Claude, ChatGPT, Cursor or LangChain to the Apify MCP server. Get clean, LLM-ready web data for RAG.

URL: https://cliqtomedia.com/actors/ai-agents/

The **Apify MCP server** lets AI apps use Apify tools. Your AI assistant can search for a tool, run it and read the results. Your agent can use our live tools on Apify. Tools marked Coming soon will be added after release.

This page explains web scraping for AI agents in simple words. You will learn what MCP is, how to connect it, and how to keep costs safe.

Claude, OpenAI, Cursor, LangChain, CrewAI, Dify, Langflow, Python, TypeScript

Works with any app that supports MCP or an HTTP API. The names and logos belong to their owners. We are not partners of these companies.

## Why AI agents need fresh web data

An **AI agent** is an AI model that can use tools. It does not only write text. It can search, open pages, call APIs and save files.

A **large language model** (LLM) learns from old data. It does not know today's prices, new jobs or this week's news. It can also make up facts. Fresh data helps the agent find recent facts. The agent can still make mistakes, so check important results.

With live data, your agent can:

- Read the newest reviews of a product before it writes a summary.
- Find new job ads at 50 companies and sort them by role.
- Turn a docs website into Markdown and answer questions about it.
- Check a list of websites for contact details or tech stack.
- Get a YouTube transcript and turn it into notes.

Our tools return **structured data**. Structured data means clear fields, like `title`, `price` and `date`. LLMs work much better with clean fields than with raw HTML.

## What is MCP

**MCP** means **Model Context Protocol**. It is an open standard. It lets AI apps connect to outside tools in one common way.

Think of MCP as a USB port for AI. One plug works with many devices.

| Part | What it is | Example |
|---|---|---|
| **MCP client** | The AI app you use | Claude, Cursor, VS Code |
| **MCP server** | A service that offers tools | The Apify MCP server |
| **Tool** | One action the AI can call | "Run the Reddit scraper" |

Without MCP, every AI app needs its own plugin for every service. With MCP, one server works in many AI apps.

## Connect the Apify MCP server
Apify runs a hosted MCP server at `https://mcp.apify.com`. "Hosted" means you do not install anything. You add the URL to your AI app.

### Option 1: hosted server with OAuth (easiest)

1. Open your AI app's settings for MCP servers or connectors.
2. Add a new remote server with the URL `https://mcp.apify.com`.
3. Your browser opens. Sign in to Apify and approve access.
4. Done. The agent can now search and run Actors.

OAuth is a safe sign-in method. You approve access in the browser. You do not paste a token into a file.

### Option 2: hosted server with an API token

Some apps ask for a header instead. Use your Apify API token:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com",
      "headers": {
        "Authorization": "Bearer YOUR_APIFY_TOKEN"
      }
    }
  }
}
```

Find your token in Apify Console under **Settings → API & Integrations**. Keep it private.

### Option 3: run the server on your computer

You can also run the server locally with Node.js (version 18 or newer):

```json
{
  "mcpServers": {
    "apify": {
      "command": "npx",
      "args": ["-y", "@apify/actors-mcp-server"],
      "env": { "APIFY_TOKEN": "YOUR_APIFY_TOKEN" }
    }
  }
}
```

### Add our tools directly

By default, the server gives your agent tools to **search** the Apify Store, read Actor details and **call** an Actor. Your agent can find our tools by itself.

You can also load exact tools with the `tools` setting in the URL:

```
https://mcp.apify.com?tools=cliqtomedia/youtube-transcript-scraper,cliqtomedia/website-contacts-scraper
```

Then the agent sees these tools right away, with their input fields. This is faster and more reliable.

Give your agent only the tools it needs. A short tool list means fewer mistakes and lower costs.

Read the full guide in the [Apify MCP docs](https://mcp.apify.com/?fpr=cliqto).

## Claude, ChatGPT and Cursor

Many popular AI apps can work with MCP. Features change fast, so check your app's own help pages.

### Claude
**Claude web scraping** works through MCP. Add the Apify server as a connector in Claude, or in Claude Desktop. Then ask: "Use the YouTube transcript tool on this video and summarize it."

### Cursor
A **Cursor MCP scraper** setup helps while you code. Add the server in Cursor's MCP settings. Ask it to read docs pages as Markdown, or to check npm packages.

### VS Code
VS Code supports MCP servers for its AI chat. Add the Apify server in the MCP settings of your workspace.

### ChatGPT
For **ChatGPT web data**, check if your plan supports custom connectors or MCP. Apify also lists ChatGPT and Codex in its integrations docs.

For coding agents like Claude Code, Cursor and GitHub Copilot, Apify also offers a plugin and agent skills. See the Apify docs for the latest setup.

### Example prompts

Try prompts like these after you connect:

- "Use the cliqtomedia Greenhouse jobs tool. Get all open jobs from these 5 boards. Show only data jobs."
- "Turn this docs site into Markdown with the cliqtomedia website-to-markdown tool. Limit it to 30 pages."
- "Check these 200 emails with the cliqtomedia email verifier. Give me only the deliverable ones."
- "Get the last 100 reviews of this app from Google Play. Group the complaints by topic."

Ask the agent to show the input before it starts a run. This helps you catch big limits early.

## LangChain and other frameworks

You can also use our tools in your own AI app. A **framework** is a code library for building AI apps.

| Framework | How it connects to Apify |
|---|---|
| **LangChain** | `pip install langchain-apify`. Use `ApifyActorsTool` to run any Actor, or `ApifyDatasetLoader` to load a dataset as documents. |
| **LlamaIndex** | `llama-index-readers-apify` has an Actor reader and a dataset reader. |
| **OpenAI Agents SDK** | Listed in the Apify integrations docs. You can also connect through MCP. |
| **Vercel AI SDK** | Listed in the Apify integrations docs. |
| **Google ADK** | Listed in the Apify integrations docs. |
| **CrewAI, Dify, Langflow** | Use MCP, or call the Apify API from an HTTP step. |

A short LangChain example:

```python
# pip install langchain-apify
# set APIFY_TOKEN in your environment
from langchain_apify import ApifyActorsTool

tool = ApifyActorsTool("cliqtomedia/youtube-transcript-scraper")
result = tool.invoke({"run_input": {"videoUrls": ["jNQXAC9IVRw"]}})
print(result)
```

You can give this tool to a LangChain agent. The agent decides when to call it. See also [Integrations](/actors/integrations/).

## LLM-ready output: Markdown and JSON
**LLM-ready data** is data an AI model can read easily. It has no menus, ads or cookie banners. It is short, clean and clear.

Our tools give two main formats:

- **Markdown.** Plain text with simple marks for headings, lists and tables. Great for reading, summaries and RAG.
- **JSON.** Data with named fields. Great for tables, filters and function calls.

### What is RAG web data

**RAG** means **retrieval-augmented generation**. The AI first finds useful text. Then it uses this text to answer. **RAG web data** is web text that you collect for this.

A simple RAG flow:

1. **Collect.** Use [Website to Markdown for RAG: Crawl & Changed Pages](/actors/website-to-markdown/) to turn a docs site into Markdown pages.
2. **Split.** Cut each page into small chunks, for example 500 words.
3. **Embed.** Turn each chunk into numbers (an embedding) with an AI model.
4. **Store.** Save the chunks in a vector database.
5. **Ask.** When a user asks a question, find the best chunks and give them to the LLM.

[PDF to Markdown for LLMs & RAG Chunks - No Login](/actors/pdf-to-markdown/) (coming soon) will return ready chunks with page numbers and token counts. A token is a small piece of a word. LLMs count text in tokens.

### Example output

Here is a short example row from [Fast Webpage to Markdown API for AI Agents](/actors/webpage-to-markdown-api/) (coming soon; the live [Website to Markdown for RAG: Crawl & Changed Pages](/actors/website-to-markdown/) returns the same kind of Markdown):

```json
{
  "url": "https://example.com/docs/start",
  "title": "Getting started",
  "statusCode": 200,
  "wordCount": 412,
  "markdown": "# Getting started\n\nInstall the app with one command..."
}
```

Field names differ a little in each tool. Each tool page shows its output fields.

## Tools that work well with agents

These tools have short inputs and clean output. They are a good start for AI agents and MCP.

### Read the web as text

- [Fast Webpage to Markdown API for AI Agents](https://cliqtomedia.com/actors/webpage-to-markdown-api/): Give a link and get the page as clean Markdown in a few seconds.
- [Website to Markdown for RAG: Crawl & Changed Pages](https://cliqtomedia.com/actors/website-to-markdown/): Give a start URL and a page limit.
- [RAG Web Search: Google Results as Clean Markdown](https://cliqtomedia.com/actors/rag-web-search/): Type a question or topic and get the top Google pages as clean text.
- [PDF to Markdown for LLMs & RAG Chunks - No Login](https://cliqtomedia.com/actors/pdf-to-markdown/): Paste a PDF link and get Markdown with real headings, lists and tables.
- [PDF & Image to Text OCR for AI Agents - No Login](https://cliqtomedia.com/actors/pdf-text-ocr/): Give a link to a PDF or an image.
- [News Article Extractor: RSS, URLs & Clean Text](https://cliqtomedia.com/actors/news-article-extractor/): Paste news links or a news site.

### Video and audio to text

- [YouTube Subtitles & Transcript Scraper API](https://cliqtomedia.com/actors/youtube-transcript-scraper/): Get the public captions of a YouTube video as full text and timed parts.
- [Video URL to Text: Transcribe TikTok, Reels, X, YouTube](https://cliqtomedia.com/actors/video-to-text/): Paste a video link and get the spoken words as text.
- [Audio & Podcast Transcriber (Speech to Text)](https://cliqtomedia.com/actors/audio-video-transcriber/): Turn audio files or podcast episodes into text.

### Research and checks

- [Website Contact Scraper](https://cliqtomedia.com/actors/website-contacts-scraper/): Add a list of company websites.
- [Website Tech Stack Detector: CMS, Versions & CDN, No Login](https://cliqtomedia.com/actors/tech-stack-detector/): Paste a list of websites. The tool checks each live page against 7,600+ technology fingerprints. You get one row per site with CMS, shop…
- [Bulk Email Verifier: MX, Disposable, Role & Typo Check](https://cliqtomedia.com/actors/email-verifier/): Paste email addresses or pick a dataset from a lead scraper.
- [Google Autocomplete Scraper](https://cliqtomedia.com/actors/google-autocomplete-scraper/): Get the phrases Google suggests when people start typing.
- [Bluesky Scraper — Posts, Profiles, Feeds & Threads, No Login](https://cliqtomedia.com/actors/bluesky-scraper/): Get public Bluesky posts, profiles, threads and feeds as clean rows.
- [Google Play Reviews Scraper](https://cliqtomedia.com/actors/google-play-reviews-scraper/): Export written Google Play reviews for your apps.

See all of them in [AI-ready content & documents](/actors/ai-content/) and in the full [catalog](/actors/).

## Keep costs under control

Agents can start many runs fast. A loop or a big list can cost more than you expect. Use these safety steps.

### Limit results
Tell the agent to always set a small "max results" value. For example, 50.

### Max cost per run
Pay-per-event tools let you set a maximum charge for each run. The run stops in a clean way at the limit.

### Monthly limit
In Apify Console, go to **Billing → Limits** and set a custom usage limit.

### Small tool list
Load only the tools you need with `?tools=`. The agent will not try random tools.

### Confirm first
Ask the agent to show you the input and wait for "yes" before each run.

### Check usage
Look at your runs and costs in Apify Console from time to time.

Read more on our [Pricing](/actors/pricing/) page.

Never share your Apify API token in a chat, a prompt or public code. Anyone with the token can run tools on your account. If it leaks, make a new token in Console.

## Questions

### What is the Apify MCP server?
It is a server that lets AI apps use Apify Actors as tools. Your agent can search for a tool, run it and read the results.

### Which AI apps can use it?
MCP clients like Claude, Cursor and VS Code can connect. Features differ by app.

### Do I need an Apify account?
Yes. The agent runs tools with your Apify account. Keep your API token private.

### Which output is best for LLMs?
Clean Markdown or JSON. Our AI content tools return Markdown that is ready for RAG.

### How do I stop an agent from spending too much?
Set a max cost per run and a monthly limit in Apify. Also limit the number of results in the input.

### What is an MCP web scraper?
It is a scraper that an AI app can call through MCP. The AI sends the input, the tool reads the web, and the AI gets clean data back.

### Can the agent find your tools by itself?
Yes. The Apify MCP server has a search tool for the Apify Store. But it is faster to add our tools directly with `?tools=`.

### Is MCP safe?
MCP is only a way to connect. Safety depends on the tools and your limits. Use OAuth or keep your token private, and set spending limits.

### Can I use your tools without MCP?
Yes. Use the Apify API, the Python or JavaScript client, or LangChain. See [How it works](/actors/how-it-works/).

### Do your tools need a login to the target site?
Most of our tools read public data with no login. The tool page says what you need.

### Is the data good for training AI models?
Check the terms of each source site and your local law first. Many sites allow reading but limit reuse. This is not legal advice.

Connect your AI agent to fresh web data today.

[Create a free Apify account](https://console.apify.com/sign-up?fpr=cliqto) [Read the Apify MCP docs](https://mcp.apify.com/?fpr=cliqto) [Browse AI-ready tools](/actors/ai-content/)