How to use it: 3 steps
- Make a free Apify accountApify is the cloud platform where our tools run. The free plan gives you $5 of usage every month, no credit card needed. You pay Apify, not us.
- Paste your input and press StartNo code needed: you fill in a form (the input). It already has a working example. For example:
https://docs.apify.com/academyin Start URLs. Each start is one run. - Download the resultsWhen the run ends, the results are in a table (the dataset). Open Output, press Export and pick Excel or CSV. JSON and the API are there too.
This tool is an Apify Actor: a small program that runs on Apify. You can also run it on a schedule, for example every Monday. New to it? Read how it works, step by step and the 8 words you need first.
Made by Cliqto Media, an independent digital media company. We made this tool and keep it working. About us
What it does
Turn a website or a docs section into clean Markdown for RAG and LLM pipelines: one Dataset row per page, without menus, footers or cookie banners. Add one start URL and a page limit. The smallest run (20 pages of a docs site) takes a few seconds. On repeat runs, onlyChanged returns only new, changed and removed pages. No login, no browser.
Good for
- Feed a docs site or help center into a vector database or a RAG chatbot.
- Refresh that index every day or week and re-embed only the pages that changed.
- Get LLM-ready Markdown of a blog section, a product guide or an API reference.
Not a good fit
- JavaScript is not executed. Single-page apps that render content in the browser give empty or short pages, marked
status: partial. - No login, cookies, CAPTCHA or paywalled pages. PDF, DOCX and images are skipped.
- Up to 2000 pages per run and up to 50 start URLs. The output is whole pages; the Actor does not split text into chunks or create embeddings.
Who uses it
Chatbot builders
Feed a help center or docs site into your RAG chatbot.
Developer relations teams
Keep an LLM index of your own docs fresh every week.
Technical writers
Get a Markdown copy of a guide or API reference section.
Price
Pay per event: you pay for each result the tool saves, plus a small start fee for each run. The price includes Apify platform costs. Apify bills you, not us. Prices below are for the Apify free plan.
| What you pay for | Price | When |
|---|---|---|
| Start fee | $0.002 | Charged once per run. This tool uses 1 GB of memory or less by default; more memory costs more. |
| Page | $0.00019 (from $0.00016 on paid Apify plans) | One web page converted to Markdown and saved to the Dataset. Failed, skipped and removed pages are not charged. |
| Unchanged page | $0.00012 | With onlyChanged, one page that was checked and did not change since the last run. It is not saved to the Dataset. |
| Extra run time | $0.0003 | One second of run time above the included time (5 seconds per run plus 0.4 seconds per charged page). Charged only for very slow sites or a long robots.txt Crawl-delay. |
Examples
- One page: about $0.0022.
- 1,000 pages: about $0.19.
- Every day 100 pages, for 30 days: about $0.63 a month.
Free to try: the Apify free plan gives you $5 of usage every month, no credit card needed. That is enough for about 26,000 pages.
Prices checked on 24 October 2026. See the current price on Apify.
Questions
Do I need an account on the website or an API key?
No account on the website you crawl. To start the tool from your own code, you need your Apify API token (Apify Console → Settings → API & Integrations).
Does it follow robots.txt?
Yes. Pages blocked by robots.txt are skipped and not charged.
Does it work on JavaScript apps?
Not well. It does not run JavaScript, so such pages come back short and marked partial.
Does it split text into chunks?
No. You get whole pages. Do the chunking and embeddings in your own pipeline.
How many pages can I crawl?
Up to 2000 pages and 50 start URLs per run.
Why was I charged extra-second?
The site was slow or its robots.txt asks to wait between requests (Crawl-delay). The Actor waits as asked, and that waiting time above the included time is charged. Check RUN_SUMMARY.counts.chargedExtraSeconds. Lower maxPages, set a shorter run timeout or a Maximum cost per run to cap it.
How does "only changed" decide that a page changed?
It compares a hash of the clean Markdown. Changes in menus, ads or whitespace do not count. A page is downloaded each time, but an unchanged page is not saved and costs less. History is kept in your account in the key-value store website-to-markdown-delta.
When do I get removed pages?
When a page from the last delta run now answers HTTP 404 or 410. Pages that the crawl no longer reaches by links are re-checked directly (they count as normal or unchanged pages). Pages that fail with other errors, or are not re-checked because maxPages was reached, stay in the history and are not reported as removed. If no page loads at all in a run (the whole site answers 404), nothing is marked removed. A page that now redirects to another URL is not marked removed either. Up to 1000 removed rows per run.
Is the Markdown complete?
It is the main content of the HTML page. Very large pages are cut (the row gets status: partial and a note). Content that appears only after JavaScript runs is not included.
Details
Step by step on Apify
- Open the Input tab.
- In Start URLs (
startUrls) keep the prefillhttps://docs.apify.com/academyor add your own page, for examplehttps://docs.example.com/guide. - Keep Max pages (
maxPages) at the prefill 20. Keep Crawl scope at the defaultpath. - Click Start.
- Open the Output tab, view Markdown, and check the
markdownfield of the first row.
Use cases
| Task | Input | You get | Key fields | Next step |
|---|---|---|---|---|
| Index a docs site for RAG | startUrls + maxPages (+ useSitemap) | Page | url, title, markdown, wordCount | Chunk markdown, embed, store with url as source |
| Keep the index fresh | same input + onlyChanged: true on a schedule | Changed page | changeStatus, url, markdown, contentHash | Upsert new/changed, delete removed by url |
| One section only | startUrls + includeGlobs / excludeGlobs | Page | url, markdown | LLM context, knowledge base |
| Collect page links | includeLinks: true | Page | url, links | Site map, link checks |
Tips for better results
- Start from a section path. The crawler stays inside the path you give.
- Turn on only changed for repeat runs. Re-embed only pages that changed.
- Delete removed pages from your index by URL. Removed rows are free.
Limits and good to know
- Included run time: 5 seconds per run plus 0.4 seconds per charged page. Normal sites stay well inside it (2000 heavy pages took 577 of 805 included seconds), so
extra-secondis not charged extra-secondappears only on very slow sites or whenrobots.txtsets a longCrawl-delayremovedrows are free- Failed pages, skipped pages (robots.txt, not HTML, duplicates) are free
- A run with no pages (for example,
robots.txtblocks everything) costs only the run start. Failed runs (site not reachable, invalid input) also pay the run start - The Actor stops when it reaches your Maximum cost per run. Rows saved before that stay in the Dataset
For developers
Sample result
One result from a real test run, in JSON. Real values change from run to run. In Excel or CSV each field is a column.
{
"url": "https://docs.apify.com/academy",
"title": "Apify Academy | Academy | Apify Documentation",
"description": "Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.",
"language": "en",
"markdown": "# Apify Academy\n\nLearn everything about web scraping and automation with our free courses…",
"wordCount": 183,
"contentHash": "f5aa630681e827a9a830955f9baaaf034734f2c4bc0d0610e82a9d76359723ce",
"changeStatus": "new",
"previousCrawledAt": null,
"lastModified": "2026-10-13T19:47:40.000Z",
"httpStatus": 200,
"depth": 0,
"referrerUrl": null,
"status": "complete",
"note": null,
"crawledAt": "2026-10-14T02:35:12.564Z",
"source": "website"
}Use it from code
Actor ID: cliqtomedia/website-to-markdown. You need your Apify API token from Apify Console → Settings → API & Integrations.
from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("cliqtomedia/website-to-markdown").call(run_input={
"startUrls": [
"https://docs.apify.com/academy"
],
"maxPages": 20
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item)curl -X POST "https://api.apify.com/v2/acts/cliqtomedia~website-to-markdown/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{"startUrls": ["https://docs.apify.com/academy"], "maxPages": 20}'# pip install langchain-apify
# set APIFY_TOKEN in your environment
from langchain_apify import ApifyActorsTool
tool = ApifyActorsTool("cliqtomedia/website-to-markdown")
result = tool.invoke({"run_input": {"startUrls": ["https://docs.apify.com/academy"], "maxPages": 20}})
print(result)The cURL call waits for the results, up to 300 seconds. For longer runs, start a run (/runs) and get the results with a webhook or by checking the run. Apify API docs. LlamaIndex, MCP for AI agents and the JavaScript client: AI agents & MCP.
Data fields (18)
| Field | Type | Meaning |
|---|---|---|
url | string | Page URL after redirects, normalized (no #fragment, no tracking parameters). Key for delta. |
title | string | Page title. |
description | string | Meta description. |
language | string | Language from <html lang>, e.g. en. |
markdown | string | Main content as Markdown (navigation, headers, footers and cookie banners removed). null for removed pages. |
wordCount | integer | Words in the Markdown. |
contentHash | string | SHA-256 of the Markdown with whitespace normalized; equal hash = same content. |
changeStatus | string | With onlyChanged: new, changed or removed. Unchanged pages are not saved. null when onlyChanged is off. |
previousCrawledAt | string | When this page was last crawled by a previous delta run. |
lastModified | string | Last-Modified HTTP header or sitemap lastmod, ISO 8601. null when the site does not say. |
httpStatus | integer | HTTP status of the page. |
depth | integer | Link depth from the start URL (0 = start URL). |
referrerUrl | string | Page where the link to this page was found. |
links | array | Links found on the page (only with includeLinks). |
status | string | complete, partial (little text; the page probably needs JavaScript, or the Markdown was cut) or removed. |
note | string | Why a page is partial or removed. |
crawledAt | string | When the page was fetched, ISO 8601. |
source | string | Always website. |
Input settings (13)
The main fields of the input form. With the API you send them as JSON.
| Setting | Name in the API | Type | Example value |
|---|---|---|---|
| Start URLs required | startUrls | array | ["https://docs.apify.com/academy"] |
| Max pages | maxPages | integer | 20 |
| Max link depth | maxDepth | integer | 5 |
| Crawl scope | crawlScope | string | "path" |
| Use sitemap.xml | useSitemap | boolean | false |
| Include URL patterns | includeGlobs | array | |
| Exclude URL patterns | excludeGlobs | array | |
| Only changed pages (delta) | onlyChanged | boolean | false |
| Delta state name | stateKey | string | |
| Extra elements to remove | removeSelectors | string | |
| Include page links | includeLinks | boolean | false |
| Parallel requests | maxConcurrency | integer | 5 |
| Proxy | proxyConfiguration | object | {"useApifyProxy": false} |
Use cases with this tool
Each guide shows all the steps and the other tools.
Similar tools
More in AI-ready content & documents.
AI & documentsLive on ApifyPDF & Image to Text OCR for AI Agents - No Login
Give a link to a PDF or an image.
AI & documentsLive on ApifyRSS & Atom Feed Reader / Monitor
Paste one or more feed links and get each item as a row.
AI & documentsLive on ApifyWordPress Posts & Pages Scraper
Add a WordPress site URL and export its public posts and pages.
AI & documentsLive on ApifyNews Article Extractor: RSS, URLs & Clean Text
Paste news links or a news site.
AI & documentsLive on ApifyRAG Web Search: Google Results as Clean Markdown
Type a question or topic and get the top Google pages as clean text.
AI & documentsLive on ApifyWebsite Screenshot & PDF Generator: Full Page & Mobile
Make full-page pictures or PDFs of a list of web pages.