# Website to Markdown: Crawl a Site or Docs Section for RAG

> Website to Markdown crawler for RAG: turn a docs site or section into clean Markdown, one row per page. Repeat runs can return only changed pages.

URL: https://cliqtomedia.com/actors/website-to-markdown/

Category: AI-ready content & documents · Status: live on Apify Store · Apify page: https://apify.com/cliqtomedia/website-to-markdown?fpr=cliqto

Give a website address. You get the text of every page as clean Markdown for AI tools and search.

## How to use it: 3 steps

1. **Make a free Apify account.** Apify is the cloud platform where our tools run. The free plan gives you $5 of usage every month, no credit card needed. You pay Apify, not us.
2. **Paste your input and press Start.** No code needed: you fill in a form (the input). It already has a working example. For example: `https://docs.apify.com/academy` in Start URLs. Each start is one run.
3. **Download the results.** When the run ends, the results are in a table (the dataset). Open Output, press Export and pick Excel or CSV. JSON and the API are there too.

## What it does

Turn a website or a docs section into clean Markdown for RAG and LLM pipelines: one Dataset row per page, without menus, footers or cookie banners. Add one start URL and a page limit. The smallest run (20 pages of a docs site) takes a few seconds. On repeat runs, `onlyChanged` returns only new, changed and removed pages. No login, no browser.



### Good for

- Feed a docs site or help center into a vector database or a RAG chatbot.
- Refresh that index every day or week and re-embed only the pages that changed.
- Get LLM-ready Markdown of a blog section, a product guide or an API reference.

### Not a good fit

- JavaScript is not executed. Single-page apps that render content in the browser give empty or short pages, marked `status: partial`.
- No login, cookies, CAPTCHA or paywalled pages. PDF, DOCX and images are skipped.
- Up to 2000 pages per run and up to 50 start URLs. The output is whole pages; the Actor does not split text into chunks or create embeddings.

## Price

Pay per event. Apify bills you, not us.

| What you pay for | Price | When |
|---|---|---|
| Start fee | $0.002 | Charged once per run. This tool uses 1 GB of memory or less by default; more memory costs more. |
| Page | $0.00019 (from $0.00016 on paid Apify plans) | One web page converted to Markdown and saved to the Dataset. Failed, skipped and removed pages are not charged. |
| Unchanged page | $0.00012 | With onlyChanged, one page that was checked and did not change since the last run. It is not saved to the Dataset. |
| Extra run time | $0.0003 | One second of run time above the included time (5 seconds per run plus 0.4 seconds per charged page). Charged only for very slow sites or a long robots.txt Crawl-delay. |

- One page: about $0.0022.
- 1,000 pages: about $0.19.
- Every day 100 pages, for 30 days: about $0.63 a month.

Free to try: the Apify free plan gives you $5 of usage every month, no credit card needed.

Prices checked on 24 October 2026. Current price: https://apify.com/cliqtomedia/website-to-markdown?fpr=cliqto

## Questions

### Do I need an account on the website or an API key?

No account on the website you crawl. To start the tool from your own code, you need your Apify API token (Apify Console → Settings → API & Integrations).

### Does it follow robots.txt?

Yes. Pages blocked by robots.txt are skipped and not charged.

### Does it work on JavaScript apps?

Not well. It does not run JavaScript, so such pages come back short and marked partial.

### Does it split text into chunks?

No. You get whole pages. Do the chunking and embeddings in your own pipeline.

### How many pages can I crawl?

Up to 2000 pages and 50 start URLs per run.

### Why was I charged `extra-second`?

The site was slow or its `robots.txt` asks to wait between requests (`Crawl-delay`). The Actor waits as asked, and that waiting time above the included time is charged. Check `RUN_SUMMARY.counts.chargedExtraSeconds`. Lower `maxPages`, set a shorter run timeout or a Maximum cost per run to cap it.

### How does "only changed" decide that a page changed?

It compares a hash of the clean Markdown. Changes in menus, ads or whitespace do not count. A page is downloaded each time, but an unchanged page is not saved and costs less. History is kept in your account in the key-value store `website-to-markdown-delta`.

### When do I get removed pages?

When a page from the last delta run now answers HTTP 404 or 410. Pages that the crawl no longer reaches by links are re-checked directly (they count as normal or unchanged pages). Pages that fail with other errors, or are not re-checked because `maxPages` was reached, stay in the history and are not reported as removed. If no page loads at all in a run (the whole site answers 404), nothing is marked removed. A page that now redirects to another URL is not marked removed either. Up to 1000 `removed` rows per run.

### Is the Markdown complete?

It is the main content of the HTML page. Very large pages are cut (the row gets `status: partial` and a note). Content that appears only after JavaScript runs is not included.

## Data fields

- `url` (string): Page URL after redirects, normalized (no #fragment, no tracking parameters). Key for delta.
- `title` (string): Page title.
- `description` (string): Meta description.
- `language` (string): Language from <html lang>, e.g. en.
- `markdown` (string): Main content as Markdown (navigation, headers, footers and cookie banners removed). null for removed pages.
- `wordCount` (integer): Words in the Markdown.
- `contentHash` (string): SHA-256 of the Markdown with whitespace normalized; equal hash = same content.
- `changeStatus` (string): With onlyChanged: new, changed or removed. Unchanged pages are not saved. null when onlyChanged is off.
- `previousCrawledAt` (string): When this page was last crawled by a previous delta run.
- `lastModified` (string): Last-Modified HTTP header or sitemap lastmod, ISO 8601. null when the site does not say.
- `httpStatus` (integer): HTTP status of the page.
- `depth` (integer): Link depth from the start URL (0 = start URL).
- `referrerUrl` (string): Page where the link to this page was found.
- `links` (array): Links found on the page (only with includeLinks).
- `status` (string): complete, partial (little text; the page probably needs JavaScript, or the Markdown was cut) or removed.
- `note` (string): Why a page is partial or removed.
- `crawledAt` (string): When the page was fetched, ISO 8601.
- `source` (string): Always website.

## Input settings

- `startUrls` (array, required): Public pages to start from (up to 50). By default only pages under the same path are crawled: https://docs.example.com/guide crawls /guide/… but not /blog/. Pages behind a login, private or internal addresses are not crawled.
- `maxPages` (integer): Stop after this many HTML pages (1–2000). The run never returns more pages than this.
- `maxDepth` (integer): How many clicks away from a start URL to follow. 0 = only the start URLs.
- `crawlScope` (string): Which links to follow: under the start URL path (recommended for docs), anywhere on the same host, or on any subdomain of the same domain.
- `useSitemap` (boolean): Also add in-scope URLs from the site's sitemap (from robots.txt or /sitemap.xml). Finds pages that are not linked. Sitemap dates fill lastModified.
- `includeGlobs` (array): Only crawl URLs matching one of these patterns (full URL). * matches inside one path segment, ** matches anything. Example: https://docs.example.com/**/api/**. When set, replaces the path rule but stays on the start hosts. Pages in between (hubs) must match too, or the pages linked only from them are not found.
- `excludeGlobs` (array): Never crawl URLs matching these patterns. Example: **/changelog/** or **/*?page=*.
- `onlyChanged` (boolean): Remember each page's content between runs and return only new and changed pages. Unchanged pages are not saved to the dataset and cost less. Pages from the previous run that now answer HTTP 404 or 410 are listed as 'removed' (free); pages the crawl no longer reaches by links are re-checked directly. The first run returns every page as 'new'.
- `stateKey` (string): Optional name for the saved delta state (letters, digits, -_.). By default it is derived from the start URLs and patterns, so the same input always continues the same state. Use different names to keep separate histories.
- `removeSelectors` (string): CSS selectors to remove before conversion, e.g. .ad, .related-posts. Navigation, headers, footers, sidebars and cookie banners are always removed.
- `includeLinks` (boolean): Add the list of links found on each page, including links to other sites (up to 1000).
- `maxConcurrency` (integer): How many pages to load at the same time (1–10). The site's robots.txt Crawl-delay is always respected.
- `proxyConfiguration` (object): Usually not needed. Use a proxy only if the site blocks the default connection. Apify Proxy with automatic or datacenter groups, or your own proxy URLs. Residential and SERP proxy groups are not supported (the run fails with an input error).

## Step by step on Apify

1. Open the Input tab.
2. In Start URLs (`startUrls`) keep the prefill `https://docs.apify.com/academy` or add your own page, for example `https://docs.example.com/guide`.
3. Keep Max pages (`maxPages`) at the prefill 20. Keep Crawl scope at the default `path`.
4. Click Start.
5. Open the Output tab, view Markdown, and check the `markdown` field of the first row.

## Use cases

| Task | Input | You get | Key fields | Next step |
|---|---|---|---|---|
| Index a docs site for RAG | `startUrls` + `maxPages` (+ `useSitemap`) | Page | `url`, `title`, `markdown`, `wordCount` | Chunk `markdown`, embed, store with `url` as source |
| Keep the index fresh | same input + `onlyChanged: true` on a schedule | Changed page | `changeStatus`, `url`, `markdown`, `contentHash` | Upsert `new`/`changed`, delete `removed` by `url` |
| One section only | `startUrls` + `includeGlobs` / `excludeGlobs` | Page | `url`, `markdown` | LLM context, knowledge base |
| Collect page links | `includeLinks: true` | Page | `url`, `links` | Site map, link checks |

## Tips

- Start from a section path. The crawler stays inside the path you give.
- Turn on only changed for repeat runs. Re-embed only pages that changed.
- Delete removed pages from your index by URL. Removed rows are free.

## Limits



- Included run time: 5 seconds per run plus 0.4 seconds per charged page. Normal sites stay well inside it (2000 heavy pages took 577 of 805 included seconds), so `extra-second` is not charged.
- `extra-second` appears only on very slow sites or when `robots.txt` sets a long `Crawl-delay`.
- `removed` rows are free.
- Failed pages, skipped pages (robots.txt, not HTML, duplicates) are free.
- A run with no pages (for example, `robots.txt` blocks everything) costs only the run start. Failed runs (site not reachable, invalid input) also pay the run start.
- The Actor stops when it reaches your Maximum cost per run. Rows saved before that stay in the Dataset.

## Sample result

```json
{
  "url": "https://docs.apify.com/academy",
  "title": "Apify Academy | Academy | Apify Documentation",
  "description": "Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.",
  "language": "en",
  "markdown": "# Apify Academy\n\nLearn everything about web scraping and automation with our free courses…",
  "wordCount": 183,
  "contentHash": "f5aa630681e827a9a830955f9baaaf034734f2c4bc0d0610e82a9d76359723ce",
  "changeStatus": "new",
  "previousCrawledAt": null,
  "lastModified": "2026-10-13T19:47:40.000Z",
  "httpStatus": 200,
  "depth": 0,
  "referrerUrl": null,
  "status": "complete",
  "note": null,
  "crawledAt": "2026-10-14T02:35:12.564Z",
  "source": "website"
}
```

## Use it from code

Actor ID: `cliqtomedia/website-to-markdown`

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("cliqtomedia/website-to-markdown").call(run_input={
    "startUrls": [
        "https://docs.apify.com/academy"
    ],
    "maxPages": 20
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
```

```bash
curl -X POST "https://api.apify.com/v2/acts/cliqtomedia~website-to-markdown/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://docs.apify.com/academy"], "maxPages": 20}'
```

```python
# pip install langchain-apify
# set APIFY_TOKEN in your environment
from langchain_apify import ApifyActorsTool

tool = ApifyActorsTool("cliqtomedia/website-to-markdown")
result = tool.invoke({"run_input": {"startUrls": ["https://docs.apify.com/academy"], "maxPages": 20}})
print(result)
```

run-sync waits up to 300 seconds; for longer runs start a run and use a webhook. LlamaIndex and MCP: https://cliqtomedia.com/actors/ai-agents/

## Use cases with this tool

- [One piece of content, ten new posts](https://cliqtomedia.com/actors/use-cases/content-repurposing/): Turn videos, podcasts, PDFs and articles into clean text. Make posts, notes and AI drafts from it.

## Similar tools

- [PDF & Image to Text OCR for AI Agents - No Login](https://cliqtomedia.com/actors/pdf-text-ocr/)
- [RSS & Atom Feed Reader / Monitor](https://cliqtomedia.com/actors/rss-atom-feed-reader-monitor/)
- [WordPress Posts & Pages Scraper](https://cliqtomedia.com/actors/wordpress-posts-pages-scraper/)
- [News Article Extractor: RSS, URLs & Clean Text](https://cliqtomedia.com/actors/news-article-extractor/)
- [RAG Web Search: Google Results as Clean Markdown](https://cliqtomedia.com/actors/rag-web-search/)
- [Website Screenshot & PDF Generator: Full Page & Mobile](https://cliqtomedia.com/actors/website-screenshot-pdf/)