Hacker News Scraper: Stories & Comments

How to use it: 3 steps

Make a free Apify accountApify is the cloud platform where our tools run. The free plan gives you $5 of usage every month, no credit card needed. You pay Apify, not us.
Paste your input and press StartNo code needed: you fill in a form (the input). It already has a working example. For example: [] in Exact item IDs. Each start is one run.
Download the resultsWhen the run ends, the results are in a table (the dataset). Open Output, press Export and pick Excel or CSV. JSON and the API are there too.

This tool is an Apify Actor: a small program that runs on Apify. You can also run it on a schedule, for example every Monday. New to it? Read how it works, step by step and the 8 words you need first.

Made by Cliqto Media, an independent digital media company. We made this tool and keep it working. About us

What it does

Export Hacker News stories and comments as rows with stable IDs and parent links. Choose Top, New, Best, Ask HN, Show HN or Jobs, or add exact item IDs. No Hacker News API key is needed.

Who uses it

Startup founders

Read the feedback on your Show HN post in one table.

Tech researchers

Build a dataset of public tech discussions.

Recruiters

Export current Hacker News job posts.

Price

Pay per event: you pay for each result the tool saves, plus a small start fee for each run. The price includes Apify platform costs. Apify bills you, not us. Prices below are for the Apify free plan.

What you pay for	Price	When
Start fee	$0.05	Charged once per run. This tool uses 1 GB of memory or less by default; more memory costs more.
Useful HN entity saved	$0.00027	One live story, job or comment saved to the Dataset. Deleted, dead, unavailable and error rows have no entity fee.

Examples

One story or comment: about $0.05.
1,000 stories and comments: about $0.32.
Every day 100 stories and comments, for 30 days: about $2.31 a month.

Free to try: the Apify free plan gives you $5 of usage every month, no credit card needed. That is enough for about 15,000 stories and comments.

Prices checked on 24 October 2026. See the current price on Apify.

Questions

Does it get every comment of a big thread?

Not always. There are caps per root and per run. The tree status tells you if a tree is complete.

Does it open the linked article?

No. You get the link from Hacker News, not the article text.

Which lists can I use?

Top, New, Best, Ask HN, Show HN and Jobs.

Do I need an API key?

No HN API key, cookie or login is needed. You need your Apify account to run the Actor. API clients use an Apify token in an authorization header, never in this Actor's input.

Does it return every comment?

Only when every reachable child is observed within the caps and treeStatus is complete. The new form intentionally uses small limits. Check the summary for each root. descendants is the source count, not a completeness check.

Why are HTML and plain text different?

textHtml keeps the source. text removes tags, adds block boundaries and decodes entities once. &lt; becomes <, not an HTML tag. Deleted items can have neither value.

What do deleted, dead and null mean?

Deleted and dead are source flags. A null response means the item was unavailable at fetch time. It does not prove why. These observations are kept; they are not silently dropped and do not have an entity fee.

Why is there a start fee?

The scan still makes requests and saves a summary when there are no useful entities. The start fee covers this bounded work. Each fresh run has its own start fee.

Is it a replacement for another HN scraper?

Use a field mapping before switching. This Actor uses numeric IDs, url for the external link and hnUrl for HN. Comments are flat rows with stable IDs, not nested message arrays.

Details

Step by step on Apify

Open Input in Apify Console.
Keep the new form as it is: Top stories, 3 roots, up to 5 comments per root and depth 2.
Keep memory at 256 MB and timeout at 300 seconds. Set a maximum cost of $0.15 for this small example.
Click Start.
Open Dataset. Look for a story row and comment rows linked by rootId and parentId.
Open Key-value store → RUN_SUMMARY. A current busy story can be partial because the example uses small caps.

Use cases

Goal	Input choice	Fields to use
Track current tech discussions	`top`, `new` or `best`	`id`, `title`, `score`, `descendants`, `retrievedAt`
Study project feedback	`show` or exact `itemIds`	`rootId`, `parentId`, `depth`, `text`, `treeStatus`
Build a public discussion corpus	Exact IDs, bounded comments	`id`, `textHtml`, `text`, `deleted`, `dead`, summary
Export HN jobs	`job`, comments off	`title`, `url`, `author`, `createdAt`

Tips for better results

Add exact item IDs to export one known discussion.
Check the tree status to see if a comment tree is complete.
Use the plain text field for AI tools and the HTML field for display.

Limits and good to know

At most 100 roots, 200 descendant observations per root, depth 10 and 1,000 total reserved nodes. A live Cloud export reached 1,000 observed nodes within the configured limits; this is a cap, not a guaranteed amount
Only the six official lists are supported. Lists can be smaller than maxStories and can include jobs. Their order can change while the run is active
Source work has a 90-second deadline, at most 1,300 HTTP attempts, concurrency 4, an 8-second request timeout, a 2 MiB response cap and an 8 MiB total response cap. Temporary transport, 429 and 5xx errors are retried up to three attempts within these caps
Set platform timeout to 300 seconds. A longer finite timeout also works but does not increase the 90-second source deadline. A smaller timeout reduces the source window to leave time for saved output
comment_cap, depth_cap, node_cap, request_cap, byte_cap, response_byte_cap, deadline and charge_cap show a limit. unavailable_child, missing_parent, parent_mismatch, unexpected_child_type and duplicate_or_cycle show an observed source/graph issue
treeStatus describes what was fetched. If the charge cap stops saving rows, use run-level status=partial to see that the export is incomplete
Original HTML is data. Do not render textHtml, titles or external URLs as trusted HTML in your own app. Use escaped text for display. The Dataset table displays plain text
Scores and comments are live. The API is not an atomic snapshot. Dead/deleted/null states may change after a run. A hard platform kill can prevent final KVS writes; rows already saved remain in the Dataset

Troubleshooting

Symptom	Likely reason	Check	Action
`partial` tree	A cap stopped children	Root `treeReasons`, source `kids` and run summary	Raise the relevant input cap within allowed values, then run again.
Only `unavailable` rows	HN returned null for the IDs	`id`, `status`, `notFound`	Check the HN item link or use a current list.
Fewer comments than `descendants`	Caps, dead/deleted items or a changing source	`collectedComments`, flags and tree reasons	Keep source count and collected count separate.
Run failed before source work	Invalid input or unsupported resources	`reason`, `resourceError`	Use numeric IDs, known fields and 256 MB memory.
Source error rows	Request failures or malformed source data	Row reason, `failed`, `retries`	Retry a small input. Do not treat this as no matching stories.
Export stopped at budget	Not enough budget for the next entity	`billing.chargeLimitReached`	Check saved rows and raise the next run's cap if needed.

For developers

Sample result

One result from a real test run, in JSON. Real values change from run to run. Some values are changed for privacy. In Excel or CSV each field is a column.

one result · JSON

[
  {
    "entityType": "story",
    "id": 8863,
    "type": "story",
    "rootId": 8863,
    "storyId": 8863,
    "parentId": null,
    "depth": 0,
    "title": "My YC app: Dropbox - Throw away your USB drive",
    "hnUrl": "https://news.ycombinator.com/item?id=8863",
    "url": "http://www.getdropbox.com/u/2/screencast.html",
    "author": "example_user",
    "time": 1175714200,
    "score": 104,
    "descendants": 71,
    "status": "complete",
    "treeStatus": "complete",
    "collectedComments": 72
  },
  {
    "entityType": "comment",
    "id": 8917,
    "rootId": 8863,
    "storyId": 8863,
    "parentId": 8863,
    "depth": 1,
    "status": "complete"
  }
]

Use it from code

Actor ID: cliqtomedia/hacker-news-stories-comments-scraper. You need your Apify API token from Apify Console → Settings → API & Integrations.

Python · apify-client

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("cliqtomedia/hacker-news-stories-comments-scraper").call(run_input={
    "listMode": "top",
    "itemIds": []
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

cURL

curl -X POST "https://api.apify.com/v2/acts/cliqtomedia~hacker-news-stories-comments-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"listMode": "top", "itemIds": []}'

LangChain · langchain-apify

# pip install langchain-apify
# set APIFY_TOKEN in your environment
from langchain_apify import ApifyActorsTool

tool = ApifyActorsTool("cliqtomedia/hacker-news-stories-comments-scraper")
result = tool.invoke({"run_input": {"listMode": "top", "itemIds": []}})
print(result)

The cURL call waits for the results, up to 300 seconds. For longer runs, start a run (/runs) and get the results with a webhook or by checking the run. Apify API docs. LlamaIndex, MCP for AI agents and the JavaScript client: AI agents & MCP.

Data fields (26)

Field	Type
`entityType`	string
`id`	integer
`type`	string
`status`	string
`reason`	string
`rootId`	integer
`storyId`	integer
`parentId`	integer
`depth`	integer
`title`	string
`hnUrl`	string
`url`	string
`author`	string
`time`	integer
`createdAt`	string
`score`	integer
`descendants`	integer
`kids`	array
`textHtml`	string
`text`	string
`deleted`	boolean
`dead`	boolean
`retrievedAt`	string
`treeStatus`	string
`treeReasons`	array
`collectedComments`	integer

Input settings (7)

The main fields of the input form. With the API you send them as JSON.

Setting	Name in the API	Type	Example value
HN list	`listMode`	string	`"top"`
Exact item IDs	`itemIds`	array	`[]`
Maximum root items	`maxStories`	integer	`3`
Include comment tree	`includeComments`	boolean	`true`
Maximum comments per root	`maxCommentsPerStory`	integer	`5`
Maximum reply depth	`maxDepth`	integer	`2`
Maximum nodes in the run	`maxNodes`	integer	`30`

Use cases with this tool

Each guide shows all the steps and the other tools.

Hear what people say about you

Hacker News Stories and Comments Scraper

How to use it: 3 steps

What it does

Who uses it

Startup founders

Tech researchers

Recruiters

Price

Examples

Questions

Details

Troubleshooting

For developers

Sample result

Use it from code

Use cases with this tool

Start on Apify

Hacker News Stories and Comments Scraper

How to use it: 3 steps

What it does

Who uses it

Startup founders

Tech researchers

Recruiters

Price

Examples

Questions

Details

Troubleshooting

For developers

Sample result

Use it from code

Use cases with this tool

Similar tools

RSS & Atom Feed Reader / Monitor

Link Route Audit — Bulk HTTP Status & Redirects

Substack Newsletter Scraper

Bilibili Scraper: Videos, Comments & Creators, No Login

Bluesky Scraper — Posts, Profiles, Feeds & Threads, No Login

Medium Article Scraper: Tags, Authors & Publications

Start on Apify