MessNoMore Data Tools › Website Categorization API: IAB Category Lookup by Domain

How do I categorize a list of websites by IAB category, brand safety and parked status with an API?

Updated 2026-10-14 · By Wong Xian Qi

The IAB Tech Lab publishes its Content Taxonomy 3.1 for free on GitHub, but the file is only a list of categories and IDs: it does not say which category a given website belongs to. To label a domain list you have to read each site and pick a category, by hand or through an API that is not affiliated with IAB Tech Lab, such as the AI Website Categorization Actor, which returns the IAB Tier 1 and Tier 2 category with its ID, brand safety flags and a parked or dead status for every domain.

What is the IAB Content Taxonomy 3.1 and where do I get it?

It is IAB Tech Lab's tiered list of content categories, each with a unique ID. The tab-separated file is in the IAB Tech Lab Taxonomies repository on GitHub, licensed under Creative Commons Attribution 3.0. The Actor uses Tier 1 and Tier 2, which are 37 and 323 categories. Examples: ID 52 is Business and Finance, ID 596 is Technology & Computing, and the Sensitive Topics group uses alphanumeric IDs such as v9i3On. Version 3.1 has no general News category, which matters for publishers (see the limits below).

How can I look up a domain's IAB category by hand?

1. Open the site and read its homepage and About page.
2. Find the closest Tier 1 category in the taxonomy file, then its best Tier 2 child.
3. Write down the ID, not only the name.

That works for a handful of domains. On a long list the effort goes into everything around the lookup: redirects, parked domains, sites that block automated visitors and robots.txt rules that forbid reading a site.

How does the API categorize a domain?

• It checks robots.txt before every page it reads, for its WebsiteCategorizationBot user agent and the * group. A robots.txt that answers with a server error (5xx) counts as "disallow all", as RFC 9309 says.
• It reads the homepage, the About page if one is linked and, with maxPagesPerDomain set to 3, one products or services page (default 2).
• It sends the title, meta description, headings, structured data types and visible text to an OpenAI model together with the full taxonomy. The model must choose a category from the list; the Actor maps it to the official ID and validates it.
• Brand safety covers the 11 IAB Sensitive Topics plus gambling. The model's flags are topped up by the IAB category itself and by keywords in page titles and descriptions, and brandSafetyRisk (none, low, medium, high) is the highest severity found.

If a site blocks automated visitors, disallows them in robots.txt or drops the connection, the optional AI knowledge fallback (on by default) asks the model what it knows about that exact domain without visiting the site again. Those results carry classificationSource: "ai_knowledge" and a confidence capped at 0.8. Parked and non-existent domains are never classified this way.

How do I find parked domains in a list?

Every domain gets a status: ok, redirect, parked, unreachable, robots_disallowed or blocked. A domain is parked when it uses parking nameservers, lands on a domain marketplace such as Sedo, Dan or Afternic, or shows a for-sale page, a default server page or a "coming soon" placeholder. statusReason explains it, for example "Domain uses parking nameservers (afternic.com)". By hand you would check each domain's nameservers and open each site.

What does a request and result look like?

curl -X POST "https://api.apify.com/v2/acts/messnomore~website-categorization/run-sync-get-dataset-items" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"domains": ["stripe.com", "shopify.com"], "taxonomy": "iab_content_3", "includeBrandSafety": true}'

For large lists, start the run asynchronously and read the dataset when it finishes. This is a real result from 2026-10-14, trimmed to one domain:

[
  {
    "domain": "stripe.com",
    "status": "ok",
    "statusReason": null,
    "finalUrl": "https://stripe.com/",
    "httpStatus": 200,
    "title": "Stripe | Financial Infrastructure to Grow Your Revenue",
    "iabTier1": { "id": "596", "name": "Technology & Computing" },
    "iabTier2": { "id": "599", "name": "Computing" },
    "industry": "Technology / Payments infrastructure",
    "businessModel": "B2B SaaS",
    "language": "en",
    "brandSafetyRisk": "none",
    "brandSafetyFlags": [],
    "customLabel": null,
    "customLabelConfidence": null,
    "description": "Stripe provides a financial services platform offering payments processing and related financial tooling for businesses.",
    "confidence": 0.88,
    "classificationSource": "website",
    "pagesAnalyzed": ["https://stripe.com/", "https://stripe.com/about"],
    "taxonomyVersion": "IAB Content Taxonomy 3.1",
    "checkedAt": "2026-10-14T11:07:34.686Z"
  }
]

Borderline sites can land in neighbouring categories. In two sample runs on 2026-10-14, shopify.com was filed under Technology & Computing > Computing both times (confidence 0.9), while stripe.com got Business and Finance > Business (0.72) in one and Technology & Computing > Computing (0.88) in the other. The Actor's documentation suggests filtering on status = ok and a confidence of 0.7 or more.

How much does it cost to categorize domains?

Pricing is pay-per-event: $6.00 per 1,000 classified domains ($0.006 each) and $1.00 per 1,000 domains that cannot be classified ($0.001 each: parked, non-existent, or blocked or unreachable and unknown to the AI). Invalid or duplicate entries, reachable sites where the AI call failed, and internal errors are free. There is a tiny start fee per run and no charge for compute or proxies. A list of 10,000 domains where 80% can be classified costs about $50. Apify Bronze, Silver and Gold subscribers get lower prices. One run takes up to 10,000 domains, and the maximum cost you set is never exceeded.

What does it not cover?

• Tier 1 and Tier 2 only. The file's Tier 3 and Tier 4 rows are not returned.
• One category per site, judged on its main purpose. It is not a classifier for single articles.
• Only the homepage, About page and at most one more page are read. Sites that need JavaScript to show text may come back as unreachable.
• General news publishers and government portals are labelled Politics unless another topic clearly dominates; the industry field can still say "News / Broadcasting".
• Bot protection is never bypassed and CAPTCHAs are never solved. Those sites are blocked unless the fallback can classify them.
• www.example.com and example.com are checked as separate domains.
• Accuracy: in the developer's own tests on more than 150 well-known websites, about 97% got a sensible IAB category (the misses were close calls) and the language was always right. This is not an independent benchmark.

Which ready-made examples can I run?

You can also pass 2 to 100 custom labels of your own with taxonomy set to custom, at the same price. The example tasks cover the common cases:

• Get IAB content categories for a list of domains
• Check brand safety of publisher websites
• Find parked, dead and redirected domains in a list
• Categorize websites with your own labels