Reference

Trawlly API

HTTP/JSON API for scraping and change-detection. Send a recipe, get structured data back — synchronously via /v1/scrape or on a schedule via /v1/jobs. Every public page, robots.txt respected by default, no login/paywall bypass.

Base URL

base url

https://trawlly.com/v1

All endpoints are relative to this base. Bodies are application/json. Responses are JSON.

Quickstart

Three calls: create an account, mint an API key, run a scrape.

1 · create account

curl -X POST https://trawlly.com/v1/auth/register \
  -H "Content-Type: application/json" \
  -d '{"email":"you@example.com","password":"at-least-12-chars"}'
# → 200 {"session_token":"trs_...","account":{"plan":"free", ...}}

2 · mint an API key (shown once)

curl -X POST https://trawlly.com/v1/keys \
  -H "Authorization: Bearer trs_YOUR_...KEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"my-key"}'
# → 201 {"id":7,"key":"trw_...","prefix":"trw_ab12"}   (copy the trw_ key now)

3 · run a scrape

curl -X POST https://trawlly.com/v1/scrape \
  -H "X-API-Key: ***" \
  -H "Content-Type: application/json" \
  -d @recipe.json

Authentication

Two credential types, both accepted on most endpoints:

Credential	Format	How to send
API key	`trw_...`	Header `X-API-Key: ***`
Session token	`trs_...`	Header `Authorization: Bearer trs_...`

Use the API key for /v1/scrape and production scraping. Use the session token (from register/login) for account, key and billing management. A missing or invalid credential returns 401. Requests are rate-limited to 5 req/s (burst 10) per key/IP → 429 when exceeded.

POST /v1/scrape

Runs a recipe synchronously and returns the report. The run is metered in credits (see Credits); a 402 means your monthly quota is exhausted.

Request: a recipe as the JSON body.

Response 200 — a report:

response · report

{
  "recipe": "example",
  "fetched_pages": 3,
  "pages": [
    {
      "url": "https://books.toscrape.com/",
      "status": 200,
      "data": {
        "books": [
          {"title": "A Light in the Attic", "price": "£51.77", "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"},
          {"title": "Tipping the Velvet", "price": "£53.74", "url": "..."}
        ]
      }
    }
  ]
}

Field	Type	Meaning
`recipe`	string	Your recipe `name`.
`fetched_pages`	int	Pages actually fetched (this is what's metered).
`pages[].url`	string	Page URL.
`pages[].status`	int	HTTP status for that page.
`pages[].error`	string	Set when the page failed (omitted on success).
`pages[].data`	object	Keys = your extract `name`s; each value is an array of item objects keyed by field name.

Relative URLs in extracted href/src values are resolved against the page URL. Response carries an X-Credits-Remaining header.

Recipe reference

A recipe is a JSON document (schema v1, strict — unknown keys are rejected). A minimal working example:

recipe.json

{
  "version": 1,
  "name": "books",
  "start": ["https://books.toscrape.com/"],
  "pagination": { "type": "next_link", "selector": ".next a" },
  "limits": { "max_pages": 3 },
  "extract": [{
    "name": "books",
    "item_selector": ".product_pod",
    "fields": [
      {"name": "title", "selector": "h3 a"},
      {"name": "price", "selector": ".price_color", "attr": "text"},
      {"name": "url",   "selector": "h3 a", "attr": "href"}
    ]
  }]
}

Field	Type	Required	Description
`version`	int	yes	Must be `1`.
`name`	string	yes	Shown in the report and as the job name.
`start`	string[]	*	Seed URLs. May be empty only when using `template` pagination.
`limits.max_pages`	int	no	Hard cap on fetched pages. Capped by your tier (and 100 absolute) on the sync endpoint.
`limits.concurrency`	int	no	Parallel fetches per host.
`limits.delay_ms`	int	no	Per-request delay (jittered ±25%).
`limits.timeout_ms`	int	no	Per-request timeout.
`limits.respect_robots`	bool	no	Honour robots.txt. Defaults to true.
`pagination.type`	string	no	`next_link` or `template`.
`pagination.selector`	string	†	For `next_link`: CSS selector of the "next" link.
`pagination.template`	string	†	For `template`: URL with `{n}` = page number.
`extract[].name`	string	yes	Key in `data` for this extract.
`extract[].item_selector`	string	no	CSS selector per item. Omit to treat the whole page as one item.
`extract[].fields[].name`	string	yes	Field key (unique within an extract).
`extract[].fields[].selector`	string	yes	CSS selector inside the item.
`extract[].fields[].attr`	string	no	`""`/`"text"` = text content, `"html"` = inner HTML, else a literal attribute name (`href`, `class`, `src`, …).
`render`	bool	no	Fetch via a headless browser for JS-heavy pages. Costs 5 credits/page.
`user_agent`	string	no	Override the default Trawlly user-agent.
`proxy`	string	no	`http(s)://` or `socks5://` upstream for this recipe.
`headers`	object	no	Extra request headers (Cookie, Referer, …).

* required unless template pagination. † required for the matching pagination.type.

Jobs (scheduled trawls)

A job runs a recipe on an interval and fires a signed change webhook when the extracted data changes between runs.

POST /v1/jobs — request body:

request · create job

{
  "name": "competitor-prices",
  "interval_seconds": 3600,
  "webhook_url": "https://your-server.example/hook",
  "webhook_secret": "s3cret",
  "recipe": { "version": 1, "name": "competitor-prices", "start": ["..."], "extract": [ ... ] }
}

201 → {"id": 12, "next_run_at": "..."}. 409 if a job with that name exists. 402 if your monthly job quota is reached.

GET /v1/jobs — 200 returns your jobs (secrets are never returned):

response · job list

[
  { "id": 12, "name": "competitor-prices", "interval_seconds": 3600,
    "has_webhook": true, "active": true, "next_run_at": "2026-09-06T15:00:00Z" }
]

DELETE /v1/jobs/{id} → 204 (or 404 if it doesn't exist / isn't yours).

GET /v1/runs

Your most recent scheduled-job runs, newest first. Optional ?limit=N (default 100, max 200). You only see runs of your own jobs.

response · run list

[
  { "id": 901, "job_id": 12, "job_name": "competitor-prices",
    "started_at": "2026-09-06T15:00:00Z", "finished_at": "2026-09-06T15:00:02Z",
    "pages": 3, "items": 28, "changed": true },
  { "id": 900, "job_id": 12, "job_name": "competitor-prices",
    "started_at": "2026-09-06T14:00:00Z", "finished_at": "2026-09-06T14:00:03Z",
    "pages": 0, "items": 0, "changed": false, "error": "target site returned 403" }
]

changed is true when this run's extracted data differed from the last successful run (what triggers the webhook). error is present only for failed runs.

Change webhooks

When a job's extracted data changes, Trawlly POSTs to your webhook_url. The first run is a baseline (no webhook). Verify the signature with your webhook_secret.

headers

Content-Type: application/json
X-Trawlly-Event: change.detected
X-Trawlly-Signature: sha256=5f2a...   (HMAC-SHA256 of the raw body, your webhook_secret)

Body — includes the full report so you can act on the new data:

body · change.detected

{
  "event": "change.detected",
  "job_id": 12,
  "job": "competitor-prices",
  "fingerprint": "sha256 of current extracted data",
  "previous": "sha256 of the prior successful run",
  "pages": 3,
  "items": 28,
  "report": { "recipe": "...", "fetched_pages": 3, "pages": [ ... ] }
}

Verifying the signature (Node.js):

verify · node

const crypto = require('crypto');
function verify(req) {
  const secret = process.env.TRAWLLY_WEBHOOK_SECRET;
  const sig = req.headers['x-trawlly-signature'];
  const expected = 'sha256=' + crypto.createHmac('sha256', secret)
    .update(req.rawBody).digest('hex');
  return crypto.timingSafeEqual(Buffer.from(sig), Buffer.from(expected));
}

API keys

Mint and revoke the trw_ keys used for scraping. A new key's plaintext is returned exactly once.

POST /v1/keys — body {"name":"optional"} → 201:

response · key created

{ "id": 7, "name": "my-key", "key": "trw_...", "prefix": "trw_ab12",
  "created_at": "2026-09-06T14:00:00Z" }

GET /v1/keys → 200 [{"id":7,"name":"my-key","prefix":"trw_ab12…","created_at":"2026-09-06"}] (hashes never leave the server).

DELETE /v1/keys/{id} → 204. Revoked keys stop working immediately.

Auth & account

Endpoint	Body	Result
`POST /v1/auth/register`	`{email, password}`	`200` `{session_token, expires_in_seconds, account}`; `409` if taken
`POST /v1/auth/login`	`{email, password}`	`200` same; `401` invalid
`POST /v1/auth/logout`	—	`204` (revokes the session)
`GET /v1/me`	—	`200` account + usage

The account object (returned by register/login, and the body of /v1/me):

account

{
  "email": "you@example.com",
  "plan": "free",
  "member_since": "2026-09-01T09:00:00Z",
  "usage": {
    "month": "2026-09",
    "scrapes_used": 12,
    "scrapes_limit": 100,
    "jobs_created": 1,
    "jobs_limit_month": 2
  }
}

Billing

GET /v1/billing/plans (no auth) — the live price table with per-currency purchasability:

response · plans

{
  "billing_enabled": true,
  "credit_note": "1 credit = 1 static page; JS-rendered page = 5 credits",
  "plans": [
    { "id": "free",     "price_usd": 0,   "price_inr": 0,    "credits_per_month": 100,    "jobs_per_month": 2, "max_pages_per_run": 5 },
    { "id": "starter",  "price_usd": 19,  "price_inr": 799,  "credits_per_month": 5000,   "jobs_per_month": 10, "max_pages_per_run": 25, "purchasable": {"INR": true} },
    { "id": "pro",      "price_usd": 49,  "price_inr": 1999, "credits_per_month": 50000,  "jobs_per_month": 50, "max_pages_per_run": 25, "purchasable": {"INR": true} },
    { "id": "business", "price_usd": 199, "price_inr": 7999, "credits_per_month": 250000, "jobs_per_month": 2147483648, "max_pages_per_run": 100, "purchasable": {"INR": true} }
  ]
}

POST /v1/billing/checkout — body {"tier":"pro","currency":"INR"} (currency defaults to USD). Returns a Razorpay checkout URL to redirect the customer to; the tier is granted automatically when payment completes.

response · checkout

{ "checkout_url": "https://rzp.io/...", "subscription_id": "sub_...", "tier": "pro", "currency": "INR" }

422 if that tier isn't purchasable in the chosen currency yet; 503 if billing isn't configured. Cancellation in the Razorpay dashboard downgrades you to free.

Credits & quotas

Metering: 1 credit = 1 static page fetched; a JS-rendered page (render: true) = 5 credits. Quotas reset each calendar month; hitting the ceiling returns 402 (hard stop). The launch ladder:

Plan	USD/mo	INR/mo	Credits/mo	Jobs/mo	Max pages/run
Free	$0	₹0	100	2	5
Starter	$19	₹799	5,000	10	25
Pro	$49	₹1,999	50,000	50	25
Business	$199	₹7,999	250,000	Unlimited	100

Errors

All errors are JSON: {"error": "human readable message"}.

Status	Meaning
`400`	Malformed body.
`401`	Missing or invalid credential.
`402`	Monthly credit or job quota exhausted.
`404`	Resource not found (or not yours).
`409`	Conflict — e.g. duplicate email or job name.
`422`	Validation failed (bad recipe, bad tier, …).
`429`	Rate limit exceeded (5 req/s, burst 10).
`502`	Payment provider unreachable/rejected (checkout).
`503`	Service unavailable (e.g. render backend down, billing not configured).