User Guide & Troubleshooting

Everything you need to get started, build recipes, debug issues, and scale your scraping.

Getting Started

From zero to your first structured dataset in 5 minutes.

Create your account

Option A — on the website: open /signup, enter your email and password, and click Create Account. Your browser stores a session token automatically and redirects you here.

Option B — via API (curl / any HTTP client): you can also register directly from your terminal. Copy and run this, replacing the email and password:

create account

curl -X POST https://trawlly.com/v1/auth/register \
  -H "Content-Type: application/json" \
  -d '{"email":"you@example.com","password":"your-secure-password"}'

Response contains session_token (starts with trs_) and your plan. Save it — you need it for the next step. Existing users: use POST /v1/auth/login with the same body to get a fresh token.

Generate an API key

Your API key is what you send with every scrape request. Generate it by calling POST /v1/keys with your session token in the Authorization header. Trawlly accepts both a session token and an API key for auth — but for scraping you need the trw_ key.

generate API key

curl -X POST https://trawlly.com/v1/keys \
  -H "Authorization: Bearer trs_YOUR_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"my-first-key"}'

Response: {"id": 123, "key": "trw_..."}. Copy the trw_... value immediately — it is shown only once. Use it as X-API-Key: trw_... on all scrape and job calls. You can create multiple keys and revoke them via DELETE /v1/keys/{id}.

Write your first recipe

A recipe is a JSON file that tells Trawlly what to scrape. Save it as recipe.json — click Copy and paste:

recipe.json — select all & copy

version: 1,
name: "example",
start: ["https://books.toscrape.com/"],
pagination: { "next_link": ".next a" },
limits: { "max_pages": 3 },
extract: [{
  "name": "books",
  "item_selector": ".product_pod",
  "fields": [
    {"name": "title", "selector": "h3 a", "attr": "text"},
    {"name": "price", "selector": ".price_color", "attr": "text"},
    {"name": "url", "selector": "h3 a", "attr": "href"}
  ]
}]

Run the scrape

Send your recipe to the API — replace trw_... with your actual API key from step 2:

run scrape — copy & paste

curl -X POST https://trawlly.com/v1/scrape \
  -H "X-API-Key: trw_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d @recipe.json

You'll get a JSON response with items (your extracted data), fetched (pages fetched), and credits_used.

Schedule it (optional)

Want this to run automatically? POST the same recipe to /v1/jobs with an interval (minimum 60 seconds). Trawlly will run it on schedule, fingerprint results, and send HMAC-signed webhooks when data changes.

Recipe Reference

All available fields in a v1 recipe.

Top-level fields

Field	Required	Description
`version`	Yes	Schema version. Currently `1`.
`name`	Yes	Human-readable identifier for your recipe.
`start`	Yes*	Array of starting URLs. Can be empty if using `template` pagination.
`pagination`	No	Object with `next_link` (CSS selector) or `template` (URL with `{n}`).
`limits`	No	Object with `max_pages` (default 25).
`render`	No	Boolean. `true` = JavaScript rendering (5× credits).
`user_agent`	No	Custom User-Agent string.
`proxy`	No	Proxy URL (http/https/socks5).
`headers`	No	Object of extra HTTP headers.
`extract`	Yes	Array of extraction rules (see below).

Extract rule fields

Field	Required	Description
`name`	Yes	Output key for this extractor's results.
`item_selector`	Yes	CSS selector matching each repeating item on the page.
`fields`	Yes	Array of field objects (see below).

Field object fields

Field	Required	Description
`name`	Yes	Output key for this field.
`selector`	Yes	CSS selector relative to `item_selector`.
`attr`	No	`text` (default), `html`, or any attribute name (`href`, `src`, etc.).
`transform`	No	Regex with capture group: `"transform": "(\\d+)"` extracts first group.
`multiple`	No	Boolean. `true` returns array of matches; `false` (default) returns first match.

Troubleshooting

Common issues and how to fix them.

Getting 401 Unauthorized

Your API key is missing, invalid, or revoked. Check that you're sending X-API-Key: trw_... header. Regenerate a key via POST /v1/keys if needed.

Getting 402 Payment Required

You've exhausted your monthly credit quota. Check the X-Credits-Remaining header on responses. Upgrade your plan at /v1/billing/checkout or wait for the monthly reset.

Getting 422 Invalid Recipe

Your recipe JSON failed validation. Common causes: unknown fields, missing required fields (version, name, start or pagination, extract), invalid CSS selectors, duplicate field names, or template pagination without {n} placeholder.

Getting 429 Too Many Requests

Rate limit exceeded (5 requests/second, burst 10). Implement exponential backoff and respect the Retry-After header if present.

Empty or incomplete results

Check your CSS selectors against the live page (use browser DevTools). The page might use JavaScript rendering — add "render": true to your recipe (costs 5× credits). Also verify the target site isn't blocking via robots.txt (enabled by default).

JavaScript rendering fails / times out

Heavy SPAs may exceed the 30s render timeout. Try increasing limits.max_pages to 1 for single-page apps, or simplify the page by blocking unnecessary resources via proxy. Ensure the target doesn't require login (Trawlly doesn't support authenticated scraping).

Scheduled job not running

Check: interval ≥ 60s, recipe is valid, DATABASE_URL is configured on the server, and the scheduler is active (check /healthz for scheduler status). Jobs with errors are retried but won't block the scheduler.

Webhook not received

Verify your endpoint accepts POST, returns 2xx within 10s, and validates the X-Trawlly-Signature header (HMAC-SHA256 of body with your webhook secret). Check server logs for delivery attempts.

Credit usage seems off

Remember: static pages = 1 credit, JS-rendered = 5 credits. Each page fetched counts, including paginated pages. The credits_used in the response and X-Credits-Remaining header reflect actual usage.

Best Practices

Get the most out of Trawlly with these patterns.

Use specific selectors

Avoid overly broad selectors like div or *.product. Target the exact repeating element (.product-card, tr.listing) to avoid extracting noise.

Set reasonable `max_pages`

Start small (5–10) during development. Large crawls consume credits fast and increase chance of hitting rate limits or target-site blocks.

Prefer `next_link` over `template`

next_link follows the site's actual pagination, handling dynamic page counts. template is fragile if the site changes URL structure.

Use `transform` for cleanup

Extract raw text then clean it with regex: "transform": "([\\d,]+\\.\\d{2})" pulls just the price number from "$29.99 USD".

Monitor via webhooks

Don't poll for changes. Set up a job with a webhook URL — Trawlly sends HMAC-signed payloads only when extracted data actually changes (SHA-256 fingerprint diff).

Rotate proxies for scale

For high-volume scraping, configure a proxy pool in your recipe ("proxy": "http://user:pass@host:port"). Trawlly uses one browser instance per unique proxy config.

Ready to build your first recipe?

Start free — 100 credits/month, no card required.

Create Free Account →