The model
1
Full sync (first run)
On the first sync for a connection, HunterAI fetches every entity from a start date
(see Time window) and pages to the end of each collection.
2
Incremental sync (every run after)
Later runs send
updated_since so you return only records that
changed since the last successful run.3
Idempotent upsert
Records are upserted by
(connection, id). Re-sending an unchanged record is safe and
expected — it updates the existing row in place and never creates a duplicate.The envelope
Every collection endpoint returns the same JSON shape:Only
data and has_more are read by the adapter. A missing data is treated as an
empty page; a missing/falsy has_more ends paging.Paging
HunterAI requests pages sequentially, starting atpage=1, always with limit=100.
- Support
limit=100. HunterAI always requests 100 records per page. Return up to that many; you may return fewer on the final page. - Stable ordering across pages. A record must not appear on two pages of the same run,
and pages must not repeat. HunterAI detects a page whose IDs exactly match the previous
page and fails that entity with a pagination error (a guard against a
has_morethat is never set tofalse). - Bounded page count. There is a hard safety ceiling of 5000 pages per entity per run. Exceeding it fails that entity. At 100 records/page that is 500,000 records in a single run — well beyond a normal incremental delta.
Ordering and the watermark
Return records ordered byupdated_at ascending. HunterAI tracks a per-entity
watermark — the largest updated_at it has committed — and resumes the next run from
there.
The watermark is written together with each committed page. If a sync dies mid-run, the
committed pages stay and the next run resumes from the last watermark. Ascending
updated_at order is what makes that resume correct: HunterAI can advance the watermark
page by page without risking skipping an older record on a later page.updated_at must change on every modification of a record — including
a stage move, an owner change, a field edit, or a soft-delete. If updated_at does not
advance, the change falls before the watermark and is never pulled again.
Incremental sync
Five endpoints receive anupdated_since query parameter and must filter on it:
/contacts · /leads · /tasks · /calls · /events
- Return records whose
updated_atis greater than or equal toupdated_since(inclusive). Re-returning the exact boundary record is expected and harmless — the upsert is idempotent. - The value is an ISO-8601 timestamp with an offset. Filter on your record’s
updated_at. - HunterAI decides the
updated_sincevalue on its side — the first run uses the start date, later runs resume from the per-entity watermark (and may add a small safety margin). You do not need to reason about the exact value: just filterupdated_at >= updated_sinceand return the matches in ascending order. Because the boundary is inclusive, HunterAI may re-send you the last record of the previous run; the idempotent upsert makes that harmless.
/pipelines and /users do not receive updated_since. They are always fetched in
full — return your complete current list every time. These collections are expected to be
small (tens to low hundreds of rows).
Time window
The start of a full sync is chosen on HunterAI’s side:- If your business owner set a sync-since date on the connection, that date is used.
- Otherwise HunterAI defaults to 1 January of the current year — a guard so a CRM with years of history does not pull its entire archive on the first run.
updated_since to the five incremental
endpoints on the first run and advances by watermark from then on.
Entity order and dependencies
Entities sync in a fixed, dependency-safe order:Stages come from
/pipelines. HunterAI reads /pipelines once for the pipelines
themselves and again to flatten their nested stages[]. There is no separate stages
endpoint.kind, so
pipelines and their stages sync before leads. If a lead references a pipeline/stage that
was not synced (for example a partial sync of only leads), HunterAI cannot classify it —
the lead lands with is_won = is_lost = false, and a warning is logged. It is never a
silent wrong-number: unresolved leads are counted and surfaced in the run summary.
Partial success
Each entity syncs independently, and within an entity each page (up to 100 records) is validated and written as one transaction. One malformed record fails its whole page. If any record on a page cannot be processed — a missingid, a non-numeric price — none of the records on that page are saved. This
is page-level, not per-record. Pages that committed earlier in the run stay committed.
A failed entity does not always fail the sync. If an entity fails — a page 500s past
its retries, a pagination guard trips, or a required endpoint returns 404 — the entities
that already committed stay committed, the failed entity is reported, and the next run
resumes each entity from its own watermark. Whether the overall sync is marked failed
depends on whether that entity was required:
- A required entity (
/pipelines,/users,/contacts,/leads) failing marks the whole sync failed. - An optional entity (
/tasks,/calls,/events) failing takes down only that entity. It is reported failed, but the overall sync still succeeds — that is the whole point of the required/optional distinction.