> ## Documentation Index
> Fetch the complete documentation index at: https://integration.hunterai.uz/llms.txt
> Use this file to discover all available pages before exploring further.

# How sync works

> A one-directional pull model: full sync, then incremental by updated_since, terminated by has_more and driven by a per-entity watermark.

HunterAI **pulls** from your API. There are no webhooks and no write-back — the data flows
one way, CRM → HunterAI, and your endpoints are read-only.

## The model

<Steps>
  <Step title="Full sync (first run)">
    On the first sync for a connection, HunterAI fetches every entity from a start date
    (see [Time window](#time-window)) and pages to the end of each collection.
  </Step>

  <Step title="Incremental sync (every run after)">
    Later runs send [`updated_since`](#incremental-sync) so you return only records that
    changed since the last successful run.
  </Step>

  <Step title="Idempotent upsert">
    Records are upserted by `(connection, id)`. Re-sending an unchanged record is safe and
    expected — it updates the existing row in place and never creates a duplicate.
  </Step>
</Steps>

## The envelope

Every collection endpoint returns the same JSON shape:

```json theme={null}
{
  "data": [ { "id": "…" }, { "id": "…" } ],
  "has_more": true,
  "page": 1,
  "limit": 100
}
```

| Field      | Type    | Who sets it | Meaning                                                                           |
| ---------- | ------- | ----------- | --------------------------------------------------------------------------------- |
| `data`     | array   | you         | The page of records. `[]` on an empty page is fine. **Required.**                 |
| `has_more` | boolean | you         | `true` if at least one more page exists. **Required** — this is what ends paging. |
| `page`     | number  | you (echo)  | Optional. HunterAI ignores it; echo it back for your own debugging.               |
| `limit`    | number  | you (echo)  | Optional. HunterAI ignores it; the requested value is authoritative.              |

<Note>
  Only `data` and `has_more` are read by the adapter. A missing `data` is treated as an
  empty page; a missing/falsy `has_more` ends paging.
</Note>

## Paging

HunterAI requests pages sequentially, starting at `page=1`, always with `limit=100`.

```http theme={null}
GET /leads?limit=100&page=1&updated_since=2025-07-01T00:00:00+00:00
GET /leads?limit=100&page=2&updated_since=2025-07-01T00:00:00+00:00
…
```

<Warning>
  **`has_more` terminates paging — not an empty `data` array.** HunterAI keeps requesting
  the next page as long as `has_more` is truthy. Set `has_more: false` on the last page.
</Warning>

Rules you must honour:

* **Support `limit=100`.** HunterAI always requests 100 records per page. Return up to
  that many; you may return fewer on the final page.
* **Stable ordering across pages.** A record must not appear on two pages of the same run,
  and pages must not repeat. HunterAI detects a page whose IDs exactly match the previous
  page and **fails that entity** with a pagination error (a guard against a `has_more` that
  is never set to `false`).
* **Bounded page count.** There is a hard safety ceiling of **5000 pages** per entity per
  run. Exceeding it fails that entity. At 100 records/page that is 500,000 records in a
  single run — well beyond a normal incremental delta.

## Ordering and the watermark

Return records **ordered by `updated_at` ascending**. HunterAI tracks a per-entity
*watermark* — the largest `updated_at` it has committed — and resumes the next run from
there.

<Info>
  The watermark is written together with each committed page. If a sync dies mid-run, the
  committed pages stay and the next run resumes from the last watermark. Ascending
  `updated_at` order is what makes that resume correct: HunterAI can advance the watermark
  page by page without risking skipping an older record on a later page.
</Info>

Because of this, **`updated_at` must change on every modification** of a record — including
a stage move, an owner change, a field edit, or a soft-delete. If `updated_at` does not
advance, the change falls before the watermark and is never pulled again.

<Warning>
  A stage move that does not bump `updated_at` will be **missed** by every incremental
  sync. Funnel analytics depend on stage changes; make sure any mutation touches
  `updated_at`.
</Warning>

## Incremental sync

Five endpoints receive an `updated_since` query parameter and must filter on it:

`/contacts` · `/leads` · `/tasks` · `/calls` · `/events`

```http theme={null}
GET /leads?limit=100&page=1&updated_since=2025-07-01T00:00:00+00:00
```

* Return records whose `updated_at` **is greater than or equal to** `updated_since`
  (inclusive). Re-returning the exact boundary record is expected and harmless — the
  upsert is idempotent.
* The value is an ISO-8601 timestamp with an offset. Filter on your record's `updated_at`.
* HunterAI decides the `updated_since` value on its side — the first run uses the
  [start date](#time-window), later runs resume from the per-entity watermark (and may add
  a small safety margin). You do not need to reason about the exact value: just filter
  `updated_at >= updated_since` and return the matches in ascending order. Because the
  boundary is inclusive, HunterAI may re-send you the last record of the previous run; the
  idempotent upsert makes that harmless.

`/pipelines` and `/users` do **not** receive `updated_since`. They are always fetched in
full — return your complete current list every time. These collections are expected to be
small (tens to low hundreds of rows).

## Time window

The start of a full sync is chosen on HunterAI's side:

* If your business owner set a **sync-since** date on the connection, that date is used.
* Otherwise HunterAI defaults to **1 January of the current year** — a guard so a CRM with
  years of history does not pull its entire archive on the first run.

Either way, HunterAI passes that date as `updated_since` to the five incremental
endpoints on the first run and advances by watermark from then on.

## Entity order and dependencies

Entities sync in a fixed, dependency-safe order:

```
pipelines → pipeline stages → users → contacts → leads → tasks → calls → events
```

<Note>
  **Stages come from `/pipelines`.** HunterAI reads `/pipelines` once for the pipelines
  themselves and again to flatten their nested `stages[]`. There is no separate stages
  endpoint.
</Note>

The one hard dependency: **leads are classified won/lost from the stage's `kind`**, so
pipelines and their stages sync before leads. If a lead references a pipeline/stage that
was not synced (for example a partial sync of only `leads`), HunterAI cannot classify it —
the lead lands with `is_won = is_lost = false`, and a warning is logged. It is never a
silent wrong-number: unresolved leads are counted and surfaced in the run summary.

## Partial success

Each entity syncs independently, and within an entity each **page** (up to 100 records) is
validated and written as one transaction.

**One malformed record fails its whole page.** If any record on a page cannot be processed
— a missing `id`, a non-numeric `price` — none of the records on that page are saved. This
is page-level, not per-record. Pages that committed earlier in the run stay committed.

**A failed entity does not always fail the sync.** If an entity fails — a page 500s past
its retries, a pagination guard trips, or a required endpoint returns `404` — the entities
that already committed **stay committed**, the failed entity is reported, and the next run
resumes each entity from its own watermark. Whether the *overall* sync is marked *failed*
depends on whether that entity was required:

* A **required** entity (`/pipelines`, `/users`, `/contacts`, `/leads`) failing marks the
  whole sync *failed*.
* An **optional** entity (`/tasks`, `/calls`, `/events`) failing takes down only that
  entity. It is reported failed, but the overall sync still **succeeds** — that is the
  whole point of the required/optional distinction.