Keep a collection up to date from cloud storage

Keep collections in Customer.io up to date by exporting data from your systems to a bucket in Amazon S3, Google Cloud Storage, or Azure Blob Storage, and Customer.io automatically pulls in the changes.

How it works

To keep a collection up to date with data that changes in your systems, like a product catalog or a list of store locations, your systems can write files to a bucket. Customer.io then loads each file to add, update, and delete rows in the collection.

graph LR A[Your export job] -->|writes a file| B[Folder in<br>your bucket] B -->|Customer.io checks<br>on a schedule| C[Sync] C -->|adds, updates, and<br>deletes rows by key| D[Collection] C -->|records| E[Sync status and<br>rejected rows]

Customer.io can load files from your bucket into a collection in two ways:

  • A sync: Customer.io checks a folder in your bucket on a schedule and loads each new or changed file that your systems write there. Use a sync when your systems export on a schedule. Most of this page covers syncs.
  • A write request: you use the API to load a specific file into a collection. Use a write request when your systems need to know that a particular file loaded before they do something else. See Load a file when you’re ready.

When Customer.io loads a file, it doesn’t replace your collection. Instead, Customer.io uses the collection’s key field, which works like a primary key in a database table, to find the row that each row in the file changes:

  • Add: If a row’s key doesn’t match an existing row, Customer.io adds the row to the collection.
  • Update: If a row’s key matches an existing row, the new row replaces the whole existing row.
  • Delete: If a row sets the collection’s delete field to true, Customer.io removes the row with that key.
  • No change: Rows that a file doesn’t include stay in the collection as they are.

Before you begin

You need the following things:

  • A value that’s unique to each row. This becomes the collection’s key field, like a product’s sku. The value can’t be empty, and no two rows can share a value.
  • Credentials that can list and read the bucket. Customer.io only reads your bucket. It never deletes, moves, or renames your files.
  • An export job that writes files to the bucket, or access to change the one you have. You’ll set the job up to match the steps on this page.

Prepare your exports

Before you set up a sync, set up your export job, the process in your own systems that writes the collection’s data to your bucket. A sync can only load the files that your export job writes, so the job needs to follow the rules in this section.

How a sync reads your files

A sync works like a reader that keeps a list of the files it already loaded. At each scheduled sync interval, Customer.io lists every file in the folder you tell Customer.io to check, including the folders inside it. Customer.io loads any new or updated files since the previous sync. Your export job depends on these rules:

  • Customer.io loads files one at a time, in alphabetical order of their paths. When two files change the same row, the file that loads last wins.
  • Customer.io loads every file that’s in the folder when you start the sync. The folder’s existing files load first, in the same alphabetical order.
  • An overwritten file loads again. If you overwrite a file more than once between checks, Customer.io only loads the latest version.
  • A file that already loaded doesn’t affect the collection after that. Deleting the file from the folder doesn’t remove its rows. If you put a deleted file back, Customer.io loads it again.
  • Customer.io loads up to 2,000 new or changed files per check. If more than 2,000 files are waiting, Customer.io loads the first 2,000 and loads the rest at later checks.
  • A file that Customer.io can’t load stops the sync. Customer.io doesn’t load the files that sort after it until you fix the failed file. If only some rows in a file are invalid, like a row without a key, Customer.io loads the rest of the file and records the rejected rows.

Mark deleted rows

Customer.io only changes the rows that a file includes, so leaving a row out of an export doesn’t remove it from the collection. To remove a row, your export needs to include it with a field that marks it as deleted.

If you use a field called deleted to mark deleted rows, you can delete rows by including "deleted": true in the row:

JSON
{ "sku": "A-100", "deleted": true }

Customer.io removes the row with that key and never stores the row that marked it. Rows that set deleted to false, or leave it out, load as usual. Customer.io only removes rows this way after you set the same field as the collection’s delete field.

We recommend that each export includes only the rows that your systems added, changed, or deleted since the last export. Files stay small, so Customer.io loads them quickly.

Organize your files

Pick a folder for the collection’s files, like products/, and keep only your collection’s files in it. Customer.io tries to load every file in the folder and the folders inside it, so a file for another system can stop your sync. Then choose a naming convention for your files:

  • Write a new file each time. Start each filename with the date and time, like products/2026-11-04T0600-products.csv, so that alphabetical order matches the order that you wrote them. This approach keeps a history of your exports in the bucket.
  • Overwrite the same file each time. For example, your export job can replace products/latest.csv. Customer.io loads the file again at the next check.

A folder can hold up to 100,000 files

If your folder is over this limit, Customer.io stops the sync and doesn’t load any files until you move files out of the folder. Moving loaded files out of the folder doesn’t change the collection, so you can remove or archive loaded files on a schedule with a lifecycle rule in your bucket.

File requirements

Each file needs to follow these rules:

  • Use JSON or CSV. Customer.io reads every CSV value as a string.
  • Include each key once per file. If more than one row has the same key, Customer.io loads the first row and rejects the rest.
  • Be less than 500 MB and 1,000,000 rows. If a file is over these limits, Customer.io can’t load it and stops the sync at that file. If you have larger files, you can split them into smaller files with names that sort in order, like 2026-11-04T0600-part-01.csv and 2026-11-04T0600-part-02.csv.

Export when your messages aren't using the collection

Customer.io loads a file at the next check after you export it, not the moment the file lands in your bucket. Customer.io also saves a file 500 rows at a time. While a large file loads, a message that queries the collection can get some rows from the new file and some from your previous export. To avoid sending a mix of old and new data, schedule your exports for a time when few messages use the collection. If a message depends on data from a specific export, send it after the Last file loaded in the Sync panel shows that export’s file.

Set up a sync in Customer.io

After your export job is ready, you set up the sync in Customer.io:

  1. Set a key field so that Customer.io can match each row in a file to a row in the collection.
  2. Set a delete field so that your exports can remove rows.
  3. Connect your bucket and start the sync.
  4. Check that your first export loaded.

Step 1: Set a key field

The key field tells Customer.io which value represents a unique row, like a product sku or a store id. You must set a key field to sync a collection.

  1. Under Configure data, click Collections, and go to your collection.
  2. Click next to Key field.
  3. Choose a Key field from the attributes in your collection’s rows, like sku.
  4. Leave the window open to set a delete field, or click Save.

When you set the key field, Customer.io checks every row in the collection first. If any row is missing a key or repeats another row’s key, Customer.io keeps the old key field and tells you which rows you need to fix.

Step 2: Set a delete field

A sync only deletes rows if you set a delete field. Set it to the field that your exports use to mark deleted rows, like deleted.

  1. In the Edit key fields window, type the name of the field that your exports use, like deleted, in Delete field. The field doesn’t need to be in your rows yet.
  2. Click Save.
The Edit key fields window, with sku as the key field and deleted as the delete field.

When you set a delete field, Customer.io permanently deletes every row in the collection where that field is already true. Check your rows for the field before you save.

Step 3: Connect your bucket and start the sync

  1. On the collection’s page, click Set up sync in the Sync panel.

  2. Under Bucket, choose a bucket that you connected before, or choose Connect a new bucket. For a new bucket, choose the Storage provider and give the bucket a Name. Then enter credentials that can list and read the bucket:

    Enter the bucket name, its Region, and the Access key ID and Secret access key of an AWS user.

    Enter the bucket name and the Service account key, which is the JSON key file of a service account.

    Enter a SAS URL for the container that grants read and list permissions.

  3. In Folder, enter the folder that your export job writes to, like products/. If you leave it empty, Customer.io loads files from the whole bucket.

  4. Under Check for new files, choose how often Customer.io checks the folder, every 1 to 24 hours.

  5. Click Save. Customer.io reads a file from a new bucket before it saves the connection, so you find out right away if your credentials don’t work. If the bucket doesn’t have any files yet, Customer.io can only check that your credentials can list the bucket.

The Set up sync window, connecting a new Amazon S3 bucket and loading files from the products/ folder every hour.

Step 4: Check that your first export loaded

  1. Run your export job, if it hasn’t written a file to the folder yet.
  2. Click Check now in the Sync panel, so that you don’t have to wait for the next scheduled check.
  3. Check that the Last file loaded shows your export’s file, and that Rejected rows is 0. If it isn’t, see Monitor a sync.

Monitor a sync

The Sync panel shows the sync’s status, the Last file loaded and when it loaded, the Next check, and the number of Rejected rows.

The Sync panel on a collection, showing an active sync with its bucket, folder, last file loaded, next check, and one rejected row

The panel can show you problems with your sync:

  • The status is Failed. The sync stopped at a file that Customer.io couldn’t load. The panel shows which file failed and why. Customer.io doesn’t load the files that sort after it until you fix the failed file.
  • Your latest export didn’t load. If a check passes after your export and the Last file loaded still shows an earlier file or time, Customer.io didn’t find your file. Usually, your export job wrote the file outside the sync’s folder.
  • The sync rejected rows. Customer.io loads the rest of a file when some rows are invalid, like a row without a key, so the status stays Active. Click View rejected rows to see each row’s file, row number, key, and the reason Customer.io rejected it. In a CSV file, the header row doesn’t count, so row 1 is the first row of data. When Customer.io loads a file again, it clears the rejected rows from the file’s earlier version. Customer.io keeps rejected rows for 90 days.

Customer.io doesn’t send you a notification when it loads a file. To alert on these problems automatically, you can check the sync with the API after each export.

Fix a failed sync

When a sync fails, the Sync panel shows why. A sync can fail for these reasons:

  • Customer.io can’t load a file. Customer.io stops at the file and doesn’t load the files that sort after it. At each check after that, Customer.io tries the same file again. To get the sync going again, overwrite the file with one that Customer.io can load, or delete the file.
  • The folder holds more than 100,000 files. Move files that Customer.io already loaded out of the folder.
  • You deleted the bucket connection that the sync uses. Click Edit sync and choose another bucket.

Then click Check now to check the folder right away.

Change or stop a sync

To change a sync, click Edit sync in the Sync panel. Your changes affect which files Customer.io loads:

  • Changing how often Customer.io checks keeps the sync’s place.
  • Changing the bucket or the folder starts the sync over. Customer.io loads every file in the folder again, starting from the first one. While the earlier files load, rows can show older values, and a large folder can take several checks to load. After Customer.io loads the newest file again, the rows match your latest export. If you changed any of those rows with write requests since then, the files overwrite your changes. To avoid loading every file again, move the files that Customer.io already loaded out of the folder before you make the change.

To stop a sync, click > Stop sync. The rows that the sync already loaded stay in the collection. Customer.io also clears the list of rejected rows, so look at them before you stop the sync.

Rotate your bucket’s credentials

You can’t change the credentials of a bucket that you connected. When you rotate a key, you need to connect the bucket again with the new credentials. If you want to avoid loading every file again, move the files that Customer.io already loaded out of the folder.

  1. Go to your sync and click Edit sync.
  2. Choose Connect a new bucket, and enter the new credentials.
  3. Delete the old connection with DELETE:/v1/collections/sources/:id. You can only delete a connection with the API.

If you only changed the bucket’s permissions, use POST:/v1/collections/sources/:id/check to confirm that Customer.io can still read the bucket.

If you delete a connection that a sync uses, the sync stops loading files, and the Sync panel tells you that the bucket was removed. Edit the sync to choose another bucket.

Use the API

Everything on this page also works with the App API, so you can set up syncs from your own deployment scripts and monitor them. Use a private key (sk_) from the API Credentials page as a bearer token. Use https://api.customer.io in the US region or https://api-eu.customer.io in the EU region.

Set up a sync with the API

  1. Set the key field and delete field with PUT:/v1/collections/:id:

    JSON
    {
      "key_field": "sku",
      "delete_field": "deleted"
    }

    Customer.io queues the change as a write request and responds with a write that has an id. Check GET:/v1/collections/:id/writes/:write_id until the state is applied. If the state is failed, the message names a row that’s missing a key or repeats another row’s key.

  2. Connect the bucket with POST:/v1/collections/sources. You can set a prefix on the connection, so that Customer.io can only read files in that folder of the bucket. This example connects an Amazon S3 bucket:

    JSON
    {
      "name": "Product exports",
      "provider": "s3",
      "bucket": "acme-exports",
      "region": "us-east-1",
      "prefix": "collections/",
      "credentials": {
        "access_key_id": "AKIAIOSFODNN7EXAMPLE",
        "secret_access_key": "<your secret access key>"
      }
    }

    The response includes the connection’s id. See the API reference for the fields that Google Cloud Storage and Azure Blob Storage need.

  3. Start the sync with PUT:/v1/collections/:id/sync. Set prefix to the folder that holds this collection’s files, relative to the connection’s prefix. Set poll_hours to how often Customer.io checks the folder, from 1 to 24 hours:

    JSON
    {
      "source_id": "01JD8X4Q6R2W7N3K5M9P1T0VZC",
      "prefix": "products/",
      "poll_hours": 1
    }

To check the folder right away, use POST:/v1/collections/:id/sync/poll. To stop the sync, use DELETE:/v1/collections/:id/sync.

Check the sync with the API

GET:/v1/collections/:id/sync shows the sync’s state, the last file that Customer.io loaded, and the most recent error:

JSON
{
  "sync": {
    "collection_id": 12,
    "source_id": "01JD8X4Q6R2W7N3K5M9P1T0VZC",
    "prefix": "products/",
    "poll_hours": 1,
    "state": "active",
    "last_file": {
      "path": "products/2026-11-04T0600-products.csv",
      "applied_at": "2026-11-04T06:31:12Z"
    },
    "next_run_at": "2026-11-04T07:31:12Z"
  }
}

Have your monitoring alert when:

  • The state is failed. The last_error shows which file failed and why.
  • More than poll_hours passed since your export, and last_file.applied_at is still earlier than your export.
  • GET:/v1/collections/:id/sync/rejects returns rejected rows. Each one includes the row’s file, row number, key, and code.

Load a file when you’re ready

If your systems need to know when a particular file loads, send a write request for the file instead of using a sync. For example, your system might need to wait for a price update to load before it triggers a sale announcement. You need the id of a connected bucket, which you can get from GET:/v1/collections/sources.

  1. Upload the file to a folder in the bucket that a sync doesn’t check, like on-demand/. A sync loads every file in its folder, so it would load your file again at its next check and could undo changes that you made since.

  2. Send the connection’s id and the file’s path to POST:/v1/collections/:id/writes. The path is relative to the connection’s prefix.

    JSON
    {
      "source_id": "01JD8X4Q6R2W7N3K5M9P1T0VZC",
      "path": "on-demand/2026-11-04-price-update.csv"
    }
  3. Check GET:/v1/collections/:id/writes/:write_id with the id from the response until the state is no longer queued.

When the state is applied, Customer.io loaded the file. When the state is failed, the code explains why. Customer.io checks every row before it saves anything, so if the code is items, the items list shows each invalid row, and Customer.io didn’t save any rows from the file. Fix those rows, upload the file again, and send another write request. Because each row replaces the row with the same key, you can send the same file more than once, and the collection ends up the same.

A write request can read a file from any folder under the connection’s prefix, and it loads a file even if you loaded it before. Customer.io keeps the result of each write request for 24 hours.

Use a sync and write requests together

You can send write requests to a collection that has a sync. Customer.io applies every change to a collection one at a time, in the order that it receives them. A sync loads a file as a series of writes of 500 rows, so a write request that you send while a file loads can land in the middle of the file. When a write request and a file change the same row, the change that Customer.io receives last wins.

To keep the results predictable, have the sync and your write requests change different rows. For example, let the sync load the products from your nightly catalog export, and use write requests for promotional rows that your export doesn’t include.

Updated November 16, 2026