Incremental Loading

Incremental Loading (Delta Loading)

Introduction

Incremental loading, also known as delta loading, is a method that brings in only the data that has changed since the last load. Instead of reloading all data every time, incremental loading identifies new or updated records and loads just those. This makes the data loading process faster and more efficient.

Key Concepts

Full (Historic) Load

A full load brings in all available data from a source (for example, a database, API, or application). This includes both past and present records. When you load data into Precog for the first time, a full load is required. This is called the initial load.

Incremental Load

An incremental load only brings in records that have been added or changed since the last load. This is achieved by tracking an Incrementing Field and a field (or fields) with a Unique Identifier per record. 

Incremental loading requires:

• One incrementing field (to detect new/changed records).
  
  
• One or more unique identifier fields (to tell records apart).
  
  

Incrementing Field

• Tracks when records are new or updated.
  
  
• Doesn’t have to be unique — just needs to show progression.
  
  
• Everyday choices: Date, DateTime, or numeric fields.
  
  

Unique Identifier Field

• Ensures each record is distinct.
  
  
• Helps Precog know whether to insert a new record or update an existing one.
  
  
• Often the same as a primary key (like Id).
  
  

Example with Shopify data:
Suggests fields for incremental loading with Shopify data. For the Incrementing Field, the “Updated At” field works well since it is updated whenever records are modified. For the Unique Identifier Field, the “Id“ works well since this field is unique for each customer. If a record with the same “Id“ comes in again but with a newer timestamp, the old record will be replaced with the new one.