Data Ingestion: Architectural Patterns

Ingestion is pivotal for transferring data from a wide range of sources in its original operational context—often called the ‘operational plane‘—to the domain of analysis, or the ‘analytical plane‘. This transition is vital for maximizing the benefits of data analytics. In the first of two articles dedicated to data ingestion, this installment delves into various data ingestion patterns and their strategic implications for the ingestion process.

Patterns

  • Pattern 1: The Unified Data Repository – where a single storage system caters to both the operational application needs and analytical processing.
  • Pattern 2: Data Virtualization – leveraging specialized software to establish a virtualized data layer over multiple underlying data sources. 
  • Pattern 3: Extract Transform Load (ETL) – where data is harvested from its source (Extract), thereafter refined on an ETL server (Transform), and ultimately, the polished output is deposited into an analytics-focused database (Load).
  • Pattern 4: Extract Load Transform (ELT) – restructures and redefines the ETL process by using separate EL and T processes. 
  • Pattern 5: Push vs Pull – where the operational plane proactively sends, or ‘pushes’, data to the analytical plane as soon as changes occur, such as Create, Read, Update, and Delete (CRUD) operations.
  • Pattern 6: Streaming Processing – the continuous flow of data as it’s generated, enabling real-time processing and analysis for immediate insights. 

Read the full article on our Medium Blog.