Tag: ETL
Ryan Van Fleet
Ryan Van Fleet
11 Jul, 2019

Data Identification and Persistence – Extract Transform and Load Part 5

The final two steps, identification and persistence, can make or break the data chain. Identification handles the deduplication of data and informs the system of it should be persistence. Understanding how data will be used by the system informs if the data is Single, Submitter, Transaction Source. Once data...
Ryan Van Fleet
Ryan Van Fleet
09 Apr, 2019

Business Rules – Extract Transform and Load Part 4

In the last article of the Extract Transform and Load (ETL) series, we talked about how using a canonical object can decouple data sources from the downstream components, making the pipeline reusable and easily maintained.  In this article we will discuss one of these downstream components, the business rules....
Ryan Van Fleet
Ryan Van Fleet
19 Mar, 2019

Reusability and Sustainability Using a Canonical – Enterprise Transform and Load Part 3

Overview In the last article in my Extract Transform and Load (ETL) series, Submission Information Packages and High-level Validation, I covered the most important part of this process: the data. The data format will drive 80 percent of the design decisions for the ETL pipeline. In this article, I...
Ryan Van Fleet
Ryan Van Fleet
26 Feb, 2019

Extract Transform and Load Part 2 – Submission Information Packages and High-level Validation

The first two major pieces of an Extract Transform and Load pipeline are the Submission Information Package (SIP) and High-level validation. The major components as well as the benefits and usages of Extract Transform and Load where discussed in the first article in the series. It is recommended that...
Ryan Van Fleet
Ryan Van Fleet
18 Dec, 2018

Extract Transform and Load Overview

Part 1 of the Data Ingest Series The process of extract transforms and load (ETL) goes by many names: Data acquisition Data ingest Enterprise transform and load But they all are about getting external data into the system. The problem that most businesses face is that there are no...
Jason Mao
Jason Mao
05 Oct, 2017

The Steady Stream of New Products from AWS

Amazon Web Services (AWS) has long been the leader in Infrastructure-as-a-Service (IaaS) and Platform-as-a-Service (PaaS). However, there’s been a veritable Cambrian explosion of new Software-as-a-Service (SaaS) and Function-as-a-Service (FaaS) products coming out of the AWS teams that further embrace AWS’s serverless/pay-as-you-go-ism. In the last 24 months or so, AWS...