Tag: ETL
03 Dec, 2019
Kitchen Sink: Audit/Governance/Work Queue – Extract Transform and Load Part 6
In the previous articles, we discussed the main pieces to consider when designing and implementing an Extract Transform and Load pipeline. Submission Information Packages and High-level Validation: Data, End Points, and validation Canonical Transformation: Converting data to a common object base Business Rules: Low-level validation Identification: Does this data...
11 Jul, 2019
Data Identification and Persistence – Extract Transform and Load Part 5
The final two steps, identification and persistence, can make or break the data chain. Identification handles the deduplication of data and informs the system of it should be persistence. Understanding how data will be used by the system informs if the data is Single, Submitter, Transaction Source. Once data...
09 Apr, 2019
Business Rules – Extract Transform and Load Part 4
In the last article of the Extract Transform and Load (ETL) series, we talked about how using a canonical object can decouple data sources from the downstream components, making the pipeline reusable and easily maintained. In this article we will discuss one of these downstream components, the business rules....
19 Mar, 2019
Reusability and Sustainability Using a Canonical – Enterprise Transform and Load Part 3
Overview In the last article in my Extract Transform and Load (ETL) series, Submission Information Packages and High-level Validation, I covered the most important part of this process: the data. The data format will drive 80 percent of the design decisions for the ETL pipeline. In this article, I...
26 Feb, 2019
Extract Transform and Load Part 2 – Submission Information Packages and High-level Validation
The first two major pieces of an Extract Transform and Load pipeline are the Submission Information Package (SIP) and High-level validation. The major components as well as the benefits and usages of Extract Transform and Load where discussed in the first article in the series. It is recommended that...
18 Dec, 2018
Extract Transform and Load Overview
Part 1 of the Data Ingest Series The process of extract transforms and load (ETL) goes by many names: Data acquisition Data ingest Enterprise transform and load But they all are about getting external data into the system. The problem that most businesses face is that there are no...
05 Oct, 2017
The Steady Stream of New Products from AWS
Amazon Web Services (AWS) has long been the leader in Infrastructure-as-a-Service (IaaS) and Platform-as-a-Service (PaaS). However, there’s been a veritable Cambrian explosion of new Software-as-a-Service (SaaS) and Function-as-a-Service (FaaS) products coming out of the AWS teams that further embrace AWS’s serverless/pay-as-you-go-ism. In the last 24 months or so, AWS...