Amazon Web Services (AWS) has long been the leader in Infrastructure-as-a-Service (IaaS) and Platform-as-a-Service (PaaS). However, there’s been a veritable Cambrian explosion of new Software-as-a-Service (SaaS) and Function-as-a-Service (FaaS) products coming out of the AWS teams that further embrace AWS’s serverless/pay-as-you-go-ism. In the last 24 months or so, AWS has started to focus on addressing customers’ challenges in managing a product’s technology assets and data movements in AWS infrastructure to further support Continuous Integration / Continuous Deployment (CI/CD). It’s hard to keep up with all of the new stuff, so here are some that deserve your attention.
Glue is an extract, transform, and load service (ETL) that you can set up and run from your AWS Management Console. What’s even more interesting is that Glue can crawl and catalog your data living in AWS data stores, such as S3, Redshift, RDS, EMR, making data ETL, discovery, and visualization a lot easier to implement. As an added benefit, the Python code it generates is customizable, reusable, and portable.
After you’ve done your ETL through Glue, you can use Athena to analyze your transformed data. AWS Athena allows you to treat disparate data sources as if they were one integrated data set. This interactive query service allows any SQL wrangler to dive right in without the overhead typically associated with such work. The AWS Management Console is the primary user interface, but Athena currently supports 5 concurrent queries/scripts from 3rd-party tools such as SQL Workbench via a JDBC connector (JDBC 4.1 and Java 8 compatible).
Lambda, originally released in 2014, lets you run event-driven FaaS in a serverless environment. This counter-intuitive approach doesn’t involve the absence of hardware, but, rather, is one where Amazon takes care of the provisioning of underlying application server resources. This is the essence of cloud computing: elasticity.
With the introduction of AWS Step Functions, users are now able to orchestrate a workflow of Lambda functions, great for building a state machine or executing business logic transformations on data ingest.
What about testing you may ask? Testing execution has been a challenge since Lambda’s initial release, but, by combining AWS Serverless Application Model and aws-lambda-local CLI tool, you can manage entire Lambda configurations as code and test Lambda function executions offline!
Performance wise, keep in mind that Lambda can be pre-warmed to reduce latency caused by initial bootstrapping. The “Serverless-plugin-warmup” npm module and CloudWatch event are some of the simple ways to keep Lambda container ready to go.
DynamoDB is a NoSQL database service with super-low latency. It can handle table, item, and attribute data model units and numerous data types. It can also auto-scale based on throughput. As with the other services we’ve talked about, this one is also managed, meaning you don’t have to worry about the service, storage, or connectivity. In addition to integration with common AWS computing resources, you can write directly to DynamoDB, write via API Gateway, or trigger with a data ingest service such as Kinesis or SQS.
Similar to Lambda, local/offline testing with DynamoDB used to be a challenge that has been somewhat mitigated by various DynamoDB Docker images and npm package. AWS recently announced DynamoDB Local as a Java package (JAR file) to allow Java applications to build and test against a local DynamoDB instance!
AWS Config will tie all of this together for you, allowing you to track all of your AWS resources. The service will record any changes you make to your resources in a consistent format and compares them to what you have defined as your desired configurations. AWS Config will also display this information in a dashboard and make it available through an API.
Over the years, Ten Mile Square has gone through many iterations of developing technology asset management systems for our enterprise clients to manage their exponential growth of configurations due to adoptions of [Infrastucture/Platform/Software]-as-a-Services (not going to generate an abbreviation for this one). We found AWS Config very much aligned with our philosophy. It is encouraging to see the path AWS has taken to facilitate a config-driven environment.
What these services share is their serverless approach. This means you can focus on your software and data, and you only pay for what you use.
Keep your eyes peeled, though. Given AWS’s cadence, new services are sure to come out on a regular basis.