skip to content W3C SVG

AWS open source news and updates #94

December 20th, 2021 - Instalment #94

Newsletter #94.

This will be the last newsletter of 2021 before I break for Christmas and New Year. I hope you have found this newsletter a useful resource for finding out about new or interesting open source projects, both from what AWS contributes to, but also from the wider builder and open source developer communities.

To finish up for 2021 we have more new open source projects, covering Amazon Location Services, AWS Fault Injection Simulator (FIS) experiment templates that are AWS CDK ready, Media Replay Engine (MRE)a really nice project to help you automate the creation of replays, a transcribe, post call analytics solution and more. We also have some great content covering Apache Airflow, Apache Flink, Hugging Face, ROS, Apache HBase, Apache Spark, ActiveMQ, PyTorch, ROSA, Argo Rollouts, lots of Kubernetes related posts and more.

Finally, make sure you check out the videos, where we have a really great story that combines AWS DeepRacer with a rodent problem. Really cool stuff.

To start however, I just want to share some important updates about Apache Log4j.

Apache Log4j rce update

We have been working very hard to help customers understand how to work through this incident. Here are some resources that you should check out, including a number of tools that you might find helpful.

Job Alert

This position is still open, and if you are looking for a fresh start to 2020 then read on. What are we looking for? We are looking for someone who will be responsible for defining, leading, and contributing to the open source and community engagement content strategy for the services and technology teams across AWS. You will combine your passion and enthusiasm for cloud technology and open source with your unmatched creativity to generate content and support for AWS among key open source communities, industry opinion makers, and technologists.

You will work closely with the product marketing leadership to translate the business priorities of the service teams into original content for a variety of audiences, including C-level, end users, developers, managers, and engineers.

Ideally, you are already a recognisable figure in the open source ecosystem, in demand to contribute to technical and business publications, with an exemplary presence on social media.

Here is the Job Spec, Principal Evangelist, Open Source, Open Source Strategy & Marketing where you can read more and see how to apply.

Celebrating open source contributors

The articles posted in this series are only possible thanks to contributors and project maintainers and so I would like to shout out and thank those folks who really do power open source and enable us all to build on top of what they have created.

So thank you to the following open source heroes: Rajarshi Das, Yasunori Kirimoto, Adam Cerin, Adrian Hornsby, John Gramila, Eira May, Ryan Donovan, Martin Paradesi, Vincent Gromakowski, Gary Stafford, Balasubramanian Sakthivel, Victor Gan, Manjula Nagineni, Amir Shenavandeh, Maryam Tavakoli, Laurence Miao, Srinivasa Shaik, Matt Aylward, Sasi Jayalekshmi, Suranjan Choudhury, Anil Sharma, Imaya Kumar Jagannathan, Michael Hausenblas, Bob Strahan, Andrew Kane, Connor Kirkpatrick, Franco Rezabek, and Steve Engledow.

Make sure you find and follow these builders and keep up to date with their open source projects and contributions.

Latest open source projects

amazon-location-service-starter

amazon-location-service-starter this is a starter open source project from Yasunori Kirimoto that gets you started with Amazon Location Service using a number of AWS Services and open source projects - AWS Amplify, MapLibre GL JS Amplify, MapLibre GL JS, webpack.

demo

aws-media-replay-engine

aws-media-replay-engine this Apache 2.0 project, Media Replay Engine (MRE) is a framework to build automated video clipping and replay (highlight) generation pipelines for live and video-on-demand content. Nice detailed docs, including some guidelines on costs, make sure you check out this project.

arch

aws-fis-templates-cdk

aws-fis-templates-cdk chatting last week with Adrian Hornsby before he spends some time in the artic circle, he shared with me this repo that contains a collection of AWS Fault Injection Simulator (FIS) experiment templates deploy-able via with the AWS CDK. These templates let you perform fault injection experiments on resources (applications, network, and infrastructure) in the AWS Cloud.

aws-securityhub-falco-ecs-eks-integration

aws-securityhub-falco-ecs-eks-integration this repo deploys a Lambda function, that enables generating Falco findings into Security Hub. To walk you through this, Rajarshi Das and Adam Cerin have put together this post, Continuous runtime security monitoring with AWS Security Hub and Falco

arch

amazon-transcribe-post-call-analytics

amazon-transcribe-post-call-analytics This open source sample solution, Post Call Analytics (PCA), does most of the heavy lifting associated with providing an end-to-end solution that can process call recordings from your existing contact center. Bob Strahan, Andrew Kane, Connor Kirkpatrick, Franco Rezabek, and Steve Engledow have collaborated on this post, Post call analytics for your contact center with Amazon language AI services to help you get started.

arch

poro

poro this script lets you scan publicly accessible assets on your AWS cloud environment for reporting. purposes. Might come in handy for folks that are looking to do this.

AWS and Community blog posts

AWS BugBust

Earlier in the year, we announced BugBust - a worldwide competition to help find and fix bugs in Java and Python applications. During re:Invent, there was an attempt to set a Guinness World Record for the largest bug fixing challenge. I don’t want to spoil the surprise, so check out this write up from Eira May and Ryan Donovan, Smashing bugs to set a world record: AWS BugBust

Amazon EMR

When it comes to saving money, I am all ears. I therefore was very thankful to John Gramila for putting together, AWS EMR Cost Optimization Guide. If you are using or planning to use Amazon EMR, then make sure you check out some of the tips in this post to help you optimise your costs.

ROS

Camilo Buscaron shares news that we have added the supporting software and simulation artefacts to integrate the AWS DeepRacer Evo 64 device software with ROS Nav2 stack, and made it available to the open source community via the AWS DeepRacer GitHub in his post, Integrating ROS Nav2 stack with AWS DeepRacer

demo

Grafana

In case you missed it, there were a number of nice announcements during pre:Invent and re:Invent for customers who are interested in Grafana. Imaya Kumar Jagannathan and Michael Hausenblas have summarised these in the post, Amazon Athena, Amazon Redshift Plugins and New Features in Amazon Managed Grafana walking you through the Geomap visualisation, Amazon Redshift data source, CloudWatch Metrics Insights, and interestingly, IoT TwinMaker integration (which is in preview). Lots to digest, so make sure you check this out.

demo

Apache Airflow

A couple of posts you should check out this week.

First up, we have Gary Stafford who writes, DevOps for DataOps: Building a CI/CD Pipeline for Apache Airflow DAGs showing you how you can use GitHub Actions to build an effective CI/CD workflow for our Apache Airflow DAGs.

arch

Following that we have a short post from myself, Setting up MWAA to use a KMS key where I walk you through how to configure your Managed Workflows for Apache Airflow to use a customer defined KMS key so that you can encrypt everything within your Airflow environment.

Apache Flink

In this post How Goldman Sachs built persona tagging using Apache Flink on Amazon EMR, Balasubramanian Sakthivel, Victor Gan, and Manjula Nagineni share with you how Goldman Sachs built a system using Apache Flink on Amazon EMR to carry out the tagging of users with various personas, in order to better curate content offerings for those users.

arch

Apache Spark

Vincent Gromakowski shares Best practices for running Spark on Amazon EKS where you will learn how to configure Apache Spark and Amazon EKS to support common requirements, including resources isolation, cost reduction, dynamic scaling, performance optimisation, and fine-grained access control.

Apache HBase

Apache HBase is a popular, open source non-relational database. One of the typical use cases where you find it being used, is when you need random, realtime read/write access to your Big Data - the project’s goal is the hosting of very large tables, billions of rows X millions of columns. In this post, Stream Apache HBase edits for real-time analytics Amir Shenavandeh and Maryam Tavakoli walk you through Apache HBase scaling and replication concepts before sharing some common use cases and solutions, along with some best practices when implementing your custom HBase streaming replication endpoints.

arch

PyTorch

Laurence Miao shares how to Build GAN with PyTorch and Amazon SageMaker, and he walks you through building your first GAN model using Amazon SageMaker, learning GANs from the perspective of practical engineering experiences, as well as opening a new AI/ML domain of generative models. The post also introduces a use case of one of the hottest GAN applications in the synthetic data generation area. [hands on]

arch

For more PyTorch content, check out the links below for “Build custom Amazon SageMaker PyTorch models for real-time handwriting text recognition”

Argo Rollouts

Argo Rollouts is a Kubernetes controller and set of CRDs which provide advanced deployment capabilities such as blue-green, canary, canary analysis, experimentation, and progressive delivery features to Kubernetes. Srinivasa Shaik, Matt Aylward, and Sasi Jayalekshmi have collaborated on the post, Use Amazon EKS and Argo Rollouts for Progressive Delivery showing you how when you implement a progressive delivery controller (via Argo Rollouts) in conjunction with AWS services, you can tune the speed of your deployments and measure your success with KPIs.

arch

ActiveMQ

Suranjan Choudhury and Anil Sharma have come together to write Using an Amazon MQ network of broker topologies for distributed microservices where they look at ActiveMQ topologies that customers can evaluate when planning hybrid deployment architectures spanning AWS Regions and customer data centers, using a network of brokers.

arch

Other posts worth checking out

Quick updates

Spring Boot

v2.3.3 is now available, and this version bumps a spring-messaging dependency to 5.3.13. Previous Spring Cloud AWS versions relied on spring-messaging that suffered from a security vulnerability (#206). If you relied on Spring Cloud AWS to determine spring-messaging version, please update to Spring Cloud AWS 2.3.3. Other changes include:

  • Fix: context order when loading properties (Parameter Store and Secrets Manager) - @WtfJoke
  • Docs: Mention no Aurora support in RDS docs (RDS) - @aravindparappil46
  • Dependency Upgrade: Upgrade Maven Wrapper - @tinexw
  • Enhancement: Log exception on message processing failure (SQS) - @ebussieres
  • Dependency Upgrade: Upgrade Spring Cloud Build to 3.0.5 - @maciejwalkowiak
  • Dependency Upgrade: Upgrade AWS SDK to 1.12.129 - @maciejwalkowiak

Videos of the week

RoboCat

Loved this video, and now have ideas of how I can use my DeepRacer to do something similar. Searching for a humane way to scare mice out of his kitchen, Martin Paradesi modified an autonomous DeepRacer model car, and its open source code, to come up with a RoboCat that could scare mice away in the dark. Find out how his young sons contributed to the project.

Events for your diary

Watch this space for events in 2022. If you have an event you want me to publish here, please contact me and I will include it in this listing.

Stay in touch with open source at AWS

I hope this summary has been useful. Remember to check out the Open Source homepage to keep up to date with all our activity in open source by following us on @AWSOpen

Copyright © - Ricardo Sueiras

Made with Hugo. Themed by Cupper.