AWS open source news and updates No.43

November 9th - Instalment #43

Week No.43 or 101011 if you prefer binary notation. This week we have the usual collection of open source projects, blog posts, events, case studies and workshops. Of note this week is the GA of the Lambda Powertools for Java, news of the announcement last week of the new RabbitMQ message broker service, more AWS Nitro Enclave goodness, a brand new workshop from Nader Dabit and more events and webinars for your diary. There are several of those events running this week so check them out and you may still have time to reserve your spot. Make sure you check out the important note below if you are currently using Docker Hub within your environment to pull/store your container images as this update is for you.

Without further ado, lets begin…

Docker Hub

If you currently are using Docker Hub then you need to read this. Docker, Inc. has announced that the Hub service will begin limiting the rate at which images are pulled under their anonymous and free plans. These limits will progressively take effect beginning November 2, 2020. Once fully in place, free plan anonymous use will be limited to 100 pulls per six hours, free plan authenticated accounts limited to 200 pulls per six hours, and Pro and Team accounts will not see any rate limits. Read, Advice for customers dealing with Docker Hub rate limits, and a Coming Soon announcement Omar Paul and Michael Hausenblas to find out more details as well as understand what your options are.

Celebrate open source contributors

The articles posted in this series are only possible thanks to contributors and project maintainers and so I would like to shout out and thank those folks who really do power open source and enable us all to build on top of what they have created.

So thank you to Philipp Schmid, Wilhelm Wonigkeit, Aidan Steele, Gunter Rotsaert, Richard Fan, Albert Corollari, Amrut Prabhu, Deniz Parmaksız , Maggie Carter, Grace Kitzmiller, Deniz Parmaksız, Michael Edge, Allan Zheng, Channy Yun, James Beswick, Deepak Singh, Ahmet Atalay, Yannick Pobiega, Tiago Reichert, Bruno Emer, Anuraag Agrawal, Matt Asay, Chris Deigan, Adir Sharabi, Cristina Fuia, Nick Lee, Matteo Figus, Luc van Donkersgoed, Duarte Nunes and Joseph M

Make sure you find and follow these builders and keep up to date with their open source projects and contributions.

Latest from open source projects

efsync

efsync is an open source CLI/SDK tool from Philipp Schmid, which automatically syncs files and dependencies to AWS EFS. This enables you to install dependencies with the AWS Lambda runtime directly into your EFS filesystem and use them in your AWS Lambda function. It enables you either combine this with syncing files from S3 or uploading them with SCP. You can also sync files from S3 and upload with SCP without installing Pip dependencies. Philipp also put togehter this walkthrough to get you started: efsync my first open-source MLOps toolkit

Qovery Engine

Qovery Engine is an open-source abstraction layer library that makes it easy to deploy your applications on AWS (and other providers in the future). The Qovery Engine is written in Rust and takes advantage of Terraform, Helm, Kubectl, and Docker to manage resources so you can deploy your applications in minutes. Detailed documentation and some demos of how it works, this is a project worth spending some time checking out.

amazon-redshift-db-driver

amazon-redshift-jdbc-driver and amazon-redshift-python-driver Amazon Redshift JDBC and Python drivers are now open source and available for the user community under the Apache-2.0 license. With this release, customers will gain enhanced visibility to the driver implementation and can contribute to its development. Users can now browse the code for both drivers on the relevant AWS GitHub repositories, submit driver functionality enhancements through Git Pull Requests, and report issues for review. These drivers introduce several enhancements: The JDBC driver now supports querying across databases (Preview), improves resultset metadata fetching, optimises batch INSERT performance through the reWriteBatchedInsertsSize parameter, improves support for client side cursor through the fetchRingBufferSize parameter, and adds support for SocketFactory. The new open source Python driver introduces IAM and IDP support for Single Sign On (SSO), NumPy and Pandas API support, and an extended database metadata API.

Check out more here.

aws-color-region-navbar-extension

aws-color-region-navbar-extension super nice project from Albert Corollari that provides an open source browser extension that changes the color of AWS' navbar depending on the region. This works on Firefox and Chrome, so take a look and start figuring out what colour scheme you will use.

aws-lambda-powertools-java

aws-lambda-powertools-java great to see this project go GA, so if you were a Java developer and were envious of those Python developers who had the awesome Lambda Powertools then you no longer need to worry. A suite of utilities for AWS Lambda Functions that makes tracing with AWS X-Ray, structured logging and creating custom metrics asynchronously easier. Pankaj Agrawal has put together this blog, Simplifying serverless best practices with AWS Lambda Powertools Java to walk you through the project and get you started.

cpr_gazebo

cpr_gazebo Clearpath have provided some open source Gazebo resources that you can use in your AWS RoboMaker simulations. Agriculture, Inspection, Office and World simulation worlds are provided complete with a quick start launch. Read the post here for more details.

AWS open source posts

RabbitMQ

Amazon MQ Update – New RabbitMQ Message Broker Service from Channy Yun provides this weeks big announcement on the launch of Amazon MQ for RabbitMQ, a managed messaging service that reduces the operational overhead for companies managing RabbitMQ message brokers. For those perhaps unfamiliar with RabbitMQ, it is an open source and popular message broker and a critical component of distributed applications. RabbitMQ is a mature project known for its feature richness, active community support, and broad range of supported clients and frameworks but it also requires knowledge and expertise to be able to build and manage clusters to support your application. This is a service (Amazon MQ) that is already super popular with customers and this launch expands the capabilities by adding RabbitMQ. Channy walks you through the service, setting up a quick message broker and then using a python client to test it.

Apache ActiveMQ

Using Amazon MQ as an event source for AWS Lambda James Beswick follows on from news in the last couple of weeks that Amazon MQ can now an event source for AWS Lambda with a post that shows you how to set up an Amazon MQ broker and networking configuration, and then create a Lambda function that is invoked by messages from Amazon MQ queues. For those perhaps unfamiliar, Amazon MQ provides a fully managed, highly available message broker service for Apache ActiveMQ.

Kubernetes

Amazon ECS vs Amazon EKS: making sense of AWS container services Deepak Singh provides this weeks must read post on the topic of how to approach thinking which container service is best for your application or workload needs. This post provides you with a list of all your options including where. This is a short read so well worth a couple of minutes of your time today.

AWS App Mesh

AWS App Mesh is a service mesh based on the Envoy proxy. This week a couple of posts. First up, we have CI/CD with Amazon EKS using AWS App Mesh and Gitlab CI where Ahmet Atalay shows you how to deploy an application to Kubernetes in a CI/CD Pipeline using AWS App Mesh using Gitlab as a source code repository. In this post he shows you how to build a complete CI/CD pipeline for applications deployed on Amazon Elastic Kubernetes Service (Amazon EKS), using Gitlab CI/CD pipelines, and how to deploy applications using canary techniques using AWS App Mesh resources.

Following that we have Connecting services across multiple accounts using AWS App Mesh and Amazon ECS from Yannick Pobiega shows how to utilise AWS App Mesh to connect multiple application components residing in different Amazon ECS clusters across multiple accounts. This is a pattern we are seeing as customers look to deploy their micro services across multiple AWS accounts to reduce the blast radius and/or give teams more independence during development. Either way, if you are looking at how to achieve this then check this post out.

Finally, Tiago Reichert and Bruno Emer follow with Create a pipeline with canary deployments for Amazon EKS with AWS App Mesh demonstrate how you can leverage different AWS services in conjunction with AWS App Mesh to implement a canary deployment strategy for applications running on Amazon Elastic Kubernetes Service (Amazon EKS).

Bonus Out this week was the launch of a new Service Mesh White Paper which you can download here. This white paper provides a good introduction, the use cases and how you can apply service mesh with your AWS workloads.

OpenTelemetry

Distributed tracing with OpenTelemetry Anuraag Agrawal gets you up and running with OpenTelemtry on Java and shows you how you can use the OpenTelemtry agent for Java and then shows how you can view this in Zipkin.

ROS

Building the future of robots development with ROS 2 Matt Asay provides insights into the world of open source Robotics and how the Robotics Operating System (ROS) is helping to change robotics in the same way that other open source frameworks (such as Linux Apache MySQL and PHP, LAMP) did for accelerating web applications on the web. If you want to read a piece to understand what this looks like, then look no further.

Data Privacy

Handling data erasure requests in your data lake with Amazon S3 Find and Forget this collaboration between Chris Deigan, Adir Sharabi, Cristina Fuia, Nick Lee, and Matteo Figus provides a ready to roll solution that allows you to remove records from data lakes of any size that are in AWS Glue Data Catalog. This is going to help you meet obligations that operators may need to meet (for example GDPR), and provides the capability for you to be able to erase private data from your data lake when requested. The solution includes a web user interface that you can use and an API that you can use to integrate with your own applications.

AWS SDKs

Introducing Middleware Stack in Modular AWS SDK for JavaScript Allan Zheng is a post showing you how you can use middleware feature of the AWS SDK for Javascript v3, which I posted a few weeks ago in this newsletter. Allan provides you with a summary of what this feature is and then some sample use cases and code of how you can use it.

Allan provides links to the GitHub repository, so feel free to post any issues to record any problems or request new features.

If you are running the AWS SDK for .NET v1 then you need to be aware of the end of support announcement that was posted here, Announcing the end of support for the AWS SDK for .NET version 1. On April 1st, 2021, the AWS SDK for .NET version 1 (v1) will reach the end of support, so read the post to find out what you can do.

HyperLedger

Running Hyperledger Explorer on Amazon Managed Blockchain Michael Edge shares in the first of a series of posts, how to build a Hyperledger Fabric network using Amazon Managed Blockchain. In this post, he shows you how to deploy and run Hyperledger Explorer to visualise the Fabric network. He walks you through the Hyperledger Explorer dashboard to view channels, chaincode, blocks, and transactions and you will also learn how to use the Swagger UI to interact with the RESTful API provided by Hyperledger Explorer.

Partner / Industry spotlight

Elasticsearch

How Insider Learned to Scale a Production Grade Elasticsearch Cluster on AWS Deniz Parmaksız who is a Sr. Machine Learning Engineer at Insider does a show and tell about Insider’s two-year journey of scaling up a production Elasticsearch cluster, a vital element for their recommendation and search products. This post covers the challenges faced and the solutions they used to overcome those.

Insider is an AWS Advanced Technology Partner with the AWS Digital Customer Experience Competency.

Case Studies and Industries

COVID

Crowdsourcing a cure for COVID-19: How the cloud is accelerating research and drug discovery Maggie Carter and Grace Kitzmiller provide an update on how the Cloud and distributed computing is help to tackle and accelerate research towards COVID-19 therapies. They share how COVID Moonshot, an open source project, is helping to bring about a low cost, patent free antiviral therapy. This post contains some pretty amazing data points, and I do not want to post any spoilers so you will have to check out what those are.

Workshop of the week

Next.js on AWS Amplify

next.js-amplify-workshop Fresh out from Nader Dabit, a new workshop where you will learn how to build a full stack cloud application with Next.js, GraphQL, & Amplify.

Latest blog posts

AWS Nitro Enclaves

Some more great posts this week covering one of my favourite AWS innovations, AWS Nitro Enclaves.

First up we have Serving IAM credentials to enclaves. In this post Aidan Steele explores some of the features of AWS Nitro Enclave. Aidan shows you a proof of concept of how you might create a service that would vend AWS IAM role session credentials to code running in enclaves. He has some code that accompanies this post, but as he says in the post, this should not be used in anger and is just to show what can be done.

Richard Fan follows that with a post, Running Python App on AWS Nitro Enclaves which introduces what AWS Nitro Enclaves is and then walks you through using a sample Python app. As Richard says in the post, the best way to learn is often to just dive in and that is what he has done in this post. Read on to find out more about the key features that AWS Nitro Enclaves provides, as well as a look at what attestation is and why it is important. Make sure you follow Richard for further posts on this topic.

Colm MacCárthaigh follows with an epic Tweet thread which you can read here. In this thread he breaks down some of the key components of AWS Nitro Enclaves as well as providing some good examples of the use cases where this capability might come in very hand. Maybe one day this will get turned into a blog post!

Finally, Luc Van Donkersgoed follows up with a third blog post, Ultra Secure Password Storage with NitroPepper. In this post Luc shows you how to build a Nitro Enclave application that will protect user passwords, even when an attacker has full control over the application server and database. Luc calls this application is called NitroPepper, and provides everything you need. Very nice post Luc.

Vorteil: Running ELK on AWS

An ELK stack without an OS from Wilhelm Wonigkeit CEO & Co-founder at Vorteil.io. Vorteil is an operating system for running cloud applications on micro virtual machines, and they provide open source tools that allow you to build images and then quickly deploy those. You can run those locally on Firecracker for example, or deploy them to AWS. In this post, Wilhelm walks you through how you can deploy an ELK stack (Elasticsearch, Logstash and Kibana) using Vorteil and deploy that stack on AWS.

This is an interesting project and well worth reading and then playing with the tool.

Spring

How to Deploy a Spring Cloud Function on AWS Lambda this tutorial from Gunter Rotsaert shows you how you can deploy Spring Cloud Function’s to AWS Lambda. Spring Cloud Function allow you to implement business logic via functions and deploy them via a Spring Boot application. Spring Boot is a very popular framework and so if you are looking to move your Spring Cloud functions to AWS Lambda, this is just the post you need. Gunter provides details breakdown of the steps and sample code in his repo.

Also this week on Spring was this post from Amrut Prabhu, Spring Boot : Handle AWS RDS password change or rotation without restarting who walks you through using the AWS Secrets Manager JDBC open source drive to enable you to manage rotating your Amazon RDS passwords without impacting your applications.

Rust

Building an AWS Lambda extension with Rust fresh from AWS Community Builder Duarte Nunes is a post that shows you how you can use Rust to develop a new AWS Lambda Extension that were recently announced (and have covered in the past few editions of this newsletter)

Tetraforce

How TetraForce runs Godot on AWS Joseph B. Manley shows you how TetraForce, an open-source multiplayer action-adventure RPG inspired by the popular Zelda game, uses the open source Godot engine on AWS. The posts talks about how they were able to move the game onto AWS within a week and breaks out the architecture and some of the key components. A great read.

Quick updates

Amazon Elasticsearch

Amazon Elasticsearch Service now provides the ability to define a custom endpoint for your domain and associate an SSL certificate from AWS Certificate Manager (ACM). Defining a friendly name makes it easier for your users to access Kibana, and allows you to move to a new domain without updating your clients. You can define a custom endpoint when creating a new domain or by editing an existing domain from the Amazon Elasticsearch Service console, or through the Amazon Elasticsearch Service APIs. You can then link the custom endpoint to a certificate in ACM, and create an Alias or CNAME mapping in Route 53, or in your preferred Domain Name System (DNS), to route traffic to the custom endpoint.

AWS IoT SDK for Embedded C

AWS IoT Device SDK for Embedded C (C-SDK) version 202011.00 now includes refactored coreHTTP, AWS IoT Device Defender, and AWS IoT Jobs libraries alongside of the existing coreMQTT, coreJSON, and AWS IoT Device Shadow libraries. All libraries have been optimized for memory usage and modularity, and have undergone code quality checks (e.g. MISRA-C compliance, Coverity static analysis), and validation of memory safety with the C Bounded Model Checker (CBMC) automated reasoning tool. C-SDK is a collection of C source files under the MIT open source license that can be used in embedded applications to securely connect IoT devices to AWS IoT Core. It is distributed in source form and intended to be built into customer firmware along with application code, other libraries, and an underlying operating system (OS) suitable for constrained embedded devices. For more details, see the README and Changelog files.

SQL Server on Linux

You can now use AWS Launch Wizard to perform single-instance deployments of Microsoft SQL Server on Windows Server and Ubuntu Server. AWS Launch Wizard offers a guided way of sizing, configuring, and deploying AWS resources for third party applications, such as Microsoft SQL Server and HANA-based SAP systems, without the need to manually identify and provision individual AWS resources. Previously, customers could use AWS Launch Wizard to easily perform SQL Server Always On deployments on Windows Server and Ubuntu Server. Now, you are able to leverage the same ease of use to perform SQL Server single-instance deployments on Windows Server and Linux. To learn more about using AWS Launch Wizard to accelerate your SQL Server deployments, visit the AWS Launch Wizard page.

Events for your diary

Check out these events this week.

OpenShift on AWS Nov 10, 2020 6:00 PM - 7:00 PM GMT

Mayur Shetty will discuss the benefits of Red Hat OpenShift 4.5 and all the consumption options available to customers on AWS.

Unifying Data Pipelines and Machine Learning with Apache Spark™ and Amazon SageMaker 11/10/2020 9:00am CST

Databricks and Slalom bring you this virtual event where you will:

• Learn how to build highly scalable and reliable pipelines for analytics
• Deeper insight into Apache Spark and Databricks, including the latest updates with Delta Lake
• Train a model against data and learn best practices for working with ML frameworks (i.e. - TensorFlow, XGBoost, Scikit-Learn, etc.)
• Learn about MLflow to track experiments, share projects and deploy models in the cloud with Amazon SageMaker

Enabling Cloud Data Lakes for Analytics 11/11/2020 9:00am - 12:00pm PST

In this virtual workshop, we’ll cover best practices for organisations to use powerful open source technologies to build and extend your AWS investments to make your data lake analytics ready. You’ll learn about the advantages of cloud-based data lakes in terms of security and cost. And finally, you’ll learn how data professionals are having a huge impact - lowering costs, changing time to market, and even revolutionising industries.

AWS Container Day: Kubernetes Edition November 17th, 10:00am to 6:00pm EST

Join us for AWS Container Day, a fully live, virtual day of sessions all about Amazon EKS and Kubernetes at AWS, hosted by Containers from the Couch. At this Day Zero KubeCon event, the AWS Kubernetes team will be discussing new launches, demoing products and features, covering best practices, and answering your question live on Twitch.

Virtual ROS-Industrial Conference 2020 December 15 - 16, 2020

The 8th edition of ROS-Industrial Conference will be held as a virtual event. It is not only the annual community meeting for the European ROS-Industrial community but this years event is also the final event of the H2020 ROSIN project. The conference gives you the chance to see the newest technical developments and to meet people and companies, which are active in the ROS community.

Check out this newsletter on Cloud Native

Cloud Native Newsletter

Whilst not necessarily open source, I wanted to let you know about a fantastic newsletter that I have recently found and subscribed to. The Cloud Native Software Engineering Newsletter (now on episode #17) is something you should all subscribe to as it has great information and the latest news on all things cloud native. And let’s face it, a lot of that is going to be open source related as well. Nice work Lou Bichard. You can sign up for the newsletter here.

Stay in touch with open source at AWS

I hope this summary has been useful. Remember to check out the Open Source homepage to keep up to date with all our activity in open source by following us on @AWSOpen.

Your feedback matters!

I have put together a short feedback survey, which I would ask you to take - it will take no more than 2 minutes. You can access here. Many thanks!

Take the survey 26t4