AWS open sources new and updates - No. 19

May 11th - Instalment #19

Episode 19, and this week’s edition is bursting at the seams with probably the biggest amount of news and updates to share (hence the later posting). Looks like everyone has been busy, and there were some things I had to leave out for another time.

Before I get started though, I think it is worth reflecting on this blog post by Matt Asay, The most important part of an open source project, where he talks about the value of all contributions in open source projects, not just the developers cutting code. It is well worth 10 minutes of your day to reflect upon this post, as well as think about this data point from a study from the World Bank on open source contributions, that highlighted:

57% of the invested resources went to writing code. The rest, 43%, was spent on things like marketing, docs, meet-ups and other activities that aided the collaboration effort.

So, without further delay, let’s dive into this weeks latest open sources news and updates.

Celebrate open source contributors

The articles posted in this series are only possible thanks to contributors and project maintainers and so I would like to shout out and thank those folks who really do power open source and enable us all to build on top of what they have created.

So thank you to Ben Kehoe (@ben11kehoe), Adrian Horn (@adhorn), Joseph Maney (@josephbmanley), Luca Bianchi(@bianchiluca), Michael Garcia, Anish Yadav, Josh Kahn, Julie Lerman, Fernando Dingler, Jared Short, Cosmin Sanda, Nader Dabit, Nico Vibert, Dario La Porta, Mikhail Shapirov, Harish Shenoy, Letu Yang and Bernd Mathiske.

Latest from open source projects

AWS SSO credential process

This project from Ben Kehoe helps you get AWS SSO working with all the AWS SDK’s that you might be using that do not currently understand it. Here is a link to the GitHub repository for aws sso credential process. Thank you Ben!

AWS Chaos Scripts

This week from Adrian Hornsby, (Operational Excellence and Chaos engineering writer and general all round good guy), comes this collection of scripts to help you inject a bit of failure into your life. Make sure you read the disclaimer, but if you have heard or read his posts on operational excellence and building resilience, then you can check these scripts out and get started on your own journey. Thank you Adrian.

Thanks Adrian for helping me with my own operational excellence by spotting the mistakes. I would love to say these were part of a deliberate plan to embed Chaos into my workflow, but alas…..

AWS Billing Alerts

Setting up billing alerts for your AWS environments is general good practice, so here is a sample architecture complete with open sourced templates to get you started from Jospeh Manley. Billing Arts architecture

You can find the details of how to set this up following Joseph’s instructions on his GitHub repository here. Be sure to check out his other repositories too. Thank you Jospeh!

IAM Policy Generator

Luca Bianchi (@bianchiluca) shared this week a nice little library you can find on his GitHub repository, IAM policy generator.

It helps you generate a valid IAM PolicyStatement object without having to “guess” the right Action (and to avoid human error and typos in general), thanks to proper validation and code completion.

Thank you Luca!

Security - Cloudsplaining

The lovely folk at Salesforce bring you this open source project to help improve the security of your deployments on AWS. Cloudsplaining is an AWS IAM Security Assessment tool that identifies violations of least privilege and generates a risk-prioritised HTML report with a triage worksheet.

Demo of cloudsplaining

Check out the GitHub repo at https://github.com/salesforce/cloudsplaining

VMware Cloud on AWS - pyVMC

Are you a python developer that has been looking for a way to automate your VMWare Cloud on AWS Software defined data centre environments? If so, then you are in luck as Nico Vibert has put together this project, pyVMC on GitHub and written up a supporting blog post, Introducing pyVMC. and you can also read this post too.

pyVMC

Check out Nico’s project on GitHub here.

Blog posts you should check out

There have been lots of great posts this week, as well as new releases and updates so check these out.

DynamoDB Workbench for Linux

NoSQL Workbench for Amazon DynamoDB is a client-side application that helps developers build scalable, high-performance data models, and simplifies query development and testing. NoSQL Workbench is now available for Ubuntu 12.04, Fedora 21, Debian 8, and any newer versions of these Linux distributions, in addition to Windows and macOS.

Read the full announcement here, and if you just want the download link, here it is for Linux, Windows and MacOS installs.

Mosquitto mqtt to AWS IoT

Mosquitto mqtt is an open source package I have used extensively to connect all sorts of devices and sensors to the Cloud. In this updated blog post, How to Bridge Mosquitto MQTT broker to AWS IoT, Michael Garcia and Anish Yadav show you how to implement the Mosquitto broker ‘Bridge’ capability to setup bi-directional exchange of data with AWS IoT Core through MQTT messages. This will enable your devices to communicate locally with the Mosquitto broker and with AWS IoT Core to benefit from the power of the AWS Cloud.

architecture diagram

Now let me dust off one of my many Raspberry Pi’s and give this a go.

AWS AppSync and Amazon QLDB

Amazon Quantum Ledger Database (QLDB) is a purpose-built database for use cases that require an authoritative data source. Amazon QLDB maintains a complete, immutable history of all changes committed to the database (referred to as a ledger). Amazon QLDB fits well in finance, eCommerce, inventory, government, and numerous other applications.

Pairing Amazon QLDB with services such as AWS AppSync allows you to safely expose data and that data’s history for mobile applications, websites, or a data lake. In this two part post, part one and part two Josh Kahn explores a reusable approach for integrating Amazon QLDB with AWS AppSync to power an example government use case.

AWS AppSync to Amazon QLDB

Amazon CloudWatch powers Prometheus metrics

You can use Amazon CloudWatch to monitor Prometheus metrics from Amazon Elastic Kubernetes Service (EKS) and Kubernetes clusters, now available in beta. With this new feature, DevOps teams can automatically discover services for containerised workloads such as AWS App Mesh, NGINX, and Java/JMX. They can also expose custom metrics on those services, and ingest the Prometheus metrics in CloudWatch.

Read the full announcement here.

.NET Core

Julie Lerman writes about AWS for .NET developers in Discovering AWS for .NET Developers. If you a .NET developer and are new or just getting started with AWS, then this will be a useful starting point for you. From getting started to actually writing some code, you will soon be on your way to running open source .NET workloads on AWS.

Thank you Julie.

Amplify DataStore

Fernando Dingler, shares a reference application and architecture that demonstrates how to build a modern mobile application with built-in offline and cloud synchronisation capabilities using React Native with the Amplify DataStore in the front end and AWS AppSync with Amazon DynamoDB in the backend.

dataStore architecture

AWS AppSync is a GraphQL serverless backend for mobile and web applications. It provides a flexible and reliable API with a built-in compute layer to run queries, mutations, and subscriptions to securely access, manipulate, and combine data from one or more data sources for any application. On the client device, the Amplify DataStore provides a familiar programming model for leveraging shared and distributed data without writing additional code for offline, real-time and online scenarios, which makes working with distributed, cross-user data just as simple as working with local-only data.

You can check out the docs and links to the GitHub repository for Amplify here, the quickest way to get started.

HPC

AWS ParallelCluster is an AWS supported Open Source cluster management tool that makes it easy for you to deploy and manage High Performance Computing (HPC) clusters in the AWS cloud. This week, we have a great post covering this area.

Using AWS ParallelCluster with a serverless API

In Using AWS ParallelCluster with a serverless API, Dario La Porta, AWS Senior Consultant for HPC writes about creating a serverless API of the AWS ParallelCluster command line interface. With this API, you can create, monitor, and destroy your clusters. This makes it possible to integrate AWS ParallelCluster programmatically with other applications you may have running on-premises or in the AWS Cloud.

Serverless HPC

Thank you Dario La Porta!

Amazon Corretto - introducing Heapothesys

Load testing and performance tuning Java stacks are going to be a lot easier with this open source project, Heapotheys. Created by the Amazon Corretto team, the Heapothesys benchmark is a synthetic workload that simulates fundamental application characteristics that affect garbage collector (GC) latency. The benchmark creates and tests GC load scenarios defined by object allocation rates, heap occupancy, and JVM flags, then reports the resulting JVM pauses. OpenJDK developers can thus produce reference points to investigate capability boundaries of the technologies they are implementing.

Read the post by Letu Yang and Bernd Mathiske, Introducing Heapothesys, an open source Java GC latency benchmark with predictable allocation rates and then go to the GitHub repository for Heapothesys, here.

Thank you Letu Yang and Bernd Mathiske!

Deep Java Library (DJL) and Spring Boot

The java programming language is still very popular and dominant in the enterprise space. Organisations looking to adopt machine learning and deep learning in their existing applications are looking for solutions that enable existing talent and resources to rapidly get started using the programming languages and tools they know.

Recognising this challenge, AWS has created a few open source projects to facilitate the adoption of ML for Java and microservices, and ultimately to help our customers, partners and the open source community as a whole. The Deep Java Library is one such open source project.

In this post, Adopting machine learning in your microservices with DJL (Deep Java Library) and Spring Boot, Mikhail Shapirov walks you through building a Spring Boot application that uses the Deep Java Library to create a simple object detection application using Apache MXNet and DJL.

Thank you Mikhail!

German readers will find this post on the DJL interesting but you can also use various translation services to go through this introduction into Deep Java Library.

Do you use AWS and love open source?

Do you have some content you want to share with a broader audience? We are always looking for guest content for the AWS Open blog. Please get in touch (via comments below) and I would love to speak with you about what you are doing in open source. We are always looking for interesting new content.

We are hiring

We are hiring, so if you love open source, then why not check out the current openings. We would love to hear from you and if you want to chat and ask me anything, connect to me via LinkedIn or Twitter and always happy to talk.

Open Source jobs at Amazon and AWS - scroll to bottom, on left.

Stay in touch with open source at AWS

I hope this summary has been useful. I have looked for all the session videos that have been uploaded to date, but if I have missed anything, please get in touch and I will update this summary. Remember to check out the Open Source homepage to keep up to date with all our activity in open source by following us on @AWSOpen.