January 4th, 2021 - Instalment # 50
Newsletter # 50 and the first one this year. I hope you all had a good break over the festive season. I am busy catching up with all the open source posts, projects, case studies and more, much of which I have shared this week. I have also covered what is happening next week with the concluding week of re:Invent - plenty more open source sessions to check out.
The AWS Viewpoint on Open Source and Kubernetes
If you missed this podcast, The AWS Viewpoint on Open Source and Kubernetes a few weeks back it is well worth re-visiting. Bob Wise, AWS’ general manager of Kubernetes, and Peder Ulander, AWS head of product marketing talk about AWS, Kubernetes and more. Well worth adding to your podcast playlist queue.
Celebrate open source contributors
The articles posted in this series are only possible thanks to contributors and project maintainers and so I would like to shout out and thank those folks who really do power open source and enable us all to build on top of what they have created.
So thank you to the following superstars: Shanthan Kesharaju, Cecil Patterson, Sean McGrail, Romain Marcadier, Trivikram Kamat, Jonah Jones, Leo Di Donato, Asser Moustafa, John Jackson, Vinay Selvaraj, Jimmy Dahlqvist, Jorel Jean-Charles, Ian Mckay, Kelvin Lo, Abbas Nemr, Calvin Wang, Chris Ghyzel, Veronika Megler and Jerome Van Der Linden.
Make sure you find and follow these builders and keep up to date with their open source projects and contributions.
If you missed the open source track or sessions during re:Invent during November 30th and December 17th, then check out a few blog posts I put together that summarises the key announcements, new launches and some of the great open source sessions that you can now view via the on-demand capability which should be around until the end of this month. You can start with re:Invent open source highlights: Week 1 and from there navigate to weeks two and three.
January 11th - re:Invent continues
The final part of re:Invent kicks off on January 11th. Here are the sessions you can look forward to next week. As these are already in the re:Invent catalog so you can add them to your calendar (links provided).
Latest from open source projects
gitlab-runners-on-aws if you missed this over the last few weeks, then catch it now. Jimmy Dahlqvist’s open source project that makes it easy to setup and run GitLab CI/CD runners on AWS in an auto scaled way using AWS Lambda. GitLab webhooks are used to start Lambda functions to perform the Job in a Docker container.
ec2-spot-converter this open source project from Jorel Jean-Charles that is going to be very popular I think. This tool converts existing AWS EC2 instances back and forth between On-Demand and ‘persistent’ Spot billing models while preserving instance attributes (Launch configuration, Tags..), network attributes (existing Private IP addresses, Elastic IP), storage (Volumes), Elastic Inference accelerators and Elastic GPUs. Detailed and easy to follow documentation with examples, check this project out and as always, Jorel welcomes any PRs you might want to make so feel free to join the project.
cloud9-sso this new project from Ian Mckay looks very interesting. Still in very early beta stage, it provides a simple way to add AWS Cloud9 environments to AWS SSO. Ian has put together a quick walkthrough for you and I am going to try this one out myself over the coming week or so.
AWS open source posts
Amazon Managed Workflows for Apache Airflow unaffected by Airflow 1.10.12 vulnerability is a quick post from John Jackson that takes a look at the recently published CVE (CVE-2020-17526) and what it means to customers who are using Amazon Managed Workflows for Apache Airflow. You might miss this, but John also talks about the efforts as part of the effort to help customers migrate to Apache Airflow 2.0. This will mean that you will be able to run your current Apache Airflow DAGs as and when version 2.0 is available. The post shares how this will work, so if you use Apache Airflow and are thinking about migration, then check it out.
Testing data quality at scale with PyDeequ Calvin Wang, Chris Ghyzel, and Veronika Megler introduce PyDeequ, an open-source Python wrapper over Deequ (an open-source tool developed and used at Amazon). This tool helps data scientists test and validate the quality of your data, and is used extensively within Amazon. This post provides an introduction into this tool and then provides a walkthrough on how you can get started.
Falco is an incubating Cloud Native Computing Foundation (CNCF), open source, cloud native runtime security tool. In this post, How Falco uses Prow on AWS for open source testing Jonah Jones and Leo Di Donato, share how this open source project is using another open source project, Prow, as part of its CI/CD build system. The key thing here is that Prow is a tool that has been developed by the Kubernetes sig-testing Special Interest Group, to address some specific requirements the project had - I don’t want to spoil this, read the post to find out. This is a really great deep dive, so suggest you check this out.
How the jsii open source framework meets developers where they are great post from Romain Marcadier that you absolutely have to check out. In this post, Romain takes a look at the open source project jsii, what it is, why we created it and how it is helping us meet developers where they are. I have shared and talked about jsii in this newsletter many times, and this post goes much deeper including how other projects are beginning to explore how they can use this project beyond the original scope of the AWS CDK. Great stuff.
The use of conversational interfaces across many different applications and uses cases continues to grow. As these conversational services become more important, there is a need to be able to monitor the performance and effectiveness of these interfaces with analytics and dashboards to drive continuous improvements in these interfaces. In this post, Analyzing Amazon Lex conversation log data with Grafana Shanthan Kesharaju and Cecil Patterson collaborate and share a solution architecture that streams conversation logs from Amazon Lex to Amazon CloudWatch, using Grafana to build visualisation and reporting insights for missed utterances, user requests, and sentiment metrics.
Integrating EC2 macOS workers with EKS and Jenkins Paul Roberts shares a great post that takes advantage of one of the announcements made during re:Invent, namely support for macOS instance types. In this post, Paul shares how you can leverage those as part of your developer workflows to build, test, package, and sign Xcode applications for the Apple platforms including macOS, iOS, iPadOS, tvOS, watchOS, and Safari. In this post, Paul uses Jenkins to show you how you can build your own build environment to do this.
Building high-quality benchmark tests for Amazon Redshift using Apache JMeter Asser Moustafa takes a look at how you can use the open source testing tool Apache JMeter, to setup and run your own benchmarking tests. Asser shares tips on what to think about as you approach this as well as how to optimise Apache JMeter.
Continuously building and delivering Maven artifacts to AWS CodeArtifact Vinay Selvaraj’s post shows you how you can build a continuous integration pipeline to deliver Maven artefacts to AWS CodeArtifact. Java developers often use Apache Maven artefact repositories to share and reuse Maven packages, and so with this solution you can now deploy those Maven packages to an AWS service called AWS CodeArtifact that provides a fully managed pay-as-you-go artefact repository service.
Testing AWS Lambda functions written in Java Jerome Van Der Linden announces the release of aws-lambda-java-tests, an opinionated library that simplifies writing tests when using Java on AWS Lambda. This post is going to go down very well writing unit and integration tests for Lambda can be a challenge, something this project and post aim to help you with.
Processing user-generated content using AWS Lambda and FFmpeg Abbas Nemr shows you how you can use the open source GPLv3 tool FFMpeg combined with AWS Lambda and storage services such as Amazon S3 and Amazon EFS to create serverless media processing workflows. Given some of the typical sizes of these artefacts, this posts covers how you can overcome those limits and still win with serverless.
Amazon Managed Streaming for Apache Kafka (Amazon MSK) now supports Apache Kafka version 2.7.0 for new and existing clusters. Apache Kafka 2.7.0 includes several bug fixes and new features that improve performance. Some key features include the ability to throttle create topic, create partition, and delete topic operations (KIP-599) and configurable TCP connection timeout (KIP-601). For a complete list of improvements and bug fixes, see the Apache Kafka release notes for 2.7.0.
FreeRTOS now supports the ability to update multiple file types in an over-the-air (OTA) update operation. With this feature, FreeRTOS users can classify different types of files (e.g. firmware, certificate, general image) while creating an OTA update job. This enables a FreeRTOS device to identify the category of an incoming update and use it for a specific purpose (e.g. firmware update, certificate rotation, picture download) or target (e.g. secondary microcontroller, external memory) in an embedded application. Learn more here.
FreeRTOS is an an MIT licensed open source, real-time operating system for microcontrollers that makes small, low-power edge devices easy to program, deploy, secure, connect, and manage. You can use AWS IoT Device Management with FreeRTOS devices for an integrated OTA update solution.
Stay in touch with open source at AWS
Your feedback matters!
I have put together a short feedback survey, which I would ask you to take - it will take no more than 2 minutes. You can access here. Many thanks!