Amazon EKS
-
AWS open source newsletter #181
Nov 27, 2023 | 20 minute read
November 27th, 2023 - Instalment #181 Welcome to #181 of the AWS open source newsletter, the place for all your AWS and open source needs. As some of you will know, this is re:Invent week, where many of my colleagues and tens of thousands of developers will be congregating to learn about Cloud, and hopefully discover plenty of open source goodness too. If you are going, make sure to visit the open source booth in the Expo floor.
- oss-newsletter
- aws open source
- Griptape
- PyTorch
- AWS Amplify
- NextJS
- WordPress
- OpenJDK
- Amazon Corretto
- PostgreSQL
- OpenSearch
- SnapStart
- Kong
- AWS Copilot
- MySQL
- dbt
- Apache Hudi
- Apache Flink
- Kubernetes
- Amazon EKS
- Kali
- Karpenter
- Istio
- LangChain
- Pinecone
- Amazon EMR
- Apache Spark
- Prometheus
- Lustre
- OpenZFS
- Apache Kafka
- Lambda PowerTuning
-
AWS open source newsletter #180
Nov 20, 2023 | 20 minute read
November 20th, 2023 - Instalment #180 Welcome to #180 of the AWS open source newsletter, the place for all your AWS and open source needs. As we ramp up to re:Invent, it is good to see that pre:Invent is giving us plenty of open source goodies. In this weeks newsletter, we have some of those in the way of new projects such as res and aws-iatk, but we also have lots of really great content too.
- oss-newsletter
- aws open source
- Ragna
- ezsmdeploy
- SnapStart
- GraalVM
- Amazon Corretto
- Amazon EMR
- Apache Airflow
- LangChain
- AWS Copilot
- Karpenter
- MWAA
- Grafana
- Prometheus
- Amazon EKS
- Kubernetes
- Apache Flink
- Apache Kafka
- Avro
- Apache Cassandra
- Apache Spark
- Red Hat Linux
- Amazon Linux 2023
- NodeJS
- AWS Amplify
- Redis
- MySQL
- PostgreSQL
- eksctl
- MapLibre
- Overture Maps
-
AWS open source newsletter #179
Nov 13, 2023 | 18 minute read
November 13th, 2023 - Instalment #179 Welcome to #179 of the AWS open source newsletter, the place for all your AWS and open source needs. This weeks new projects include an open source tool that provides similar capabilities to AWS Control Tower, a tool for enrolling your Mac based EC2 instances into mobile device management (MDM) solution, a very neat tool to help you compare costs of running your CodePipeline jobs, as well as sample code that shows you how you can use Karpenter to optimise IP address use, examples of using Test Containers with AWS CDK, generative AI technologies such as LangChain, and more.
-
AWS open source newsletter #178
Nov 6, 2023 | 19 minute read
November 6th, 2023 - Instalment #178 Welcome to #178 of the AWS open source newsletter, the place for all your AWS and open source needs. This week we feature more new open source projects for you to practice your four freedoms. We have a useful tool that helps you synchronise your AWS Identity Centre users with the users you provision in your Amazon RDS databases, we share the AWS data solutions framework that helps you build data solutions following opinionated best practices, a resource explorer for your AWS accounts, a guardrails solution for your AWS account, and a couple of demo repositories that take a look at Localstack and RSS.
-
AWS open source newsletter #177
Oct 30, 2023 | 19 minute read
October 30th, 2023 - Instalment #177 Welcome to #177 of the AWS open source newsletter, the Halloween special. You will find no tricks in this edition, only treats, with more new projects for you to check out and content that are a feast for your eyes. This weeks new projects include a tool to help you easily deploy vector databases on Kubernetes, an observability toolkit, a tool to help you benchmark network latency, as well as lots of new demos on generative AI.
- oss-newsletter
- aws open source
- Bottlerocket
- KubeArmor
- NGINX
- Wordpress
- Milvus
- Falcon-40B
- JupyterHub
- Dask
- Flux GitOps
- Crossplane
- Kubernetes
- Amazon EKS
- Babelfish for Aurora PostgreSQL
- PostgreSQL
- Linux
- Apache Hive
- Apache Spark
- Apache Kafka
- Apache Hudi
- Delta Lake
- Apache Iceberg
- OpenSearch
- Dremio
- OpenShift
- OpenCLIP
- Apache Airflow
- MWAA
- Amazon Corretto
- OpenJDK
- AWS CDK
-
AWS open source newsletter #176
Oct 23, 2023 | 19 minute read
October 23rd, 2023 - Instalment #176 Welcome to #176 of the AWS open source newsletter, heading into Autumn and getting ready for the clocks to go back later in the week. How will you use that extra hour? Well, perhaps some of you might use it to check out this weeks new projects, which include a nice cli chat tool that uses Amazon Bedrock, a tool to help simplify deploying VSCode in the Cloud environments, and sample demos and code on chaos engineering, deploying Amazon Bedrock via AWS Lambda, and how you can use IAM Roles Anywhere to authenticate against external identity providers.
- oss-newsletter
- aws open source
- LangChain
- Trivy
- AWS CDK
- Apache Spark
- Pinecone
- Redis
- Postgres
- pgVector
- OpenSearch
- ClickHouse
- Chroma
- Apache Iceberg
- CfnGuard
- Kubernetes
- Amazon EKS
- Karpenter
- Kubecost
- Amazon Corretto
- NextJS
- AWS Amplify
- GraphQL
- Ray
- AWS Distro for OpenTelemetry
- Grafana
- Core WCF
- AWS SAM
- Amazon EMR on EKS
- RabbitMQ
- Swift
-
AWS open source newsletter #175
Oct 16, 2023 | 25 minute read
October 16th, 2023 - Instalment #175 Welcome to #175 of the AWS open source newsletter, back after recharging in the wonderful countyside of Yorkshire. I am publishing this weeks newsletter from Raleigh, North Carolina. All Things Open is happening this week, and you will catch me at the AWS booth where I will be showing off some cool open source stuff (Cedar, Apache Airflow, and a few others), and I also have a talk on Tuesday.
- oss-newsletter
- aws open source
- Redis
- MariaDB
- and PostgreSQL
- Cloud Native Operational Excellence
- CNOE
- Crossplane
- Apache Airflow
- MWAA
- Apache Flink
- Amazon EMR
- Powertools for AWS Lambda
- Kubernetes
- Amazon EKS
- Prometheus
- Grafana
- AWS Distributed OpenTelemetry(ADOT)
- Karpenter
- CoreDNS
- etcd
- Istio
- SUSE
- AWS-LC
- Spring Boot
- SOCI
- Apache Kafka
- Amazon Corretto
- AWS CDK
- Amazon Linux
- Bottlerocket
- cdk8s
- AWS Amplify
- Stable Diffusion
- NextJS
-
AWS open source newsletter #169
Aug 14, 2023 | 26 minute read
August 14th, 2023 - Instalment #169 Welcome to #169 of the AWS open source newsletter, featuring the latest and greatest open source news, projects, videos, and community content that you need to know about. Featured in this weeks edition we have more great projects, including a new ODBC driver for Amazon Timestream database, a nice tool to simplify your ssh tunnelling, an essential VSCode extension for working with Cedar policies, a couple of projects that help you shift left and validate / monitor your policies, a solution to help you monitor your Apache Kafka environments, as well as some great sample applications.
- oss-newsletter
- aws open source
- OpenSearch
- AWS CDK
- Juypter AI
- dbt
- Apache Airflow
- Managed Workflows for Apache Airflow
- MWAA
- Cedar
- cfnguard
- Grafana
- Prometheus
- Apache Kafka
- Amazon Timestream
- Open Cybersecurity Schema Framework
- OCSF
- AWS Lambda Web Adapter
- Smithy
- Apache Spark
- Linux
- Amazon Linux
- AWS ParallelCluster
- PostgreSQL
- Spring Boot
- Amazon EKS
- Kubernetes
- Mountpoint for Amazon S3
- MySQL
- Lustre
- OpenZFS
- Redis
- Amazon EMR
- Karpenter
- Seekable OCI
- SOCI
- Firecracker
-
AWS open source newsletter #168
Aug 7, 2023 | 24 minute read
August 7th, 2023 - Instalment #168 Welcome to #168 of the AWS open source newsletter, the only newsletter that features the freshest open source and AWS content* New projects for you to feast on in this issue include the obligatory projects that look at how Generative AI can help developers be more productive, in this case by making documentation more relevant and easier to find and to help with code reviews, a nice tool to help you query your AWS Identity Access Management (IAM) policies, and a security focused tool to help you search for potentially incorrect configured S3 buckets.
-
AWS open source newsletter #167
Jul 31, 2023 | 22 minute read
July 31st, 2023 - Instalment #167 Welcome to #167 of the AWS open source newsletter, and another edition packed with new open source projects for you to explore. Whether you are new to this newsletter, or returning (we thank you!) there is something for you. This week we have projects that will help use Terraform to deploy your monolith applications, a tool to accelerate your GraphQL building when working with Amazon DynamoDB, a project that will help you introduce chaos without changes to your code, and a couple of really nice demos of using generative AI that allows you to use natural language to query your data.
- oss-newsletter
- aws open source
- Terraform
- Rust
- LangChain
- Streamlit
- PostgreSQL
- Supabase
- OpenSearch
- AWS SAM
- AWS CDK
- Powertools for AWS Lambda
- Open Application Model
- OAM
- Traefik
- Keycloak
- FastAPI
- PyTorch
- MySQL
- IAM Roles Anywhere
- Apache Iceberg
- MQTT
- Amazon EMR
- Kubernetes
- Amazon EKS
- Apache Hudi
- Go
- Cedar
- RabbitMQ
- Pacu
-
AWS open source newsletter #165
Jul 17, 2023 | 17 minute read
July 17th, 2023 - Instalment #165 Welcome to #165 of the AWS open source newsletter, the only* newsletter that brings you the best and latest open source content. We have some great new projects this week, including a tool for IoT developers to help you validate your SQL statements, a command line interface tool for Amazon Verified Permissions, an Amazon DynamoDB estimation tool, and more. Also featured this week is content on Apache Iceberg, OpenSearch, PostgreSQL, Kubernetes, Power Tools for AWS Lambda, Spring Boot, Babelfish for Aurora PostgreSQL, Karpenter, Apollo GraphQL, JupyterHub, dbt, Apache Airflow, Cedar, and Apache Flink.
-
AWS open source newsletter #164
Jul 10, 2023 | 17 minute read
July 10th, 2023 - Instalment #164 Welcome to #164 of the AWS open source newsletter. As always, we search high and low for the best and latest open source content, and I think you will love what we have lined up this week. New projects this week will help you implement single table designs easily on Amazon DynamoDB, an experimental project to help you get to grips with Cedar, a comprehensive clickstream analytics project for your applications, web sites, and mobile applications, and some cool projects to help you with edge and hybrid use cases.
- oss-newsletter
- aws open source
- Apache Flink
- Apache Airflow
- Kubernetes
- Amazon EKS
- AWS Lambda Powertools
- Spring Boot
- Linux
- Apache Parquet
- OpenZFS
- OpenSearch
- Mountpoint for Amazon S3
- PostgreSQL
- AWS Amplify
- Next.js
- AWS Distro for OpenTelemetry
- Grafana
- Babelfish for Aurora PostgreSQL
- AWS ParallelCluster
- Consul
- Apache Iceberg
- Cedar
- Steampipe
- VS Code Server
- nomad
-
AWS open source newsletter #159
Jun 5, 2023 | 16 minute read
June 5th, 2023 - Instalment #159 Welcome to #159 of the AWS open source newsletter, where we try and share all the important open source news, projects, events, and content that open source builders want. This week we have new projects that include tools to help you generate test data, an accelerator for stable diffusion, tools to help you with managing application credentials and variables, a new connector for Apache Kafka, and more!
-
AWS open source newsletter #158
May 30, 2023 | 21 minute read
May 30th, 2023 - Instalment #158 Welcome Hello and welcome to the AWS open source newsletter, #158. I hope some of you were able to catch the last episode of season two of Build on Open Source where we looked at some of the projects featured in this newsletter (specctl, eksdemo, and ec2-spot-placement-score-tracker). As always we pride ourself on this newsletter on giving you the newest, shiniest open source projects and this week we have some really great ones to share with you.
-
AWS open source newsletter #157
May 22, 2023 | 19 minute read
May 22nd, 2023 - Instalment #157 Welcome Hello and welcome to the AWS open source newsletter, #157. Apologies for the lack of newsletter last week, but hopefully this week will make up for that as we have a bumper selection of great open source content for you. This weeks new projects include repos that help you get OpenEMR up and running (“host-openemr-on-aws-fargate”), two new security related open source projects that you definitely need to check out, (“cedar” and “snapchange”), integration of clickstream analytics using Swift (“clickstream-swift”), deployment of Backstage to serve up access to your AWS resources, (“app-development-for-backstage-io-on-aws”), a tool to help you clean up ecs tasks definitions (“aws-ecs-task-definition-cleanup”) and many more.
-
AWS open source newsletter #156
May 8, 2023 | 21 minute read
May 8th, 2023 - Instalment #156 Welcome Hello and welcome to the AWS open source newsletter, #156, the newsletter that just keeps on giving….in this case, keeps giving you brand new open source projects to practice your four freedoms on. So what do we have for you this week? Coming up later in this newsletter we have projects such as “sustainability-scanner” helps you check your Cloudformation templates against sustainability good practices, “synthtable” helps you create synthetic data for different use cases, “neptune-gremlin-client” a Java based Gremlin client, “s3zipper” a tool to quickly download entire S3 buckets, “chataws” a nice demo of how you can use ChatGPT to aid your AWS deployments, and plenty of other great projects.
-
AWS open source newsletter #154
Apr 24, 2023 | 22 minute read
April 24th, 2023 - Instalment #154 Welcome Hello and welcome to the AWS open source newsletter, #154, the newsletter that just keeps on giving….in this case, keeps giving you brand new open source projects to practice your four freedoms on. We have another great selection of projects for you as always, starting off with “cfn-teleport” an essential cli tool for Cloudformation users, “aither” an interesting collaborative development tool using virtualised desktops on containers, “tabular-column-semantic-search” a tool to help you find similar types of data in your data lakes, “resource-lister” and “komiser” tools that help you manage your AWS resources, “resource-utilization” helps you track your AWS resource utilisation, “iot-network-traffic-control-and-load-testing-simulator” an interesting load and chaos testing example, and more!
- oss-newsletter
- aws open source
- Apache Oozie
- Apache Airflow
- Deep Java Library
- DJL
- mwaa-local-runner
- MWAA
- Trusted Language Extensions for PostgreSQL
- Supabase
- PostgreSQL
- Jupyter
- Grafana
- Opus
- Papermill
- Apache Spark
- HiveQL
- Amazon EKS
- Kubernetes
- Amazon EMR
- RStudio
- USBGuard
- Amazon Corretto
- AWS Amplify
- Apache Hive Metastore
- LoRaWAN
- gMSA
- Python
- OpenSearch
- AWS Copilot
- Marten
- Flutter
-
AWS open source newsletter #153
Apr 17, 2023 | 19 minute read
April 17th, 2023 - Instalment #153 Welcome Hello and welcome to the AWS open source newsletter, #153 as featured in the latest episode of Build on Open Source (S2E5) . We have lots of great projects for you this week, with a strong chatGPT influence. “pg_gpt”, “cw-logs-insights-gpt”, and “aiws” all integrate chatGPT to help you do different things on AWS, “semantic-search-aws-docs” is a very interesting demo on how to build a more coherent search for your documentation, “aws-chime-chat-demo” a very nice demo using the Chime SDK, “ckia” is an open source AWS Trusted Advisor tool, “AWS_ED” helps you keep your local IP in sync with external DNS records, “cfnctl” provides a Terraform like cli experience to CloudFormation, and more!
-
AWS open source newsletter #152
Apr 10, 2023 | 16 minute read
April 10th, 2023 - Instalment #152 Welcome Hello and welcome to the AWS open source newsletter, #152 an Easter special. This week sees more great new projects including, “redshift-test-drive” a set of essential tools for Amazon Redshift users, “simple-database-archival-solution” a nice tool to help you archive your data, “attribution-gen” a Go tool to help you build open source attribution documents, “aws-glue-data-catalog-federation” a library to help you federate your Glue catalog, “subnet-utilization-monitor-for-amazon-vpc” a handy tool to keep on top of your IP address allocation, “AlexaGPT” a demo of integrating Alexa with you know what, and more!
-
Self managed Apache Airflow with Data on EKS
Mar 22, 2023 | 11 minute read
I have written in the past about how you can get started with Apache Airflow using the AWS managed service, Managed Workflows for Apache Airflow. But what if you want to self managed Apache Airflow? When I speak with developers, there are sometimes reasons why a managed service might not fit their needs. Some of the common things that come up include: whether you need the increase level of access, a greater level of control of the configuration of Apache Airflow have the need to have the very latest versions or features of Apache Airflow if you have the need to run workflows that use more resources that managed services provide (for example, need significant compute) Total Cost Ownership One thing to consider when assessing managed vs self managed is the cost of the managed service against the total costs of you having to do the same thing.
-
AWS open source newsletter #146
Feb 27, 2023 | 17 minute read
Feb 27th, 2023 - Instalment #146 Welcome to edition #146 of the AWS open source newsletter. This week we have another great selection of brand new, shiny open source projects for you to practice your four freedoms on. Some of the projects this week include “aws-lambda-web-adapter” that will help you build portable Lambda functions, “aws-marketplace-cli” if you want to escape the GUI when working with AWS Marketplace, this one is for you, ““otel-config-validator” helps you sanity check your configuration files, “aws-serverless-openai-chatbot-demo” build a personal assistance powered by ChatGPT, “drone-video-analysis” if you have wanted to use AI services to process your drone video footage, check this project out, “mask-words-in-image” a nice command line tool to quick mask data based on your needs, and many more cool projects.
-
AWS open source newsletter #145
Feb 20, 2023 | 24 minute read
Feb 20th, 2023 - Instalment #145 Welcome to edition #145 of the AWS open source newsletter. I hope some of you were able to catch the new Build on Open Source show we live streamed last Friday. You can catch up and replay the session by clicking on this link, where we went over a number of projects from this and a few previous newsletters, and we had special guest Valter who walked us through his project terraform-dev-containers.
-
AWS open source newsletter #144
Feb 5, 2023 | 19 minute read
Feb 5th, 2023 - Instalment #144 Welcome to edition #144 of the AWS open source newsletter, and another week of great new open source projects for you to try out. Some of the treats in store for you this week include “dynamodb-shell”, a project that provides a cli to your favourite AWS database, “precloud” a tool to help you catch issues with your configuration before you deploy, “node-latency-for-k8s” a tool to analyse your node logs, “stepfunctions-lambda-ec2-ssm” a very nice way of using step functions to overcome the 15 minute timeout of your lambda functions, “terraform-ec2-image-builder-container-hardening-pipeline” a very cool example of how to build an EC2 image hardening pipeline using Terraform, and “cloudtrail-event-fuzzy-viewer” a tool to copy your AWS CloudTrail events and then fuzzy search them on the command line.
-
Configuring the KubernetesPodOperator on Managed Workflows for Apache Airflow (MWAA) - non OIDC Amazon EKS Clusters
Jan 26, 2023 | 5 minute read
Configuring the KubernetesPodOperator on Managed Workflows for Apache Airflow (MWAA) - non OIDC Amazon EKS Clusters Today I came across an interesting question around the use of the KubernetesPodOperator working on EKS Clusters where you have not configured OIDC. They had followed my blog post, and when it came to running the DAG, they got the following error: [2023-01-26, 13:03:18 UTC] {{kubernetes_pod.py:566}} INFO - Creating pod mwaa-pod-test.0ab20a7075b84175b2a9a3fe32796f53 with labels: {'dag_id': 'kubernetes_pod_example_iam_authenticator', 'task_id': 'pod-task', 'execution_date': '2023-01-26T130310.
-
AWS open source newsletter #142
Jan 23, 2023 | 14 minute read
January 23rd, 2023 - Instalment #142 Welcome Welcome to edition #142 of the AWS open source newsletter. We have another great round up of new projects for you to get stuck into. Here are just a taste of some of the projects, kicking off with “sls-mentor” a new tool to help you assess your serverless applications, “subnet-watcher”, a tool to help you monitor your IP addresses, “aws-cdk-web-administered-apps” a very nice reference solution for applications that have a user and admin component, “serverless-newsletter-app” if you are looking for a newsletter solution and want to host your own, look here first, “aws-iot-with-privatelink” shows you how you use private networks for your IoT traffic, “emr-spark-benchmark” benchmarking tool for assessing your Amazon EMR environments, and “update-aws-ip-ranges” keep automatically updated on Amazon’s IP address ranges.
-
AWS open source newsletter #141
Jan 15, 2023 | 13 minute read
January 16th, 2023 - Instalment #141 Welcome Welcome to the AWS open source newsletter of 2023, edition #141. This week we have more new projects for you to practice your four freedoms, including “distributed-compute-on-aws-with-cross-regional-dask”, a solution to simplify distributed compute using Dask, “amazon-emr-serverless-image-cli” a tool to verify your Amazon EMR custom container images, “serverless-run-watch” a tool to help accelerate your local development if you are using the Serverless Framework, “aws-sso-auto-expand-accounts” a quick browser extension for those using AWS SSO, “basti” a cool Bastion Host alternative, “klotho” generate cloud native code from your code, “amazon-route53-hosted-zone-sync” a nice solution for hybrid DNS use cases, and many more.
-
Running the KubernetesPodOperator in different AWS accounts when using Amazon Managed Workflows for Apache Airflow v2.x
Jan 9, 2023 | 18 minute read
Running KubernetesPodOperator in different AWS accounts update August, 14th I wanted to update to newer version of MWAA, so I have tested the original blog post against EKS 1.24 and MWAA version 2.4.3. I also had a few messages about whether this would work across different AWS regions. The good news is that it does. I have also put together a repo for this here I thought that I would also check/update that it works for newer versions of MWAA, so I had 2.
-
AWS open source newsletter #140
Jan 9, 2023 | 19 minute read
January 9th, 2023 - Instalment #140 Welcome Happy New Year and welcome to the first AWS open source newsletter of 2023, edition #140. If you have not already checked it out, I put together a short retrospective summary of 2022 in the post, AWS open source newsletter - 2022 in review. There are some interesting facts and figures in there. I am also taking time to collect feedback from readers to help shape where this newsletter goes in 2023.
- oss-newsletter
- aws open source
- MySQL
- PostgreSQL
- Next.js
- AWS SDK for Java
- AWS ParallelCluster
- Docker
- MariaDB
- Amazon EKS
- Kubernetes
- MQTT
- ArgoCD
- AWS Distro for OpenTelemetry
- Prometheus
- DAMON
- Crossplane
- Log4Shell
- .NET
- Apache Spark
- Apache Kafka
- Apache Flink
- Apache Pinot
- Apache Superset
- Apache NiFi
- Delta Lake
- OpenShift
- Redis
- Red Hat Enterprise Linux
- SUSE Linux Enterprise Server
- Ubuntu Pro
- AWS Copilot
- RabbitMQ
- Apache Airflow
- Rust
- Terraform
- Amazon EMR
- Apache ShardingSphere-Proxy
-
AWS open source news and updates #101
Feb 21, 2022 | 12 minute read
Feb 21st, 2022 - Instalment #101 Newsletter #101. There is nothing basic and fundamental about edition 101 of the AWS open source newsletter, with another great round up of new open source projects including eks-creation-engine from the folks at Lightspin helping you all to stay safer with this handy tool you should check out, idp-scim-sync to help users of AWS SSO who want to synchronise with their Google Workspace Directory, typecart an analysis tool for proof evolution and many other great projects and sample code.
-
AWS open source news and updates #100
Feb 14, 2022 | 12 minute read
Feb 14th, 2022 - Instalment #100 Newsletter #100. Happy Valentines everyone, and welcome to this landmark 100st edition of this newsletter. This week we celebrate the love that many builders have for open source with more great new open source projects and content. Cuddle up to new projects that will help you build scalable systems, simplify your work with AWS DynamoDB, integrate your .NET applications with OpenSearch, keep on top of your VPC networks, and more.
-
AWS open source news and updates #99
Feb 7, 2022 | 12 minute read
Feb 7th, 2022 - Instalment #99 Newsletter #99. While Nena gave you 99 red balloons, I give you the latest version of the AWS open source news letter. This week we feature more great new open source projects including a project to help you with drift detection in your CloudFormation stacks, new Terraform modules, an open-source prometheus exporter, some AWS CDK resources and sample projects and more. This weeks AWS and Community posts cover PostgreSQL, Apache Airflow, AWS CDK, Redis, GraphQL, Apollo GraphQL, Kubernetes, AWS EKS and more.