skip to content W3C SVG

Posts

  • Manage context rot by exploring new experimental features in Amazon Q CLI

    Calendar Jul 2, 2025 |  Stopwatch  12 minute read

    Like many folk who have been spending their time with AI Coding Assistants like Amazon Q Developer and Amazon Q CLI, understanding how to manage context is one of the key things you need to develop intuition for to improve the outputs these tools give you. More recently I have started hearing about new terms such as "context rot", and others exploring the field of context engineering. Understanding how to manage your context will be key to your success.

  • Vibe coding with Amazon Q CLI - creating some load testing code

    Calendar Jun 24, 2025 |  Stopwatch  8 minute read

    I have a long history with testing, and specifically load testing. Many many years ago, I spent many a happy hour designing, building, running, and then assessing load tests for the applications I was working with. It was one of the first technical interviews I ever did and it was also how I found AWS, spinning up Centos load runners to simulate thousands of virtual test users. Ahh the good old days!

  • Running Model Context Protocol (MCP) Servers on containers using Finch

    Calendar May 2, 2025 |  Stopwatch  6 minute read

    I was chatting with AWS Hero Matt Lewis on the topic of how to run MCP Servers via a container image, and realised that I had not actually tried this yet. So this post was inspired by that conversation, and I hope it helps anyone else who is looking to try it out. In a previous post I introduced how Amazon Q CLI now supports Model Context Protocol (MCP) (check out Configuring Model Context Protocol (MCP) with Amazon Q CLI for more details).

  • Configuring Model Context Protocol (MCP) with Amazon Q CLI

    Calendar May 1, 2025 |  Stopwatch  7 minute read

    Amazon Q CLI is a next generation developer tool that brings IDE-style autocomplete and agentic capabilities to your terminal. I have spent a lot of time recently writing about this amazing tool, and so was super excited by the news today that with the v1.9.x release, Amazon Q CLI now supports Model Context Protocol (MCP) for tools use. What is Model Context Protocol (MCP)? if you have not heard about MCP (where have you been?

  • From PHP to Python - porting a Reddit clone with the help of Amazon Q Developer

    Calendar Mar 16, 2025 |  Stopwatch  12 minute read

    In this blog post I share how I was able to use Amazon Q Developer CLI to refactor code from one programming language to another Many years ago, I spend many happy years developing code in PHP. It was such an accessible and tactile language, with a great community momentum that continues to this day. One of the things I used to enjoy, was trying out many of the open source clones of well known websites that the PHP community would release.

  • Using Amazon Q Developer CLI to build applications from the command line

    Calendar Mar 9, 2025 |  Stopwatch  2 minute read

    I have been writing a lot recently about AI Coding Assistants, and I have been mostly using Amazon Q Developer within VSCode. This week though, saw a very nice update to the Amazon Q Developer CLI, a separate download that provides you with Amazon Q within your command line. It is available for MacOS and various flavours of Linux (you can download it from here. I wanted to see how good the recent update is, so I decided to try and build a quick web application from the command line.

  • Using AI coding assistants to develop a spinning wheel web application

    Calendar Jan 9, 2025 |  Stopwatch  6 minute read

    This post was inspired by my colleague Matheus, who has used the aws spinning wheel (see blog post here) to help make running re:Cap’s a little more interesting and fun. That is a really neat solution, but I didn’t want to use that project as it was a little more comprehensive than I needed. As it is the new year, and I am starting to get hands on with all the really cool new features in Amazon Q Developer announced at re:Invent 2024, I thought I would create a simpler version.

  • Amazon Q Developer Tips: 25 tips to supercharge your development

    Calendar Dec 25, 2024 |  Stopwatch  4 minute read

    Tip 25 - Tips to help you get the most out of Amazon Q Developer Over the past weeks I have shared daily tips on how you can get the best out of AI coding assistants like Amazon Q Developer. Over the festive period some of you may find yourself with some free time, or perhaps you are going to be setting yourself some new years resolutions. For those that are planning on building new projects, or perhaps wanting to learn something new, then I hope this set of tips and tricks will help you towards that goal.

  • Amazon Q Developer Tips: No.24 Amazon Q Developer community resources

    Calendar Dec 24, 2024 |  Stopwatch  3 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.23 Debugging with Amazon Q

    Calendar Dec 23, 2024 |  Stopwatch  6 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.22 Amazon Q Keyboard shortcuts

    Calendar Dec 23, 2024 |  Stopwatch  4 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.21 Amazon Q Developer Agents - /test

    Calendar Dec 21, 2024 |  Stopwatch  5 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.20 Amazon Q Developer Agents - /review

    Calendar Dec 20, 2024 |  Stopwatch  7 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.19 Amazon Q Developer Agents - /doc

    Calendar Dec 19, 2024 |  Stopwatch  7 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.18 Amazon Q Developer Agents - /dev

    Calendar Dec 18, 2024 |  Stopwatch  5 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.17 Choose the right tool

    Calendar Dec 17, 2024 |  Stopwatch  4 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.16 How to tackle LLM training data cutoff

    Calendar Dec 16, 2024 |  Stopwatch  5 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.15 CHat Orientated Programming (CHOP)

    Calendar Dec 15, 2024 |  Stopwatch  7 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.14 Navigating through your prompt history

    Calendar Dec 14, 2024 |  Stopwatch  2 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.13 Generating perfect functions

    Calendar Dec 13, 2024 |  Stopwatch  3 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.12 Mastering in-line prompts

    Calendar Dec 12, 2024 |  Stopwatch  6 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.11 Scaffolding

    Calendar Dec 11, 2024 |  Stopwatch  3 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.10 Personalise Amazon Qs output

    Calendar Dec 10, 2024 |  Stopwatch  3 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.9 Using import statements to direct suggestions

    Calendar Dec 9, 2024 |  Stopwatch  2 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.8 Understanding Context

    Calendar Dec 8, 2024 |  Stopwatch  8 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.7 Generating better prompts

    Calendar Dec 7, 2024 |  Stopwatch  7 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.6 Exploring Use Cases

    Calendar Dec 6, 2024 |  Stopwatch  11 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.5 Break down large problems

    Calendar Dec 5, 2024 |  Stopwatch  6 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.4 No AWS account needed

    Calendar Dec 4, 2024 |  Stopwatch  4 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.3 Enable Amazon Q Developer Workspace Index

    Calendar Dec 3, 2024 |  Stopwatch  8 minute read

    In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer. Each day I will share a new tip, categorised against a few themes and topics. If you have your own tips or your own experiences with tips I share, please use the comment feature or feel free to contact me directly. Check out the previous tips here.

  • Amazon Q Developer Tips: No.2 IDE Layout

    Calendar Dec 2, 2024 |  Stopwatch  3 minute read

    In the previous post I shared the importance of forming daily habits to help you discover your own flow when working with AI coding assistants like Amazon Q Developer. I also shared the first tip - make sure you keep your Amazon Q Developer plugin auto updated. In this series I will be sharing daily hints and tips to help you get ahead and start to accelerate you on your own journey with Amazon Q Developer.

  • Daily Tips to supercharge your Amazon Q Developer experience

    Calendar Dec 1, 2024 |  Stopwatch  2 minute read

    The importance of building muscle memory When I talk to developers about the use of AI coding assistants, one of the key takeaways is that daily habits are one of the most effective tools in helping to build muscle memory and explore new possibilities that these tools provide. To help folk with this, over the next few weeks I am going to share daily hints and tips that I have found useful and made my use of Amazon Q Developer more effective.

  • Using Finch to run Apache Airflow using mwaa-local-runner

    Calendar Feb 12, 2024 |  Stopwatch  9 minute read

    I show you how you can use the Finch to run Apache Airflow using the mwaa-local-runner tool, and how you can do this for your applications too As some of you may know, I have been creating content on Apache Airflow for a few years now. One of the open source projects that AWS has produced to make it easier for developers to get started with Apache Airflow, is mwaa-local-runner.

  • Writing simple Python scripts faster with Amazon Q

    Calendar Jan 11, 2024 |  Stopwatch  20 minute read

    The future is here folks! I share my experiences using Amazon Q in the IDE, and how I was able to write more quickly and with less friction. Some of you might be familiar with the AWS open source newsletter, a weekly publication that features a curated set of content for open source developers on AWS. I rely on a number of folk to share with me interesting open source projects that I feature, but I wanted to see if I could create an automated script that I could run on a weekly schedule that would find new open source projects for me to review.

  • Adding Amazon Bedrock Llama2 as an assistant in Ragna

    Calendar Nov 16, 2023 |  Stopwatch  4 minute read

    Adding a new assistant in Ragna Following up from my previous post on Ragna, I wanted to share following the announcement of Meta’s Llama2 13b model availability within Amazon Bedrock, how you can incorporate that. I have also put together a GitHub repo that shares the code, something that I got quite a few questions from the original post. Adding Meta’s Llama2 As with adding Amazon Bedrock’s Anthropic’s Claude support, it was pretty straight forward to modify the original code to add support for Llama2.

  • Unboxing Ragna: Getting hands on and making it to work with Amazon Bedrock

    Calendar Nov 12, 2023 |  Stopwatch  27 minute read

    Unboxing Ragna: Getting hands on and making it to work with Amazon Bedrock I am always on the look out for interesting new projects to check out, and this week I came across Ragna, an open source Retrieval Augmented Generation RAG orchestration framework. It is a new project with a committed and active community, so I wanted to find out more about this project. What piqued my interest was reading this blog post, Unveiling Ragna: An Open Source RAG-based AI Orchestration Framework Designed to Scale From Research to Production which takes a look at the background, or as I like to think of it, the “scratch that needed to be itched”.

  • Getting gnarly with AI - a quick look at Griptape, an enterprise ready alternative to LangChain

    Calendar Sep 5, 2023 |  Stopwatch  14 minute read

    Getting started with Griptape Getting gnarly with AI When I was much much younger, I was known to (very occasionally) drop an ollie, kick-turn, or very occasionally dare a drop in on my trusty skateboard. Thanks to the grip tape, my shoes would stick to the board, and gave me the confidence I could try these tricks. I was always grateful to that grip tape and who knew all these years later, I would become re-aquatinted with it (albeit in a different form!

  • Using Amazon CodeWhisperer as my scripting sidekick

    Calendar Sep 4, 2023 |  Stopwatch  5 minute read

    How are you using these new AI coding assistants like Amazon CodeWhisperer? I want to share a quick story about how I am finding some success with using these tools, which I hope will encourage you to experiment for yourselves. As I work on new projects, demos, blog posts, I often need to do supporting activities such as creating or cleaning up AWS environments, which I sometimes do by hand but mostly do via the AWS CLI or via Python scripts and boto3.

  • A look at airflowctl, a tool to help you manage Apache Airflow projects

    Calendar Aug 14, 2023 |  Stopwatch  10 minute read

    I have written in the past about setting up developer environments and tools when working with Apache Airflow. Today I came across a new tool from Kaxil Naik, directory of engineering at Astronomer and all round Apache Airflow good guy. Kaxil has put together airflowctl, a command-line tool for managing Apache Airflow™ projects, and making it super easy to get up and running. What does it do? Well, it helps you install and use different versions of Apache Airflow, work with Variables and Connections, provide live logs, and more.

  • Deploying a serverless web analytics solution for your websites

    Calendar Jun 29, 2023 |  Stopwatch  12 minute read

    Update, August 1st I have been running this project for five weeks now, so have a better understanding of costs. My bill in July was $0.85, and I suspect this will be reasonably stable over the coming months. I think this represents great value! I will report back at the end of the year and update this Like many folk, I run a personal blog. This blog runs Hugo and turns my markdown pages into static content, which I then deploy on Netlify.

  • Using CDK to deploy AWS managed Active Directory

    Calendar Jun 12, 2023 |  Stopwatch  5 minute read

    CIFS (Common Internet File System) and SMB (Server Message Block) are both Windows file-sharing protocols used in storage systems. As part of a new demo/blog post that looks at how to use data stored on SMB/CIF file shares with Apache Airflow, I have been exploring the various options of creating SMB/CIF compatible resources. (There are LOTS of ways you could do this, so there is plenty for me to play around with!

  • Integrating Keycloak as my Identity Provider for IAM Identity Centre: Part two, configuring Keycloak as my Identity provider

    Calendar Jun 12, 2023 |  Stopwatch  5 minute read

    This is the follow up post to Integrating Keycloak as my Identity Provider for IAM Identity Centre: Part one, deploying Keycloak on AWS, where I looked at how to deploy Keycloak on AWS in order to have an Identity Provider to use when configuring AWS Identity Centre. In this post, I am going to use that setup, and show you how I configured it to integrate with AWS Identity Centre to provide access to my AWS resources.

  • Integrating Keycloak as my Identity Provider for IAM Identity Centre: Part one, deploying Keycloak on AWS

    Calendar Jun 6, 2023 |  Stopwatch  15 minute read

    Integrating Keycloak as my Identity Provider for IAM Identity Centre: Part one, deploying Keycloak on AWS “It was the best of times, it was the worst of times…” A Tale of Two Cities It started out innocently enough. As part of working on a new blog post, I needed a way to use an open source tool called saml2aws that generates AWS short lived credentials that you can use to access your AWS resources.

  • Getting mwaa-local-runner up on AWS Cloud9

    Calendar Apr 17, 2023 |  Stopwatch  5 minute read

    Here is a quick recipe if you are looking to get mwaa-local-runner up and running on your Cloud9 developer setup. This might not be the most optimised way, so I am very happy to received suggestions on how to improve this. What I will cover here is how to deploy mwaa-local-runner onto a standard Cloud9 IDE, deployed in a default VPC. Updating my AWS Cloud9 environment The first thing I needed to do was to increase the size of my local disk as Cloud9 only provides 10gb of storage.

  • Exploring Shell Launch Scripts on Managed Workflows for Apache Airflow (MWAA) and mwaa-local-runner

    Calendar Apr 17, 2023 |  Stopwatch  9 minute read

    Managed Workflows for Apache Airflow (MWAA) recently launched a new feature that a lot of folk had been asking for, which was the ability to add additional libraries, binaries, or environment variables when launching Airflow workers. If you missed the announcement, Amazon MWAA now supports Shell Launch Scripts, this new capability allows you to easily do this by creating a script and then configuring your MWAA environments to use that script during the start-up phase.

  • Working with Managed Workflows for Apache Airflow (MWAA) and Amazon Redshift

    Calendar Apr 7, 2023 |  Stopwatch  19 minute read

    Working with Managed Workflows for Apache Airflow (MWAA) and Amazon Redshift I was recently looking at some Stack Overflow questions from the AWS Collective and saw a number of folk having questions about the integration between Amazon Redshift and Managed Workflows for Apache Airflow (MWAA). I thought I would put together a quick post that might help folk address what I saw were some of the common challenges. There is some code that accompanies this post, which you can find at the GitHub repository cdk-mwaa-redshift.

  • Self managed Apache Airflow with Data on EKS

    Calendar Mar 22, 2023 |  Stopwatch  11 minute read

    I have written in the past about how you can get started with Apache Airflow using the AWS managed service, Managed Workflows for Apache Airflow. But what if you want to self managed Apache Airflow? When I speak with developers, there are sometimes reasons why a managed service might not fit their needs. Some of the common things that come up include: whether you need the increase level of access, a greater level of control of the configuration of Apache Airflow have the need to have the very latest versions or features of Apache Airflow if you have the need to run workflows that use more resources that managed services provide (for example, need significant compute) Total Cost Ownership One thing to consider when assessing managed vs self managed is the cost of the managed service against the total costs of you having to do the same thing.

  • VSCode and Apache Airflow

    Calendar Feb 20, 2023 |  Stopwatch  8 minute read

    VSCode and Apache Airflow In this short post, I wanted to highlight how you can use a VSCode plugin to work with a local running instance of Apache Airflow to improve the developer experience. This post was inspired by a tweet from Kaxil Naik who was asking about what features developers are looking for when using VSCode and Pycharm and Apache Airflow. In this post I will show you how you can configure mwaa-local-runner, an open source project that provides you with an easy way to get a local Apache Airflow environment up and running (that is configuration wide, aligned to the Amazon Managed Workflows for Apache Airflow service MWAA), together with some VSCode plugins.

  • sbomqs, an open source tool to quality check your SBOMS

    Calendar Feb 2, 2023 |  Stopwatch  6 minute read

    When putting together a previous post on how to use open source tools to create a software bill of materials (SBOM), Ritesh Noronha alerted me to another project, sbomqs that aims to simplify the evaluation of SBOM quality for both producers and consumers. A quality SBOM is one that is accurate, complete, and up-to-date. It should accurately reflect the components and dependencies used in the software application, including their version and optionally any known vulnerabilities.

  • Building a software bill of materials (SBOM) using open source tools

    Calendar Feb 1, 2023 |  Stopwatch  6 minute read

    This is the second post exploring how you can use open source tools to help you build a stronger defence against common software supply chain attacks. In this blog post, I look at syft, an open source CLI tool and Go library for generating a Software Bill of Materials (SBOM) from container images and filesystems. We will use examples and build on the previous post, Getting hands on with Sigstore Cosign on AWS.

  • Getting hands on with Sigstore Cosign on AWS

    Calendar Jan 31, 2023 |  Stopwatch  13 minute read

    Getting hands on with Sigstore Cosign on AWS I am currently putting together some content around how you can use a number of open source tools to help build a stronger defence against common software supply chain attacks. In this blog post, I look at emerging tools from Sigstore, and focus in this post on Cosign, a tool that supports container image signing, verification, and storage in an Open Container Initiative (OCI) registry.

  • Configuring the KubernetesPodOperator on Managed Workflows for Apache Airflow (MWAA) - non OIDC Amazon EKS Clusters

    Calendar Jan 26, 2023 |  Stopwatch  5 minute read

    Configuring the KubernetesPodOperator on Managed Workflows for Apache Airflow (MWAA) - non OIDC Amazon EKS Clusters Today I came across an interesting question around the use of the KubernetesPodOperator working on EKS Clusters where you have not configured OIDC. They had followed my blog post, and when it came to running the DAG, they got the following error: [2023-01-26, 13:03:18 UTC] {{kubernetes_pod.py:566}} INFO - Creating pod mwaa-pod-test.0ab20a7075b84175b2a9a3fe32796f53 with labels: {'dag_id': 'kubernetes_pod_example_iam_authenticator', 'task_id': 'pod-task', 'execution_date': '2023-01-26T130310.

  • Running the KubernetesPodOperator in different AWS accounts when using Amazon Managed Workflows for Apache Airflow v2.x

    Calendar Jan 9, 2023 |  Stopwatch  18 minute read

    Running KubernetesPodOperator in different AWS accounts update August, 14th I wanted to update to newer version of MWAA, so I have tested the original blog post against EKS 1.24 and MWAA version 2.4.3. I also had a few messages about whether this would work across different AWS regions. The good news is that it does. I have also put together a repo for this here I thought that I would also check/update that it works for newer versions of MWAA, so I had 2.

  • Experimenting with digital lanyards - introducing the Badger2040

    Calendar May 9, 2022 |  Stopwatch  6 minute read

    Experimenting with digital lanyards As someone who attends events on a regular basis, I have spent a fair bit of time over the years looking at interesting ways to engage with attendees. One of the problems I was looking to solve was how do I share useful information with attendees without having to interrupt the conversations (something that typically happens as I try and find those links on my mobile phone).

  • Contributing to the Apache Airflow project - Part Two

    Calendar Mar 11, 2022 |  Stopwatch  11 minute read

    This is the second and concluding post providing an overview of the experience and journey contributing to the Apache Airflow project. You can catch Part One here. Contributing to Apache Airflow - Part Deux In Part One of this series, we took our first steps in contributing to the Apache Airflow project. With a little bit more knowledge and experience, our first interactions with the Airflow community, we are ready to start exploring how the code works and see how we might go about fixing this.

  • Orchestrating hybrid workflows using Amazon Managed Workflows for Apache Airflow (MWAA)

    Calendar Mar 7, 2022 |  Stopwatch  46 minute read

    Using Apache Airflow to orchestrate hybrid workflows In some recent discussions with customers, the topic of how open source is increasingly being used as a common mechanisms to help build re-usable solutions that can protect investments in engineering and development time, skills and that work across on premises and Cloud environment. In 2021 my most viewed blog post talked about how you can build and deploy containerised applications, anywhere (Cloud, your data centre, other Clouds) and on anything (Intel and Arm).

  • Contributing to the Apache Airflow project - Part One

    Calendar Feb 10, 2022 |  Stopwatch  18 minute read

    Contributing to Apache Airflow Introduction In this series of posts, I am going to share what I learn as embark on my first upstream contribution to the Apache Airflow project. The purpose is to show you how typical open source projects like Apache Airflow work, how you engage with the community to orchestrate change and hopefully inspire more people to contribute to this open source project. I will post regular updates as a series of posts, as the journey unfolds.

  • Running my dev.to blog using Hugo on Netlify

    Calendar Jan 7, 2022 |  Stopwatch  6 minute read

    Running my dev.to blog using Hugo on Netlify I am a big fan of dev.to, and the work that the team do to foster a great community of builders is something that keeps me there. I have always maintained another blog (running on Netlify, which is also super awesome), kind of like a mirror. Up until last year, I was able to publish to dev.to and it would take care of publishing to that mirror.

  • Setting up MWAA to use a KMS key

    Calendar Dec 14, 2021 |  Stopwatch  6 minute read

    Introduction In a previous post, I shared how you can using AWS CDK to provision your Apache Airflow environments using the Managed Workflows for Apache Airflow service (MWAA). I was contacted this week by Michael Grabenstein, who flagged an issue with the code in that post. The post used code that configured a kms key for the MWAA environment, but when trying to deploy the app it would fail with the following error:

  • Integrating Amazon Timestream in your Amazon Managed Workflows for Apache Airflow v2.x

    Calendar Sep 23, 2021 |  Stopwatch  28 minute read

    Integrating with Amazon Timestream in your Apache Airflow DAGs Amazon Timestream is a fast, scalable, and serverless time series database service perfect for use cases that generate huge amounts of events per day, optimised to make it faster and more cost effective that using relational databases. I have been playing around with Amazon Timestream to prepare for a talk I am doing with some colleagues, and wanted to see how I could integrate it with other AWS services in the context of leveraging some of the key capabilities of Amazon Timestream.

  • Reading and writing data across different AWS accounts with Amazon Managed Workflows for Apache Airflow v2.x

    Calendar Sep 7, 2021 |  Stopwatch  13 minute read

    Reading and writing data across different AWS accounts in you Apache Airflow DAGs As regular readers will know, I sometimes lurk in the Apache Airflow slack channel to see what is going on. If you are new to Apache Airflow, or want to get a deeper understanding then I highly recommend spending some time here. The community is super welcoming and eager to help new participants. It was during a recent session I came across an interesting problem that one of the builders was having, which was how to access (read/write) data in an S3 bucket which was in a different account to the one hosting Amazon Managed Workflows for Apache Airflow (MWAA).

  • Working with parameters and variables in Amazon Managed Workflows for Apache Airflow

    Calendar Jul 27, 2021 |  Stopwatch  36 minute read

    Maximising the re-use of your DAGs in MWAA During some recently conversations with customers, one of the topics that they were interested in was how to create re-usable, parameterised Apache Airflow workflows (DAGs) that could be executed dynamically through the use variables and/or parameters (either submitted via the UI or the command line). This makes a lot of sense, as you may find that you repeat similar tasks in your workflows, and so this approach allows you to maximise the re-use of that work.

  • Creating a multi architecture CI/CD solution with Amazon ECS and ECS Anywhere

    Calendar Jul 16, 2021 |  Stopwatch  39 minute read

    Please let me know how I can improve posts such as this one, by completing this very short survey. $25 AWS credits will be provided for the first 20 completed - take the survey Organisations are moving their workloads to the cloud as quickly as they can. While most applications can be easily migrated to the cloud, some applications need to remain on-premises due to low-latency or data sovereignty requirements.

  • Working with Amazon EKS and Amazon Managed Workflows for Apache Airflow v2.x

    Calendar Jun 10, 2021 |  Stopwatch  11 minute read

    Introduction The Apache Airflow slack channel is a vibrant community of open source builders that is a great source of feedback, knowledge and answers to problems and use cases you might have when trying to do stuff with Apache Airflow. This week I picked up on someone seeing errors with Amazon EKS, and so I thought what better time to try out the new Apache Airflow 2.x version that was recently launched in Amazon Managed Workflows for Apache Airflow (MWAA).

  • Working with the RedshiftToS3Transfer operator and Amazon Managed Workflows for Apache Airflow

    Calendar May 15, 2021 |  Stopwatch  18 minute read

    Introduction Inspired by a recent conversation within the Apache Airflow open source slack community, I decided to channel the inner terrier within me to tackle this particular issue, around getting an Apache Airflow operator (the protagonist for this post) to work. I found the perfect catalyst in the way of the original launch post of Amazon Managed Workflows for Apache Airflow (MWAA). As is often the way, diving into that post (creating a workflow to take some source files, transform them and then move them into Amazon Redshift) led me down some unexpected paths to here, this post.

  • Using AWS CDK to deploy your Amazon Managed Workflows for Apache Airflow environment

    Calendar Apr 28, 2021 |  Stopwatch  11 minute read

    update I am grateful to Michael Grabenstein for spotting some mistakes in the original post/code. I hope these have now been rectified in this post. Using AWS CDK to deploy your Amazon Managed Workflows for Apache Airflow environment What better way to celebrate CDK Day than to return to a previous blog where I wrote about automating the installation and configuration of Amazon Managed Workflows for Apache Airflow (MWAA), and take a look at doing the same thing but this time using AWS CDK.

  • Automating your ELT Workflows with Managed Workflows for Apache Airflow - Part Two

    Calendar Apr 21, 2021 |  Stopwatch  17 minute read

    Part Two - Automating Amazon EMR In Part One, we automated an example ELT workflow on Amazon Athena using Apache Airflow. In this post, Part Two, we will do the same thing but automate the same example ELT workflow using Amazon EMR. Make sure you recap the setup from Part One. All the code so you can reproduce this yourself can be found in the GitHub repository here. Automating Amazon EMR

  • Automating your ELT Workflows with Managed Workflows for Apache Airflow - Part One

    Calendar Apr 21, 2021 |  Stopwatch  18 minute read

    update: I have changed the post to use standard Apache Airflow variables rather than using AWS Secrets Manager. Part One - Automating Amazon Athena As part of an upcoming DevDay event, I have been working on how you can use Apache Airflow to help automate your Extract, Load and Transform (ELT) Workflows. Amazon Athena and Amazon EMR are two AWS services that help customers who have existing SQL skills/expertise and are looking at tools such as Presto or Apache Hive when undertaking those transformations.

  • Monitoring and logging with Amazon Managed Workflows for Apache Airflow

    Calendar Feb 9, 2021 |  Stopwatch  12 minute read

    Part of a series of posts to support an up-coming online event, the Innovate AI/ML on February 24th, from 9:00am GMT - you can sign up here Part 1 - Installation and configuration of Managed Workflows for Apache Airflow Part 2 - Working with Permissions Part 3 - Accessing Amazon Managed Workflows for Apache Airflow Part 4 - Interacting with Amazon Managed Workflows for Apache Airflow via the command line Part 5 - A simple CI/CD system for your development workflow Part 6 - Monitoring and logging <- this post Part 7 - Automating a simple AI/ML pipeline with Apache Airflow In this post I will be covering Part 6, where to find logs to help you understand and troubleshoot your Apache Airflow workflows, and how you can monitor your Apache Airflow environments.

  • A simple CI/CD system for your Amazon Managed Workflows for Apache Airflow development workflow

    Calendar Feb 3, 2021 |  Stopwatch  16 minute read

    updated Feb 19th Part of a series of posts to support an up-coming online event, the Innovate AI/ML on February 24th, from 9:00am GMT - you can sign up here Part 1 - Installation and configuration of Managed Workflows for Apache Airflow Part 2 - Working with Permissions Part 3 - Accessing Amazon Managed Workflows for Apache Airflow Part 4 - Interacting with Amazon Managed Workflows for Apache Airflow via the command line Part 5 - A simple CI/CD system for your development workflow <- this post Part 6 - Monitoring and logging Part 7 - Automating a simple AI/ML pipeline with Apache Airflow In this post I will be covering Part 5, how you can setup a very simple CI/CD setup to enable faster development of your Apache Airflow DAGs.

  • Interacting with Amazon Managed Workflows for Apache Airflow via the command line

    Calendar Feb 1, 2021 |  Stopwatch  12 minute read

    Part of a series of posts to support an up-coming online event, the Innovate AI/ML on February 24th, from 9:00am GMT - you can sign up here Part 1 - Installation and configuration of Managed Workflows for Apache Airflow Part 2 - Working with Permissions Part 3 - Accessing Amazon Managed Workflows for Apache Airflow environments Part 4 - Interacting with Amazon Managed Workflows for Apache Airflow via the command line < this post Part 5 - A simple CI/CD system for your development workflow Part 6 - Monitoring and logging Part 7 - Automating a simple AI/ML pipeline with Apache Airflow In this post I will be covering Part 4, how you can interact and access the Apache Airflow via the command line.

  • Accessing your Amazon Managed Workflows for Apache Airflow environments

    Calendar Jan 28, 2021 |  Stopwatch  8 minute read

    Part of a series of posts to support an up-coming online event, the Innovate AI/ML on February 24th, from 9:00am GMT - you can sign up here Part 1 - Installation and configuration of Managed Workflows for Apache Airflow Part 2 - Working with Permissions Part 3 - Accessing Amazon Managed Workflows for Apache Airflow environments < this post Part 4 - Interacting with Amazon Managed Workflows for Apache Airflow via the command line Part 5 - A simple CI/CD system for your development workflow Part 6 - Monitoring and logging Part 7 - Automating a simple AI/ML pipeline with Apache Airflow In this post I will be covering Part 3, how you can interact and access the Apache Airflow environments.

  • Working with permissions in Amazon Managed Workflows for Apache Airflow

    Calendar Jan 27, 2021 |  Stopwatch  10 minute read

    Part of a series of posts to support an up-coming online event, the Innovate AI/ML on February 24th, from 9:00am GMT - you can sign up here Part 1 - Installation and configuration of Managed Workflows for Apache Airflow Part 2 - Working with Permissions <- this post Part 3 - Accessing Amazon Managed Workflows for Apache Airflow environments Part 4 - Interacting with Amazon Managed Workflows for Apache Airflow via the command line Part 5 - A simple CI/CD system for your development workflow Part 6 - Monitoring and logging Part 7 - Automating a simple AI/ML pipeline with Apache Airflow In this post I will be covering Part 2, how to ensure that you control access to Apache Airflow following best practices such as default no access/least privilege.

  • Automating the installation and configuration of Amazon Managed Workflows for Apache Airflow

    Calendar Jan 26, 2021 |  Stopwatch  15 minute read

    updated, August 25th Thanks to Philip T for spotting a typo in the cloudformation code below - it is ok in the GitHub repo, but I have fixed it now below. Part of a series of posts to support an up-coming online event, the Innovate AI/ML on February 24th, from 9:00am GMT - you can sign up here Part 1 - Installation and configuration of Managed Workflows for Apache Airflow <- this post Part 2 - Working with Permissions Part 3 - Accessing Amazon Managed Workflows for Apache Airflow environments Part 4 - Interacting with Amazon Managed Workflows for Apache Airflow via the command line Part 5 - A simple CI/CD system for your development workflow Part 6 - Monitoring and logging Part 7 - Automating a simple AI/ML pipeline with Apache Airflow In this post I will be covering Part 1, automating the installation and configuration of Managed Workflows for Apache Airflow (MWAA).

  • TIL: Testing an Amazon Cloudwatch alarm

    Calendar Jan 7, 2021 |  Stopwatch  2 minute read

    Today I was setting up an application load balancer that sits in front of a test application I have put together. Setting this up was super easy, and very quickly I had my domain pointing to the alias and serving requests. As part of the setup, I wanted to monitor the application load balancer to let me know when requests were failing to the downstream application (anything other than an HTTP 200) and so I set this up super easily in Amazon Cloudwatch.

  • Amazon Aurora - setting up and configuration, four ways

    Calendar Oct 15, 2020 |  Stopwatch  8 minute read

    In this post I want to share four different approaches to installing and configuring your Amazon Aurora database clusters. Everything in this post is covered in detail in the embedded video, but I wanted to share some additional information that I did not include in the video that was easier done in this blog. {% youtube wZfh9PurE9E %} Why four ways? The approach in the video was to look at the journey you might take when learning a new technology and then how you move to productise that technology.

  • Long running data import jobs with AWS Session Manager

    Calendar Sep 10, 2020 |  Stopwatch  4 minute read

    Yesterday I was looking to import the TPC-H dataset (some 600 or so million rows) into Amazon Aurora from a workstation that I connect to using AWS Session Manager. AWS Session Manager is a great way to simplify your life by allowing you to connect to a machine via the AWS console and not worry about having to manage ssh keys or remembering to lock down external public access from the net.

  • Building a culture of security in open source software development

    Calendar Jul 15, 2020 |  Stopwatch  9 minute read

    Updated on Jan 18th to remove broken link to report According to a number of recent studies, the use and adoption of open source software continues to rise. From studies such as the State of Enterprise Open Source by Red Hat (in which nearly 70% of respondents stated that open source software is either extremely or very important) or TideLift’s April 2019 survey report (that found more than 90% of professional developers use open source in building their applications) it is clear that developers from startups to highly regulated enterprises have embraced open source solutions.

  • Automating AWS SSO and G-Suite synchronisation with SSO Sync

    Calendar Jun 3, 2020 |  Stopwatch  5 minute read

    update-July 28th The ssosync tool has had a lot of interest and the community has updated the tool. This means that you should refer to the project home page https://github.com/awslabs/ssosync and check out the README.md for what changes you might need to make to get this tool working. Next level ssosync In a previous post, I talked about setting up AWS Single Sign On (AWS SSO) with G-Suite, and then using an open source project called ssosync to syncronise users and groups from G-Suite into AWS SSO.

  • Setting up G-Suite, AWS SSO and ssosync

    Calendar May 27, 2020 |  Stopwatch  17 minute read

    update-July 28th The ssosync tool has had a lot of interest and the community has updated the tool. This means that you should refer to the project home page https://github.com/awslabs/ssosync and check out the README.md for what changes you might need to make to get this tool working. Enabling AWS SSO with Google G-Suite Many customers have existing directory technologies where they manage their users, and then use this central identity store as a way to simplify the way they authenticate and provide access to applications and other resources.

  • Making the most of mentoring

    Calendar May 8, 2020 |  Stopwatch  5 minute read

    Some recent experiences mentoring has provided the motivation for this piece. It is not intended to be right or wrong, but just my personal opinion and experience and I hope it is read that way. I have put this together to share what I think are the critical things that make a mentoring relationship work for both the mentor and mentee. So with that out of the way, I invite you to read on…

  • Mentoring and reverse mentoring

    Calendar Dec 29, 2019 |  Stopwatch  4 minute read

    As I reflect on 2019, one of the common themes whilst engaging with builders at the start of their career, has been how do those of us who have deep experience working in the IT industry and technology help bring those who are just starting out? Some common themes when talking that have come up include; How do I get started on Cloud or AWS? What tools and languages should I learn?

  • reInvent 2019 workshop list

    Calendar Dec 9, 2019 |  Stopwatch  1 minute read

    So here is a list scraped from Twitter and following various other folk, of just a small taster of the workshops that ran during reInvent. As I find more I will update, and feel free to add yours in the comments (oh, and let me know if any of these are dead links) Serverless - https://github.com/aws-samples/aws-serverless-workshop-innovator-island/ Serverless image process workshop - https://image-processing.serverlessworkshops.io/ Amplify preductions workshop - https://github.com/mlabieniec/IonicPredictions Full stack serverless Amplify lab - https://github.

  • Make your business more resilient in the digital age

    Calendar Oct 17, 2019 |  Stopwatch  1 minute read

    Very humbled to write a guest post on Adrian Hornsby excellent blog where he provides guidance to help customers build resilient architecture and champions operational excellence. In this post I talk about what you need to think about to build a more resilient business fit for the digital age. Here is the link: https://medium.com/@adhorn/make-your-business-more-resilient-in-the-digital-age-888da3f5deaf

  • Innovate Machine Learning and AI - learn how to kick start your journey

    Calendar Oct 4, 2019 |  Stopwatch  1 minute read

    On October 17th we have a free, online event covering several tracks on Machine Learning. Whether you are a complete beginner or seasoned data scientist, we have gentle introductions to deep dives. What I am most excited about however, is that we will have an AWS DeepRacer racing challenge. You will learn how to create your first reinforcement learning model that will race a virtual car, with prizes for the fastest times.