Max Kelsen is only one of three Australian companies to attain AWS’s machine learning competency status, and was named AWS’s 2020 Partner of the Year for Data, Analytics and Machine Learning. We are constantly in the thick of upskilling our team by attending conferences that fuel our thirst for knowledge and keeping our curiosity alive.

AWS re:Invent is a free 3-week virtual conference held by AWS as an opportunity for AWS experts to lead sessions and share the latest news, technologies, and trends within the space. The Max Kelsen team are attending for the entirety of the conference and sharing with you a weekly summary of the key sessions that touch upon the components and sub-services that we use, what we are excited about and why these are important.

#1 — Container Support in Lambda

Session Introduction:

With AWS Lambda, you upload your code and run it without thinking about servers. Many customers enjoy the way this works, but if you’ve invested in container tooling for your development workflows, it’s not easy to use the same approach to build applications using Lambda. To help you with that, you can now package and deploy Lambda functions as container images of up to 10 GB in size. Access the full recording here.

Max Kelsen Summary:

People are already beginning to leverage this to support CPU-based inference workloads — as seen in Matt Coulter from Liberty Mutual’s tweet below:

This represents a big step forward for deploying highly customised python environments, whilst allowing developers to leverage tooling such as CDK to get up and running quickly — the ability to stand up a customised python container behind an API gateway in a few lines is a very exciting feat.

#2 — GP3 EBS Volumes

Session Introduction:

Amazon Elastic Block Store (EBS) is an easy-to-use, high-performance block storage service designed for use with Amazon EC2 instances for both throughput and transaction-intensive workloads of all sizes. Gp3 is a new type of SSD EBS volume that lets users provision performance independent of storage capacity, and offers a 20% lower price than existing gp2 volume types. Access the full recording here.

Max Kelsen Summary:

Our team was really excited to see the decoupling of volume size and performance with the gp3, which allowed for cost optimisation and performance right-sizings without the need to transition to using the more expensive io2 volume types. A number of our own workloads as well as our customers’ workloads do not generally require such large storage volumes, but they do however, need to be highly performant. This represents a great opportunity to drive greater cost efficiencies.

#3 — AWS Amplify Admin Console

Session Introduction:

AWS Amplify announces a new Admin UI to configure an application backend, and manage app users and content outside the AWS console. This new feature makes it easier to use AWS services and accelerates the development and management of full-stack web and mobile apps. Access the full recording here.

Max Kelsen Summary:

Our team here at Max Kelsen heavily use AWS Amplify to develop mobile and web applications, especially during the ‘Mythbuster’ phase, or more commonly known as ‘Proof of Concept’. We find that it allows us to quickly get up and running, and iterate more rapidly. Whilst this is not a huge step change functionality, the continued improvement of the developer experience when working in the AWS Amplify ecosystem is great to see and we trust that there is more yet to come — a space we will be keeping a close eye on.

#4 — Amazon EKS Announcements — Distro, Console Improvements, Managed Spot Instances

Session Introduction:

Amazon Elastic Kubernetes Service (EKS) has been very popular; removing the burden of managing Kubernetes while our customers glean the benefits. EKS Distro is a distribution of the same version of Kubernetes deployed by Amazon EKS, which you can use to manually create your own Kubernetes clusters anywhere you choose. EKS Distro provides the installable builds and code of open source Kubernetes used by Amazon EKS, including the dependencies and AWS-maintained patches. Using a choice of cluster creation and management tooling, you can create EKS Distro clusters in AWS on Amazon Elastic Compute Cloud (EC2), in other clouds, and on your on-premises hardware. Access the full recording here.

Max Kelsen Summary:

At Max Kelsen, we are heavily invested in Kubernetes, especially for all of our machine learning workloads. Throughout some of the week one sessions, it was great to see AWS continue to expand their service offering in this space. The updates to the EKS console offer a greatly improved developer experience, helping operations teams better understand the configuration of their cluster and the workloads deployed. This brings the EKS console much closer to the GKE console experience.

We will be very interested to see the implementations and uses of the EKS Distro, as we continue to think one step ahead as to how we can utilise this for our on-premise research environments, both across customer environments and within CI pipelines. As we already make heavy use of spot instances for our workloads on Amazon EKS, the introduction of the managed spot instances greatly lowers the overhead of provisioning and managing spot instances within our nodegroups. Integration within the new console also greatly improves user experience and helps provide clear visibility over the configuration of the nodes and workloads in our clusters!

Also with their easy to read quick start instructions guide, this considerably lowers the barrier for anyone wanting to try Amazon EKS for the first time, so with this, we highly encourage everyone to check this out and realise its potential. Can you tell we were all personally very pleased with the updates on Amazon EKS?

#5 — AWS S3 — Strong Read After Write Consistency

Session Introduction:

When S3 was launched back in 2006, discussions were made of its virtually unlimited capacity , the fact that it was designed to provide 99.99% availability, and that it offered durable storage, with data transparently stored in multiple locations. Since that launch, AWS customers have used S3 in an amazing diverse set of ways: backup and restore, data archiving, enterprise applications, web sites, big data, and (at last count) over 10,000 data lakes.

To help customers run big data workloads in the cloud, Amazon EMR built EMRFS Consistent View and open source Hadoop developers built S3Guard, which provided a layer of strong consistency for these applications. Access the full recording here.

Max Kelsen Summary:

AWS S3 forms the backbone of nearly everything we do on AWS here at Max Kelsen, from data lakes, to application development, and to machine learning. Many of our data processing architectures utilise S3 to exchange data between different components, and these changes to the consistency model greatly simplify the engineering and design consideration requirements for our teams. At the scale S3 operates at, this is a very impressive feat of engineering!

#6 — SageMaker — Feature Store & Pipelines

Max Kelsen Summary:

Lastly, one of the key highlights that excited us at the first week of AWS re:Invent is the attention Amazon SageMaker Feature Stores have been getting over the past number of months. We’ve been closely following the Feast project and it is great to see AWS make similar functionality to Machine Learning engineers inside the SageMaker Studio ecosystem.

Feature Stores are instrumental in scaling Machine Learning engineering teams, especially if they are developing many different models from the same or similar datasets. Amazon SageMaker Pipelines and the various ways to orchestrate and instrument them have been a hot topic as of late. The introduction of the SageMaker Pipelines SDK and integration within the SageMaker Studio ecosystem represents a great low-overhead way for developers to begin productionising their work into repeatable & shareable end to end pipelines.

Within Max Kelsen’s Machine Learning development lifecycle and production systems, Pipelines are the core components of how we execute our work.

And that’s it for week one! Please stay tuned as we provide weekly updates for the next couple of weeks of the AWS re:Invent conference, and do not be afraid to reach out to us at hello@maxkelsen.com if you would like to start a conversation about any of the above topics and updates. See you next week!

We are an Artificial Intelligence and Machine Learning consultancy that delivers competitive advantage for government and enterprise. https://maxkelsen.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store