Blog de Joe Freeman

Linux package repositories often only hold the latest LTS versions.  You have to look elsewhere if you need something like Java 8.  Look to Amazon Coretto if you want specific versions of Java installed on your Linux instance.  Amazon maintains Corretto distributions going back to Java 8 when checked at 2023 06. Debian, Ubuntu, Kali, etc. Users can add the Corretto repository to their instances and then install specific OpenJDK versions using standard apt . Installation Partially derived from https://docs.aws.amazon.com/corretto/latest/corretto-8-ug/generic-linux-install.html The Corretto install instructions assume you have add-apt-repository  installed.  So install add-apt-repository sudo apt update sudo apt install software - properties - common Add the Corretto repository wget - O - https: // apt.corretto.aws / corretto.key | sudo apt - key add - sudo add-apt - repository 'deb https://apt.corretto.aws stable main' Install Java 8 Corretto sudo apt - get up...

It is surprising how a vendor's internal culture or local management culture biases your opinion of that company.  I worked with two different cloud vendors for 5 years each over a 10-year period. Both were in it to make money but one was significantly easier and more productive to work with. Transcript YouTube Video of Two Cloud Vendors   A slightly edited YouTube AI-generated transcript of the video I've worked with two different Cloud vendors while working for four different companies and two different Cloud vendors. Let me rephrase that.  I've worked at four different outfits working with two different Cloud vendors. The two Cloud vendors had completely different operating models. I worked with Amazon AWS and Microsoft Azure I've never worked with the Google team but they always weird me out because they cancel stuff. This is across 10 years of interactions, five years in Azure two different times and five years in AWS two different times four different teams. Th...

It is hard for many to remember when Amazon and AWS weren't really enterprise-ready.  AWS wasn't really AWS in the beginning. It was really a scalable data center. The company restricted its employees to locations where the Amazon store  collected sales tax  which really ended in 2016.  Amazon pivoted to PaaS with the introduction of  S3 and EC2 in 2012 .  This is when AWS started moving from being a data center to the internet's operating system   Amazon Called and I had no idea what they were thinking An Amazon cloud engineer called me in the early 2010s wanting to know if I was interested in a Field Engineering position with AWS. I had worked for a few vendors by this point and was pretty interested.  There were only two  field/cloud/customer engineers  covering the entire USA , and possibly the planet. I don't remember if they had anything outside the USA.   Both engineers worked in Seattle.  We discussed how they work...

A zero-patch environment is one in which we redeploy our applications with updated dependencies as part of our normal Continuous Deployment process.  This is part of You Build It - You Own It We need to create a repeatable cadence for bringing in Operating systems, containers, application patches, and binary updates.  We can reduce some of our work by pushing as much of our workload as possible onto PaaS, serverless, and cloud services. Three Month Cycle Example This is a prototypical 3-month update cycle where an application is redeployed with security and bug fix updates at least every quarter. Click to enlarge The Image team identifies required operating and container image updates.  Application teams identify CVE library updates via automated scanners. The application teams start integrating library updates.  T...

In-place patching is an anti-pattern for cloud-native applications that originates from the data center with low levels of automation for infrastructure and application management. The shift-left movement pushes ownership to the development teams that are best served by applying all changes via their CI/CD pipelines. This goes for infrastructure, containers, serverless, code, et al. Video Slides used in video The talk did not cover zero-day emergency patching.  Sometimes a mass update must be applied to all like systems.  In that case, patching may be supported with a full redeployment as soon as possible after that. Created 2022 03

AWS and Microsoft have both been busy demonstrating how their clouds are ideal for fleets of IoT devices.  Both introduced their own sample hardware.  The AWS entry is an M5Stack Core2 ESP32 device that comes with I/O, ports, and a certificate store.  Amazon has a couple relatively straightforward examples including a simple "blinking light" program that reacts to cloud messages with lights and that sends regular messages to cloud topics for processing. I created the video down below to give a quick overview of the steps that make up that sample. AWS Branded IoT Device This is the device.  It is an AWS-enhanced version of the standard M5Stack Core2 component. The Demo Application The sample program operates on two paths. It sends a regular message to AWS via MQTT.  The sample tells you how to view that message in the AWS portal. The sample then shows you how to use the IOT test client to generate test messages that are sent to the EduKit IoT dev...

PaaS and SaaS persistence services store your data in their systems, often in their accounts or subscriptions.  The service provider protects the system and its associated storage. We need to determine our appetite for risk when deciding what additional work must be done to secure the externally hosted data. Risks Data at rest must be protected with a multi-layered approach. Identify the attacks that you wish to prevent in order to determine how much protection you want. The list below is just a sample. disk re-use  is hardware or technology-related. It can be mitigated without any application or user experience changes.  Vendor -related access issues exist because a 3rd party is hosting the data. This includes vendor staff access and the ability to remove or render unusable your data in their ho Control Plane refers to dashboards or admin screens that are vendor-provided.  Many have preview functions that let you validate the data.  The built-in user permissio...

Cloud platforms like Azure, and AWS, and GCP provide new-user free tier accounts that help people get started in the cloud. They have no low-cost or capped offerings after that short period. They all expect you to move to uncapped fully metered pay-as-you-go services after that. This makes it hard for data scientists, developers and architects is how to stay current and innovate without running the risk of financial catastrophe.  I don't want to go bankrupt while experimenting We're talking about relatively small non-production use cases.  The dollar limits on these could be capped to relatively small amounts. I recently accidentally provisioned a dedicated Azure Event Hubs cluster that burned through my fixed $150 credit in 1 day.  The account ran up a $400 bill before Azure caught up and shut down the subscription.  The hard cap meant I was dead to Azure for a month but not dead to my spouse for spending our mortgage payment. After the time-limited free tier T...

Many companies end up creating their own cloud management  control planes on top of their cloud provider's management APIs.  These homegrown  management systems provide a central location for provisioning cloud services, for configuring SaaS offerings. They also interact with corporate compliance and control systems like artifact inventories and data catalogs. What drives companies to this effort and expense? Video Walkthrough Self Created Cloud Control Plane Companies create their own API and Web UI-driven control planes as proxies for their cloud provider, their SaaS provider, and any internal providers that must be communicated with as part of infrastructure provisioning and deployments.   Manages the company's cloud resources Categorize those resources for compliance and budget Coordinates unified deployments and configurations across other control ...

Two Parts ML for the masses Covid Intake ML demonstration from 2021 Snowflake Summit Accelerating the ML revolution Sagemaker Autopilot is moving ML from custom programming to a commodity service The end of the need for Custom ML platforms ML for the masses with less investment and startup costs Easy access to open data Partner data sharing with manageable risk Snowflake and AWS Sagemaker Autopilot Snowflake and AWS provided a low code demonstration that merged public health data with intake surveys to create a set of Machine model-based services that could help prioritize covid intake patients based on past patterns.   The truly interesting part of the demonstration is that they Restructured and merged data sets inside Snowflake with simple SQL Create a full ML environment Created and trained a model Deploy...

Streaming Data into a Lake is a powerful tool, modern, approach that replaces traditional ETL.  There are use cases where streaming  can make things difficult for business systems or direct data users. Your data ingestion tier may have to support both data streaming  and bulk processing. Speaker's Notes Notes to be added Notes to be added Notes to be added Notes to be added Notes to be added Notes to be added Notes to be added

Streaming data in Data Lakes is one of those ideas that doesn't work exactly as you expect it to.   Your pretty data streams are packed up and sent into your data lake by old school batch process.  The primary reason for this is the way that data is stored in the cloud in non-append-able file/blob storage. Data files cannot have data appended to them . New files/objects must continually be generated for any kind of producer performance.  Streams are Micro Batched Groups of records are written as a single operation to cloud storage.  Data lake writers must batch up  streaming data in order to write it to cloud storage. Writing individual records is inefficient and may not even be possible depending on the run rate of the message stream. Files in the Data Lake in the same dataset may be different sizes based on the batch writing trigger definitions. ...

What does it mean to move from a Relational Database style Schema on Read to  Schema on Write ? Schema on Write  is used to stage data in a consumer friendly form.  It can also be used in poor-join-performance environments to restructure and stage data in consumer read  format. It is pretty much mandatory for Document Databases.  Ingestion stores data in its original format for compliance, audit or other purposes.  This copy may be called True Source. Format Standardization converts the raw information into and agreed on standard format.  Examples include  Data Tables  in a lake or documents in a document store.  This is purely a mechanical conversion. Consumption Model are built from raw data, reference data and applies view and business rules creating a consumer ready dataset...

Data Lakes often have the some of the same performance and security decisions as past year's data warehouses.   Teams need to decide if the data in a lake is stored in producer formats or consumer formats or a combination of the two.  Storage is essentially unlimited which means we may choose to store the data in multiple consumer oriented fashion. Compute is essentially unlimited. We may decide to apply view style restrictions and access controls at read time.              Video Speaker's Notes This discussion is really only about tabular style data stored in cloud blob/object stores.  See data lakes for squares.  Record oriented data can be built up from fil...