A simplified version of Kubernetes Network for developing on Windows and WSL and Docker Desktop

Pictures always help. We can use this one to put together a basic description of how the Windows, WSL2 Linux distributions, and Docker on WSL and Kubernetes talk to each other over localhost. We’ll cover why you got to do things like port-forwarding or proxy with a Kubernetes network. There is asymmetric behavior between the different components. I'm sure somebody knows the magical explanation. 


Click to enlarge

Video Walkthrough

Magical localhost networking with Docker WSL2 Windows and Linux

Heavily Edited Video Transcript

You have a Windows host with its ethernet adapter. Each Linux WSL2 instance has an eth0 network adapter. The Windows host and WSL instances are attached to a private network on the Ethernet Switch (WSL) Then you have the localhost adapter. WSL makes it look like localhost is visible to all WSL instances and to the Windows host.  A port on WSL2 127.0.0.1 is also available on any other WSL2 instance and from the Windows host all on 127.0.0.1.  

This sharing is actually interesting right because Kubernetes and Docker each have their own private node pool and Docker networks are isolated from the public Ethernet and private WSL network. If I want to expose a Kubernetes service like the Kubernetes Dashboard or the Kubernetes API Gateway or some deployed Kubernetes Service to WSL or to the Windows 10/11 host then I actually need to run a forwarder like
  • Acube control proxy generally for the API server or 
  • A port forwarder for TCP or HTTP 

By default, the kubectl proxy in this environment actually puts itself on localhost 127.0.0.1 in the system where the kubectl command ran. Those proxies become visible across that localhost network. If you bring it up as a proxy/forwarder on any of the Linux machines, then it is visible on the Windows 10/11 host on its localhost.

If I need to see the Kubernetes Dashboard or some deployed private Kubernetes Service, then I do a kubectl port-forward. If I want to talk to the Kubernetes API Server then I use the Kubernetes proxy, kubectl proxy, or forwarder. 

There is some weirdness about the way this thing works. Windows and Linux VMs can see ports forwarded to localhost in either Windows or Linux VMs.  Only the Windows machine can see the forwarded ports on localhost if the forwarder/proxy ran on the Windows machine itself. You have to decide where you want to run the proxies on Windows 10/11, or a Linux instance.  You only have to run it once if you put it in the right place because it drops it on the localhost adapters and everyone can see it.  You can see this in the diagram.

kubectl proxy can only be run once on one machine/VM because the forwarding happens on localhost. Port 8001 for the two different Linux instances or Windows hosts are really the same port. The big deal is there can only be one because all these things are being exposed on the localhost address. So if you bring up one of these kubectl proxy or kubectl port-forwarder then you only need one of them but if you put it on Windows 10/11 host then none of these Linux machines are going to see the proxy There's an asymmetrical set of behavior here between the Linux host and idem.

<video did walkthrough with 6 terminal windows that demonstrated where routes work and they don’t.  Python servers were run on each of the two Linux VMs and Windows host>

Video Walkthrough

WSL Binds Posts to localhost

A proxy or forwarder on WSL can be seen from Windows.  A proxy or forwarder on Windows cannot be seen from WSL. WSL services are exposed on "localhost" which is why they can't be seen anywhere but "localhost" machines.  In this case all the 
  • If you deploy a server in your Linux distribution, you can access it from a Windows App or Browser using localhost.
  • If you want to access a service running on Windows from a Linux distribution (ie Ubuntu), then you need to use the IP address of your host machine. This drives the traffic off of the internal WSL vnet onto the host ethernet adapter

Port-Forward vs Proxy

Kubernetes provides two types of proxies.
  1. kubectl proxy routes HTTP/HTTPS traffic through the apiserver
    1. Creates a local web server that proxies data through the API server
  2. kubectl port-forward establishes TCPconnection betwen your localhost and a port on a pod

Communication

Notes to be cleaned up.

  1. container to container
    1. Network namespace.  Every pood has its network namespace, IP and port space
    2. Containers within a pod communicate through localhost and share the same IP and ports
  2. pod to pod
    1. Pods all have IP in the cluster CIDR range.  pod-to-pod communication uses these IPs
  3. pod to service
    1. Service is given a virtual IP backed by one or more pods
    2. Pods can change address when pod is rehydrated.  Pods are transient
    3. Service gets an IP from the CIDR range that may be different than the pod CIDR range
  4. external ingress
    1. A proxy needs to be set up somewhere.  Kubernetes is only a single node in the WSL2 environment.  You would need an external load balancer to LB across nodes if the cluster has more than one host.
  5. external egress
    1. This just works

Services URLs

The Kubernetes API server can route to all deployed service endpoints via namespace routes. You can also/instead use port-forwarding or another proxy to directly route to a Service endpoint.

This URL shows how to access a Deployed Service's endpoint by routing through the a proxy that is forwarding to the API server. Just run  kubectl proxy.

http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

Service Load Balancers in Windows / WSL

External cross-node LB is not needed because we only have a single Kubernetes node when running Docker Desktop or any other docker in a default configuration. The single Service entrypoint is all you need.

References


Comments

Post a Comment

Popular posts from this blog

Installing the RNDIS driver on Windows 11 to use USB Raspberry Pi as network attached

Image
I do a lot of my development and configuration via ssh into my Raspberry Pi Zero over the RNDIS connection. Some models of the Raspberry PIs can be configured with  gadget drivers  that let the Raspberry pi emulate different devices when plugged into computers via USB. My favorite gadget  is the network profile that makes a Raspberry Pi look like an RNDIS-attached network device.  All types of network services travel over an RNDIS device without knowing it is a USB hardware connection. A Raspberry Pi shows up as a Remote NDIS (RNDIS) device when you plug the Pi into a PC or Mac via a USB cable. The gadget  in the Windows Device Manager picture shows this RNDIS Gadget connectivity between a Windows machine and a Raspberry Pi. The Problem Windows 11 and Windows 10 no longer auto-installs the RNDIS driver that makes magic happen. Windows recognizes that the Raspberry Pi is some type of generic USB COM device. Manually running W indows Update  or  Upd...
Read more

Understanding your WSL2 RAM and swap - Changing the default 50%-25%

Image
The Windows Subsystem for Linux operates as a virtual machine that can dynamically grow the amount of RAM to a maximum set at startup time.  Microsoft sets a default maximum RAM available to 50% of the physical memory and a swap-space that is 1/4 of the maximum WSL RAM.  You can scale those numbers up or down to allocate more or less RAM to the Linux instance. The first drawing shows the default WSL memory and swap space sizing. The images below show a developer machine that is running a dev environment in WSL2 and Docker Desktop.  Docker Desktop has two of its own WSL modules that need to be accounted for.  You can see that the memory would actually be oversubscribed, 3 x 50% if every VM used its maximum memory.  The actual amount of memory used is significantly smaller allowing every piece to fit. Click to Enlarge The second drawing shows the memory allocation on my 64GB laptop. WSL Linux defaul...
Read more

Almost PaaS Document Parsing with Tika and AWS Elastic Beanstalk

Image
The Apache Tika project provides a  library capable of parsing and extracting data and meta data from over 1000 file types.  Tika is available as a single jar file that can be included inside applications or as a deployable jar file that runs Tika as a standalone service. This blog describes deploying the Tika jar as an auto-scale service in Amazon AWS Elastic Beanstalk.  I selected Elastic Beanstalk because it supports jar based deployments without any real Infrastructure configuration. Elastic Beanstalk auto-scale should take care of scaling up and down for for the number of requests you get. Tika parses documents and extracts their text completely in memory. Tika was deployed for this blog using EC2 t2.micro instances available in the AWS free tier. t2.micro VMs are 1GB which means that you are restricted in document complexity and size. You would size your instances appropriately for your largest documents.   Preconditions An AWS account. AWS ac...
Read more