Raspberry Pi Kubernetes Cluster
Here I am looking at setting up my Raspberry Pi cluster to host the various Docker containers I run at home. I am using four Raspberry Pi 4B which are all build with Ubuntu 21.04 and configured as per Ubuntu Base configuration.
There are various ways to setup a Kubernetes cluster including:
- Rancher K3S
When learning Kubernetes it is useful practise to configure all components manually. Often people follow the tutorial Kubernetes The Hard Way. For this installation I want to get up and running quickly so will be looking at automated build solution.
The official CNCF tool for provisioning Kubernetes clusters in a variety of shapes and forms e.g. single-node, multi-node, HA, self-hosted. It’s as close to a “vanilla” Kubernetes you get in production but consumes a lot of resources.
## Rancher K3S OR Canonical Microk8s K3s and Microk8s are both lightweight implementation of Kubernetes they have various differences but a key one to understand is high availablity approach.
|Kubernetes Node Roles||Every node is a worker node||Agent Node(s)|
|Kubernetes API Server||Every node is an API server||Server Node (s)|
|Datastore HA||Embedded Dqlite when >3 nodes||Requires External datastore|
I want my four node cluster to be highly available as easily as possible so for this deployment I chose Mircok8s.
Visual Studio Code Remote SSH
The four Raspberry Pi which make up my cluster are headless. The Visual Studio Code Remote Development extension pack makes configuring headless systems very easy.
/boot/firmware/cmdline.txt and add following options:
sudo reboot for changes to take effect.
MicroK8s is supplied as a snap, there are various Kubernetes releases these are available as snap channels. To see all available versions we can query what channels are available. When I’m running this the current Kubernetes version is 1.21 and I filter results on this we can see all.
snap info microk8s | grep 1.21 latest/stable: v1.21.1 2021-05-17 (2215) 168MB classic latest/candidate: v1.21.1 2021-05-13 (2214) 168MB classic latest/beta: v1.21.1 2021-05-13 (2214) 168MB classic latest/edge: v1.21.1 2021-05-19 (2227) 168MB classic 1.21/stable: v1.21.1 2021-05-17 (2215) 168MB classic 1.21/candidate: v1.21.1 2021-05-14 (2215) 168MB classic 1.21/beta: v1.21.1 2021-05-14 (2215) 168MB classic 1.21/edge: v1.21.1 2021-05-20 (2232) 168MB classic 1.14/stable: v1.14.10 2019-12-20 (1121) 164MB classic
I’ll look to install the stable release of 1.21 on each using:
sudo snap install microk8s --classic --channel=1.21/stable
When the snap is loaded and running on all nodes we can look to form them into a cluster. On first node run following and it will generate a token to run on remote node to add it.
## From first node in cluster generate token sudo microk8s add-node From the node you wish to join to this cluster, run the following: microk8s join 192.168.1.100:25000/73d707e38ddb7c7bbcf29a328a505179/7e8d2c9a15af
Then on remote node run command with generated token.
sudo microk8s join 192.168.1.100:25000/73d707e38ddb7c7bbcf29a328a505179/7e8d2c9a15af Contacting cluster at 192.168.1.100 Waiting for this node to finish joining the cluster. ..
Repeat this process (generate a token, run it from the joining node) for the third and forth nodes. When all done from any node we can query the cluster state see all four nodes and check high availability status.
sudo microk8s status --wait-readymicrok8s is running high-availability: yes datastore master nodes: 192.168.1.100:19001 192.168.1.101:19001 192.168.1.102:19001 datastore standby nodes: 192.168.1.103:19001 addons: enabled: ha-cluster # Configure high availability on the current node disabled: dashboard # The Kubernetes dashboard dns # CoreDNS helm # Helm 2 - the package manager for Kubernetes helm3 # Helm 3 - Kubernetes package manager host-access # Allow Pods connecting to Host services smoothly ingress # Ingress controller for external access linkerd # Linkerd is a service mesh for Kubernetes and other frameworks metallb # Loadbalancer for your Kubernetes cluster metrics-server # K8s Metrics Server for API access to service metrics portainer # Portainer UI for your Kubernetes cluster prometheus # Prometheus operator for monitoring and logging rbac # Role-Based Access Control for authorisation registry # Private image registry exposed on localhost:32000 storage # Storage class; allocates storage from host directory traefik # traefik Ingress controller for external access
## Install kubectl The Microk8s install deploys its own client and to execute have to remember to prefix everything with microk8s. I have to switch between systems often and to avoid confusion prefer to install normal kubectl. Again this is available as a snap in multiple versions.
snap info kubectl | grep 1.21 latest/stable: 1.21.1 2021-05-14 (1970) 9MB classic latest/candidate: 1.21.1 2021-05-14 (1970) 9MB classic latest/beta: 1.21.1 2021-05-14 (1970) 9MB classic latest/edge: 1.21.1 2021-05-14 (1970) 9MB classic 1.21/stable: 1.21.1 2021-05-13 (1970) 9MB classic 1.21/candidate: 1.21.1 2021-05-13 (1970) 9MB classic 1.21/beta: 1.21.1 2021-05-13 (1970) 9MB classic 1.21/edge: 1.21.1 2021-05-13 (1970) 9MB classic
We want to install version to match Kubernetes and copy the Microk8s kubectl config to be kubectl default config.
sudo snap install kubectl --classic --channel=1.21/stable mkdir ~/.kube sudo microk8s config > ~/.kube/config
One of the benefits I hoped to realize by using Microk8s was low memory footprint. We can see here our 8GB Raspberry Pi4B with Microk8s running it still shows 6GB available.
free -m total used free shared buff/cache available Mem: 7809 1173 4718 4 1918 6200 Swap: 0 0 0
So all in all very easy to setup and get a low footprint highly available Kubernetes cluster running on Raspberry Pi cluster with Microk8s.