Getting started with microk8s

Overview

I built a k8s cluster on 3 Intel NUCs with microk8s that includes clustered storage using Mayastore (OpenEBS), snapshots, load balancing and control plane resilience.

Selecting Hardware

I chose Intel NUC devices specifically for their support of an extra 2.5" drive in addition to an NVMe OS drive. There are plenty of other options including a bunch of Thinkpads. I did quite a bit of CPU comparison to optimize cost to performance. The exact processor I ended up with was Intel(R) Core(TM) i7-10710U CPU in a NUC10i7FNHN. This was great in my opinion because each processor has 6 cores and I was able to pick up 3 for about USD $500 a piece. I added a 256G NVMe drive for the OS, 32G of memory, and 1TB Samsung 860 SSD drive for data.

Your mileage may vary but I'd recommend at least 4 cores and 16GB of memory per device.

For network there is a Ubiquiti 24 port gig switch connecting the NUCs along with a Synology for NFS but any gig switch will work and the NFS mount is optional.

Once you have the hardware in place and connected, you're ready for the next step - Initial config and OS installation.

Initial Config and OS Installation

Since I wanted to use microk8s I chose to use Ubuntu. Yes, you can run it on things other than Ubuntu but I didn't want to.

  1. Get a Ubuntu install on USB drive. I won't go into detail since there are many good guides on this process. I used BalenaEtcher.
  2. Boot to the BIOS and hit F9 for defaults. I found this was the fastest way to ensure my hardware was identically configured. If you have a different preference for configuration, just apply it before loading the OS.
  3. Perform a minimal Ubuntu install. This reduces the number of packages that need to be updated.
    1. I used the entire disk for Ubuntu with no special partitioning.
    2. Ensure that SSH Server is selected for installation.
    3. Set a static IP for the node.
  4. Once the installation is complete, I installed a few packages.
    1. sudo apt update && sudo apt upgrade -y
    2. sudo apt install vim iputils-ping fwupd -y
  5. I then upgraded the firmware on the devices with the following commands.
    1. sudo fwupdmgr get-updates
    2. sudo fwupdmgr upgrade
  6. Update /etc/hosts with entries to each node in the cluster.
    1. sudo hostnamectl hostname nuc001
    2. echo "192.168.1.100 nuc001 nuc001.local" | sudo tee -a /etc/hosts
    3. echo "192.168.1.101 nuc002 nuc002.local" | sudo tee -a /etc/hosts
    4. echo "192.168.1.102 nuc003 nuc003.local" | sudo tee -a /etc/hosts

If you didn't set a static IP during installation you can do so by editing the default netplan config file and restarting the service. Ensure that DNS is pointing to something that will generally always be valid. Not a DNS service running on your k8s cluster :).

Example netplan config

network:
  ethernets:
    eno1:
      dhcp4: false
      addresses:
        - 192.168.1.100/24
      routes:
        - to: default
          via: 192.168.1.1
          metric: 100
          on-link: true
      nameservers:
          addresses:
            - 192.168.1.1
  version: 2

Hugepages

Huges pages will be needed by Mayastor. This creates a bit of an issue for any container you may start that needs huge pages because Mayastor will consume everything allocated by default. You'll want to increase this but how much you increase it will depend on how much memory you have.

# Check huge page config
# https://blog.nuvotex.de/running-postgres-in-kubernetes-with-hugepages/
# https://help.ubuntu.com/community/KVM%20-%20Using%20Hugepages
cat /proc/meminfo | grep Huge
# /etc/sysctl.conf:vm.nr_hugepages = 2560
# Reboot after changing

Once this is done you should have 3 computers that have static IPs and are able to ping each other by hostname. If you chose to add the extra SSD drive for storage, don't worry about it for now. That will be used once we install Mayastor.

blog