Specification

Object	Amount	Image
Raspberry Pi 5 8GB	4
Rack panel ERUSCUS	1
Gurdini USB HUB 120W	1
Fan 30x30	4
SmartBuy 128 ГБ	4
Top NVME adapter x1001	4
Original metal body radiator	4
Cables	4
Patches 0.15	3
Patches 0.30	1
TP-Link Switch TL-SG108	1

Eventual setup

Cluster setup

how-to

Master

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='--flannel-backend=none --disable-network-policy' sh -

helml install cilium cilium/cilium --namespace=kube-system --version 1.17.1 --set bpf.masquerade=true --set hubble.relay.enabled=true --set hubble.ui.enabled=true --set kubeProxyReplacement=true

If error:

• add cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 to the end of the line in /boot/firmware/cmdline.txt

Workers

Get token on master /var/lib/rancher/k3s/server/node-token

Run

curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=mynodetoken sh -

Access

If error while kubectl get smth:

how-to

export KUBECONFIG=~/.kube/config

mkdir ~/.kube 2> /dev/null
sudo k3s kubectl config view --raw > "$KUBECONFIG"
chmod 600 "$KUBECONFIG"

Access from my machine

k3s kubectl config view --raw
Copy to local ~/kube/config and change 127.0.0.1 ip

Machines preparation

Initial setup

Prepare bootable SD Card (can use raspberry tool or just balenaEtcher) with Raspberry Pi OS Lite 64-bit (cause it's just clear Debian 12 you can keep it afterwards)

Than mount it and change:

1. add dtparam=nvme to config.txt
2. HASH=$(openssl passwd -6 -stdin)
   Type the password, hit enter, then Control + D
   echo user:$HASH > userconf.txt
   

Now we've prepared SD Card, insert

sudo rpi-eeprom-config --edit

Change BOOT_ORDER to BOOT_ORDER=0xf416 - it's the 6 which represents NVMe boot mode.

Add a line PCIE_PROBE=1

Copy rpi image from other local for ex machine:
wget 192.169.0.2:8080/rasp.img
and then
dd if=./rasp.img of=/dev/nvme0n1

That's it, then same:

mount /dev/nvme0n1p1 /mnt
touch /mnt/ssh
echo "dtparam=nvme" | sudo tee /mnt/config.txt

HASH=$(openssl passwd -6 -stdin)

# Type the password, hit enter, then Control + D

echo user:$HASH > /mnt/userconf.txt

Values of booting devices priority: link
In case if the OS was installed on nvme(and bricked for ex as in my case :) ) you should disconnect the disk(cause I just can't boot into the OS to change boot priority) and boot from SD Card or delete PCIE_PROBE=1 and change boot order to 0xf1 (restart, SD Card)

The main boot config file is in /boot/firmware/config.txt

Manual link

Disable swap!

Kernel recompilation

In case you need extra features, for ex BTF, steps: Official man

A pair of moments:

1. If you don't see BTF feature in menuconfig: link
2. If libelf not found -> apt install libelf-dev
3. apt install dwarves