Cluster Setup | Dans Dev Diary

Pre-requisites

Configuring cgroups

sudo nano /boot/firmware/cmdline.txt

append the following to the end of the file: cgroup_memory=1 cgroup_enable=memory

Master Nodes

K3S Token stored in 1Password
MySQL password stored in 1Password

Simple Master Node Setup

For a simple master node setup, run the following command on each master node:

curl -sfL https://get.k3s.io | sh -s - server --token="<k3s_token>" --datastore-endpoint="mysql://<mysql_usere>:<my_sql_password>@tcp(<my_sql_ip>:3306)/homelab" --tls-san=<nginx_ip>

Master Node Setup for Prometheus

If you want to setup a master node with a custom label, you can use the following command:

curl -sfL https://get.k3s.io | sh -s - server --token="<k3s_token>" --datastore-endpoint="mysql://<mysql_usere>:<my_sql_password>@tcp(<my_sql_ip>:3306)/homelab" --tls-san=<nginx_ip> --node-label="prometheus=true"

Agent Nodes

Setting up each agent node with:

curl -sfL https://get.k3s.io | sh -s - agent --token="<k3s_token>" --server https://<nginx_ip>:6443

Other Settings

Increase the number of open files

add the following to /etc/security/limits.conf

*               soft    nofile          100000
*               hard    nofile          100000

edit the following file: /etc/pam.d/common-session adding the following before end of the file.

session required pam_unix.so

edit /etc/pam.d/common-session-noninteractive and add the following line to the end of the file:

session required pam_unix.so

Common Commands

Copy kube config file to local machine

cp /etc/rancher/k3s/k3s.yaml ~/.kube/config

Uninstall K3s

/usr/local/bin/k3s-uninstall.sh