My Homelab Setup

Getting started

Dependencies

Install dependencies (Arch):

pacman -Sy opentofu kubectl helm helmfile python fluxcd

Promxox

We first need to configure a Proxmox user for terraform to act on behalf of and a token for the user.

# Create the user
pveum user add terraform@pve

# Create a role for the user above
pveum role add Terraform -privs "Datastore.Allocate Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit Pool.Allocate Sys.Audit Sys.Console Sys.Modify SDN.Use VM.Allocate VM.Audit VM.Clone VM.Config.CDROM VM.Config.Cloudinit VM.Config.CPU VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options VM.Migrate VM.Monitor VM.PowerMgmt User.Modify Pool.Audit"

# Assign the terraform user to the above role
pveum aclmod / -user terraform@pve -role Terraform

# Create the token and save it for later
pveum user token add terraform@pve provider --privsep=0

Provisioning with OpenTofu/Terraform

Create a file proxmox/tf/credentials.auto.tfvars with the following content, making sure to replace as necessary:

proxmox_api_endpoint = "https://<domain or ip>"
proxmox_api_token    = "terraform@pve!provider=<token from last step>"

Customize the other variables in proxmox/tf/vars.auto.tfvars and double check the configuration.

When ready, run opentofu apply. The command might fail the first time if provisioning from scratch, but it seems to be fine when running it a second time.

Creating a Docker swarm

The Docker swarm acts as a launchpad for the rest of the infrastructure. It bootstraps a Portainer, Traefik, and Gitea deployment so that remaining configuration can be done through Portainer and Git.

# Add SSH keys to known_hosts
ansible-inventory -i inventory/dolo --list |\
  jq -r '._meta.hostvars | keys[]' |\
  grep 'stingray' |\
  while read -r line; do
    ssh-keygen -R "$line"
    ssh-keyscan -H "$line" >> ~/.ssh/known_hosts
  done

# Initialize swarm
ansible-playbook -i inventory/stingray swarm.yml

Traefik will be listening on hosts:

• git.mnke.org
• git.stingray.mnke.org
• portainer.stingray.mnke.org

Set DNS records or edit your hosts file to point those domains to a swarm node.

Creating a k3s cluster

Set up Ansible:

# Tested on Python 3.13.1
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
ansible-galaxy collection install -r proxmox/ansible/collections/requirements.yml

Set up the k3s cluster:

# Necessary because the hosts.yml file contains a relative path to the terraform
# project directory
cd proxmox/ansible
# Remove/scan keys
ansible-inventory -i inventory/dolo --list |\
  jq -r '._meta.hostvars | keys[]' |\
  while read -r line; do
    ssh-keygen -R "$line"
    ssh-keyscan -H "$line" >> ~/.ssh/known_hosts
  done
ansible-playbook lvm.yml site.yml -i inventory/dolo
# You should be left with a kubeconfig. Move it to ~/.kube/config. If you
# already have a ~/.kube/config file, make sure to back it up first.
mv kubeconfig ~/.kube/config
# Verify that you can connect to the cluster
kubectl get nodes

# Back to root repo directory
cd -
# Verify deployment and service
kubectl apply -f proxmox/k8s/examples/001-example.yml
# This should succeed, and an IP should have been allocated by metallb. Check
# with the following command:
kubectl describe nginx
# Now try checking that the deployment works:
curl http://[allocated-ip]
# Clean it up
kubectl delete -f proxmox/k8s/examples/001-example.yml

Lastly, run kubectl apply -f k8s/pre-infrastructure/dns-config-map.yaml so pods in the cluster automatically pick up on the DNS server.

Set up GitOps

Prerequisites:

• Gitea is set up
• Infisical or some other secrets provider is set up

Follow the Infisical guide to get a client id and secret. Use it to apply a manifest in the external-secrets namespace.

Create a Gitea token with at least enough privileges mentioned in this guide.

Run the commands below.

export GITEA_TOKEN=<token>
flux bootstrap gitea \
  --owner=tony \
  --repository=homelab \
  --hostname=https://git.mnke.org \
  --token-auth \
  --path=k8s/clusters/dolo \
  --personal \
  --branch=master

Credits

• Some inspiration and guidance was taken from Andreas Marqvardsen's blog post
• An automated setup of a k3s cluster from Techno Tim's Ansible roles