Pet Kubernetes cluster on a budget

Lately, I have been having a few ideas and problems I would like to solve with software, and I need a way to deploy these artifacts; moreover, I want to move my blog away from Netlify because I am running out of build minutes.

Buuuuuut I am a cheap person, a bean counter, a penny-pincher, for as long as I can remember. When I was a child, my sister and I would get our weekly allowance, and almost immediately she would run to the nearby bakery to get sweets. I, on the other hand, would run directly to my piggy bank to deposit my shiny new coin, like Scrooge McDuck. 🦆

Old habits die hard, and while I have learned that money is to be spent (among other lessons, like “buy cheap, buy twice” 🫣), I like to spend my money wisely. This means I try to be cost-efficient when running my side projects, while still automating as much as possible - bringing me right back to self-hosting my stuff.

I went server shopping, and ended up choosing a machine from Netcup: a VPS 1000 ARM G11 with 6 virtual cores, 8GB RAM and 256GB SSD storage, plus unlimited traffic. After VAT, it costs 9,07€ per month, which is a pretty good bang for the buck.

Commercial Break 🤑

You can find the latest deals from Netcup here: https://www.netcup.com/en/deals

I have also generated a few 5€ affiliate coupons that can be used by anyone: 36nc17873944054, 36nc17873944053, 36nc17873944052, 36nc17873944051, 36nc17873944050

Afterwards, I needed to decide how I wanted to deploy stuff to my server. What I wanted was a declarative way to deploy my applications, much like we do with Kubernetes at $WORK, but without its complexity. I tried platforms like Coolify, Dokku and Dokploy, and also bare metal with Docker/Podman, and they worked, but not the way I wanted - not because they are bad, but because I am not a fan of UI-driven setups or having a shell-script powered setup. This experience helped me realize what I really want is Kubernetes. 😬

I have heard multiple times that operating a Kubernetes cluster is hard and complex, but I guess I will have to see for myself. Time to go shopping for k8s distributions.

The Contenders

There are almost 70 Kubernetes distributions certified by CNCF, plus many others that are not certified. Most of them are full distributions that come with all the bells and whistles, meaning they are sort of resource-heavy. I narrowed down my choices to the lighter distributions, and came up with the following table:

In the end I chose k3s - objectively, only because of its large community and its stability. I suppose any of the distributions above would have worked for me.

The Special Mention

Talos Linux is also worth a mention, because it is actually an immutable Linux distro built exclusively for running Kubernetes. Most of the traditional system components are stripped out, and you can only interact with the cluster using the talosctl CLI tool - you cannot even SSH into the machine. The biggest advantage is that we get a more secure system with a smaller attack surface - which, in my case, was a big disadvantage, since I still want to tinker around via SSH.

Setting up the cluster

Before getting my hands dirty, I refined the workflow I wanted to have:

[ Code Commit ]
      │
      ▼
[ GitLab CI: Build & Tag Container ] ──► [ Container Registry ]
                                                    │
                                                    ▼
[ Live via Cloudflare ] ◄── [ CD Detects Image Change & Deploys ]

To achieve that, I would use the following tools and components:

• GitLab: I know it quite well because of $WORK, and they provide a generous free tier that includes a private Container Repository.
• k3s: the Kubernetes distribution I chose.
• FluxCD: I like Infrastructure as Code (IaC) and GitOps, and FluxCD lets you keep your manifests in Git and does not require CI/CD pipelines to access your cluster, which is a nice security bonus.
• Sealed Secrets: they enable you to encrypt secrets outside of the cluster and store them in version control.
• Cloudflare Tunnel: it lets you expose your application to the web without an external IP or webserver. I was going to use Cloudflare anyway, so removing the ingress server from my cluster is an added bonus.

Now, let’s get our hands dirty.

Flashing the server

Before anything, I flashed my server with the Ubuntu cloudimg 26.04 image. It is a minimal Ubuntu-based image that comes with unattended-upgrades enabled, reducing maintenance effort.

Creating a git repository

Since we are doing IaC, we need a git repository to store the cluster configuration. In my case, I created a repository called infrastructure in my GitLab account. Later we will need to generate a deploy token for this repository.

k3s installation

Installing k3s is almost trivial:

1. SSH into the server.
2. Run curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable traefik" sh. I have disabled traefik because I will later expose my services to the web via Cloudflare.
3. Verify the single node is available with k3s kubectl get node.

You can also export the KUBECONFIG file from /etc/rancher/k3s/k3s.yaml if needed.

FluxCD setup

Flux needs access to the infrastructure GitLab repository to read manifests and push updates. To create an access token:

1. Go to GitLab -> Edit Profile -> Access Tokens (or use a Project Access Token).
2. Create a token with the api, read_repository, and write_repository scopes.
3. Copy and export it as an environment variable locally:

We can now install the Flux CLI with:

Afterwards, we need to bootstrap Flux to our cluster. The command below includes the image automation controller, which we will use to automatically deploy the latest version of our application images:

Finally, we need to grant Flux access to our container registry, where we will store the OCI containers with our application code. To do that, create a new deploy token in the GitLab infrastructure repository by going to Settings > Repository > Deploy Tokens. Give it the read_registry permission to be able to scan the container registry. Afterwards, run the following command in your cluster:

Sealed Secrets

In the infrastructure Git repository, we start by creating the Helm repository for Sealed Secrets:

# clusters/gubernetes/sealed-secrets/repository.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: sealed-secrets
  namespace: flux-system
spec:
  interval: 1h
  url: https://bitnami.github.io/sealed-secrets

Then, we create the Helm release:

# clusters/gubernetes/sealed-secrets/release.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: sealed-secrets
  namespace: flux-system
spec:
  interval: 1h
  chart:
    spec:
      chart: sealed-secrets
      version: ">=2.0.0"
      sourceRef:
        kind: HelmRepository
        name: sealed-secrets
  install:
    crds: CreateReplace
  upgrade:
    crds: CreateReplace
  values:
    fullnameOverride: sealed-secrets-controller

And finally, we let FluxCD know about them:

# clusters/gubernetes/sealed-secrets-sync.yaml 
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: sealed-secrets
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./sealed-secrets
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

After committing the changes, the encryption certificate can be retrieved from the cluster by using kubeseal:

You can then use the certificate to encrypt your secrets:

Cloudflare Tunnel

For the sake of simplicity and security I did not set up an ingress in my cluster, and will be using Cloudflare Tunnel instead. With Cloudflare Tunnels, the cluster reaches out to Cloudflare, meaning you don’t need to open any inbound ports on your server. The obvious downside is that your infrastructure is tightly coupled to it.

Setting up a tunnel is very simple:

1. Log into the Cloudflare dashboard.
2. Go to Networks -> Tunnels and click Create a tunnel.
3. Choose Docker as your connector, and copy the command - you need to extract the tunnel token from it.
4. Create a new secret in the cluster, replacing <token> with the value from the previous step. In a strict GitOps setup, you would encrypt this token with kubeseal and commit it to your repository, but for the initial bootstrap, creating the secret directly via kubectl works too:

5. Create a manifest to deploy the tunnel:

# cloudflare-tunnel.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudflared
  labels:
    app: cloudflared
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cloudflared
  template:
    metadata:
      labels:
        app: cloudflared
    spec:
      containers:
        - name: cloudflared
          image: cloudflare/cloudflared:latest
          args:
            - tunnel
            - --no-autoupdate
            - run
          env:
            - name: TUNNEL_TOKEN
              valueFrom:
                secretKeyRef:
                  name: cloudflare-tunnel-token
                  key: TUNNEL_TOKEN
          resources:
            limits:
              cpu: "500m"
              memory: "256Mi"
            requests:
              cpu: "100m"
              memory: "128Mi"

6. Apply the manifest to the cluster:

You should now see the tunnel connection in the Cloudflare dashboard, and you can start exposing your services to the web:

1. In Networks -> Tunnels, click the Routes tab, then Add route.
2. Select Published Application.
3. Choose your domain, subdomain, and path, if applicable. The domain/subdomain should have no A or CNAME records set up, otherwise you will receive an error.
4. In the Service URL field, enter the service path inside the cluster - a simple web app running on port 80 would look like http://<service>.<namespace>.svc.cluster.local:80.
5. Click Save changes, and wait until Cloudflare applies the changes.
6. Verify your application is working.

Deploying applications

This is a regular Kubernetes cluster, so you can write a YAML manifest and deploy anything using kubectl apply -f manifest.yaml, or you can leverage Flux to automatically deploy your apps.

As an example, this is how I deploy my blog - ./apps/blog contains standard service and deployment manifests, and the manifest below tells Flux how often to check for changes, where to find my application images, and how my images are versioned:

# clusters/gubernetes/blog-kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: blog
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./apps/blog
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
  name: blog-image-repo
  namespace: flux-system
spec:
  image: registry.gitlab.com/<redacted>/blog
  interval: 1m0s
  secretRef:
    name: gitlab-registry-credentials
---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
  name: blog-image-policy
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: blog-image-repo
  filterTags:
    pattern: '^main-(?P<build>[1-9][0-9]*)-[a-f0-9]+$'
    extract: '$build'
  policy:
    numerical:
      order: asc

Pay special attention to the ImagePolicy: Flux expects your image version to be incremental, meaning you must use a monotonically increasing value like your pipeline ID in every image tag, otherwise you will have problems.

Conclusion

Phew, this was more complicated than I imagined! 🥵

While setting up each individual component of the stack was easy, putting everything together was an effort that cost me a few thousand lines of YAML and a few days of exploration and debugging. Rather than a hassle, it was quite a good learning opportunity, and by leveraging IaC I can rebuild the whole thing in 10 minutes.

When compared with managed solutions, I got almost 7GB RAM free for workloads for a fraction of the cost of AWS or GCP. And if I ignore the maintenance effort of a few minutes per week, I got a pretty good bang for the buck.

Before wrapping up, I must admit my job here is not done… I still want to add telemetry to the cluster - I tried using Grafana Alloy, but it keeps breaking - and to take proper care of my databases - using a PersistentVolume is not the ideal solution. But that is a problem for another day. :)