Bootstrap cluster with FluxCD

That is the next iteration of the "5 AM Club" Kubernetes migration. As you can remember from other entries published in this series, I started playing with Kubernetes daily. Not on a daily basis, but literally every single day. To be honest, I'm pretty happy with the results. However, my plan has one challenge, that requires to be solved. Build time was long ~15 minutes every day, and sometimes ESO operator faced an issue regarding using kustomize for Helm-based deployment. And just due to random constraints, I wanted to use one tool for bootstrapping a cluster. That is why I started with the layout presented last time. So logical split per operator with the base/overlay sub-group. Then I thought, why not just use some standard solution for that? The first idea was to extend the usage of Argo for my infrastructure. But... It requires setting ESO, Doppler secret, and ArgoCD installed. Then I can reconfigure apps from the git level. The challenge was, to make it easier, faster, and more standardised, than it was currently. Starting from beginning What Flux is? And what is not? Flux is a tool for keeping Kubernetes clusters in sync with sources of configuration (like Git repositories), and automating updates to the configuration when there is new code to deploy. We can use Flux to keep the configuration of our whole cluster in sync, with the Git repository. The funny thing is that we can configure both apps and infra with the usage of Flux. However, managing apps with Flux in my opinion is not the easiest and most comfortable solution. Especially, if we're changing versions quite often and we would like to have at least a few dependencies between apps and infra. For example, my Immich instance needs csi-driver-smb, which on the other hand requires external-secret-operator and external-secret-secret (the actual link between in-cluster secret and ESO ClusterSecretStore). So, every new relese, needs to be built and checks that all kustomizations are in place. Then actually deploy a new version. Very long process, also ArgoCD UI is just better, easier to use, and definitely more user-friendly - at least in my opinion. Repository structure So after a few initial rounds it's ended in the following state: . ├── README.md ├── clusters │ └── cluster0 │ └── flux-system │ ├── gotk-components.yaml │ ├── gotk-sync.yaml │ ├── infrastructure.yaml │ └── kustomization.yaml └── infrastructure └── controllers ├── argocd-operator │ ├── configmap-patch.yaml │ ├── kustomization.yaml │ ├── namespace.yaml │ └── patch-argocd-server-annotations.yaml ├── argocd-operator-apps │ ├── applications.yaml │ ├── kustomization.yaml │ ├── projects.yaml │ └── repositories.yaml ├── csi-driver-smb │ ├── csi-driver-smb.yaml │ └── kustomization.yaml ├── external-secrets │ ├── external-secrets-operator.yaml │ └── kustomization.yaml ├── external-secrets-secret │ ├── cluster-secret-store.yaml │ └── kustomization.yaml ├── tailscale-operator │ ├── kustomization.yaml │ └── tailscale-operator.yaml ├── tailscale-operator-secrets │ ├── kustomization.yaml │ └── tailscale-operator-exteral-secret.yaml └── traefik ├── kustomization.yaml └── traefik-ext-conf.yaml Now we need some explanation, right? Flux configuration clusters └── cluster0 └── flux-system ├── gotk-components.yaml ├── gotk-sync.yaml ├── infrastructure.yaml └── kustomization.yaml Here we have the configuration of our cluster, which is an awesome idea. In one repository we can have configurations for multiple clusters, based on provider, environment, or location, and manage them in a very simple way. Then we have regular flux-system files, so flux-system/gotk-components.yaml and flux-system/gotk-sync.yaml. Next, let's talk about my simple Kustomization file apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - gotk-components.yaml - gotk-sync.yaml - infrastructure.yaml This just tells Flux, that after bootstrapping itself, install manifests based on infrastructure.yaml file. So let's take a look at the most crucial part of the config. --- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: csi-driver-smb namespace: flux-system spec: interval: 1h retryInterval: 1m timeout: 5m sourceRef: kind: GitRepository name: flux-system path: ./infrastructure/controllers/csi-driver-smb prune: true wait: true --- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: external-secrets namespace: flux-system spec: interval: 1h retryInterval: 1m timeout: 5m sourceRef: kind: GitRe

Apr 15, 2025 - 08:49

That is the next iteration of the "5 AM Club" Kubernetes migration. As you can remember from other entries published in this series, I started playing with Kubernetes daily. Not on a daily basis, but literally every single day. To be honest, I'm pretty happy with the results. However, my plan has one challenge, that requires to be solved. Build time was long ~15 minutes every day, and sometimes ESO operator faced an issue regarding using kustomize for Helm-based deployment. And just due to random constraints, I wanted to use one tool for bootstrapping a cluster.

That is why I started with the layout presented last time. So logical split per operator with the base/overlay sub-group.

Then I thought, why not just use some standard solution for that?
The first idea was to extend the usage of Argo for my infrastructure. But... It requires setting ESO, Doppler secret, and ArgoCD installed. Then I can reconfigure apps from the git level.

The challenge was, to make it easier, faster, and more standardised, than it was currently.

Starting from beginning

What Flux is? And what is not?

Flux is a tool for keeping Kubernetes
clusters in sync with sources of configuration (like Git repositories),
and automating updates to the configuration when there is new code to deploy.

We can use Flux to keep the configuration of our whole cluster in sync, with the Git repository. The funny thing is that we can configure both apps and infra with the usage of Flux.

However, managing apps with Flux in my opinion is not the easiest and most comfortable solution. Especially, if we're changing versions quite often and we would like to have at least a few dependencies between apps and infra. For example, my Immich instance needs csi-driver-smb, which on the other hand requires external-secret-operator and external-secret-secret
(the actual link between in-cluster secret and ESO ClusterSecretStore). So, every new relese, needs to be built and checks that all kustomizations are in place.
Then actually deploy a new version. Very long process, also ArgoCD UI is just better, easier to use, and definitely more user-friendly - at least in my opinion.

Repository structure

So after a few initial rounds it's ended in the following state:

.
├── README.md
├── clusters
│   └── cluster0
│       └── flux-system
│           ├── gotk-components.yaml
│           ├── gotk-sync.yaml
│           ├── infrastructure.yaml
│           └── kustomization.yaml
└── infrastructure
    └── controllers
        ├── argocd-operator
        │   ├── configmap-patch.yaml
        │   ├── kustomization.yaml
        │   ├── namespace.yaml
        │   └── patch-argocd-server-annotations.yaml
        ├── argocd-operator-apps
        │   ├── applications.yaml
        │   ├── kustomization.yaml
        │   ├── projects.yaml
        │   └── repositories.yaml
        ├── csi-driver-smb
        │   ├── csi-driver-smb.yaml
        │   └── kustomization.yaml
        ├── external-secrets
        │   ├── external-secrets-operator.yaml
        │   └── kustomization.yaml
        ├── external-secrets-secret
        │   ├── cluster-secret-store.yaml
        │   └── kustomization.yaml
        ├── tailscale-operator
        │   ├── kustomization.yaml
        │   └── tailscale-operator.yaml
        ├── tailscale-operator-secrets
        │   ├── kustomization.yaml
        │   └── tailscale-operator-exteral-secret.yaml
        └── traefik
            ├── kustomization.yaml
            └── traefik-ext-conf.yaml

Now we need some explanation, right?

Flux configuration

clusters
└── cluster0
    └── flux-system
        ├── gotk-components.yaml
        ├── gotk-sync.yaml
        ├── infrastructure.yaml
        └── kustomization.yaml

Here we have the configuration of our cluster, which is an awesome idea. In one repository we can have configurations for multiple clusters, based on provider, environment, or location, and manage them in a very simple way. Then we have regular flux-system files, so flux-system/gotk-components.yaml and flux-system/gotk-sync.yaml. Next, let's talk about my simple Kustomization file

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
  - infrastructure.yaml

This just tells Flux, that after bootstrapping itself, install
manifests based on infrastructure.yaml file. So let's take a look at the most crucial part of the config.

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: csi-driver-smb
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/csi-driver-smb
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: external-secrets
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/external-secrets
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: external-secrets-secret
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/external-secrets-secret
  dependsOn:
    - name: external-secrets
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tailscale-operator
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/tailscale-operator
  dependsOn:
    - name: external-secrets-secret
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tailscale-operator-secret
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/tailscale-operator-secrets
  dependsOn:
    - name: external-secrets
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: traefik
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/traefik
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: argocd-operator
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/argocd-operator
  dependsOn:
    - name: tailscale-operator
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: argocd-apps
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/argocd-operator-apps
  dependsOn:
    - name: argocd-operator
  prune: true
  wait: true

As you can see I heavily rely on dependsOn command. That is due Flux and Kubernetes archtecture design. When we're applying a Kustomization, we do not control the order of creating the resources. Of course, if we put in one file, service, ingress, and deployment, Kubernetes will know how to handle that. The problem is when we deploy an application that has dependencies on each other, that is not what Kubernetes understands. So we need to specify it directly. In my example csi-driver-smb, external-secrets, and traefik can be installed in parallel, but then we have some relations. At the end, we're installing Argo's app-of-apps, which requires in general all previous components, besides traefik - I'm routing my traffic through Tailscale there, not via the Internet.

Now you can think:

App-of-apps? You said infra only!

That is right, my logic was quite complex, to be honest here. Do you remember, when I wrote about use-case? Bootstraping the whole cluster so that I will be able to work with it fast. Apps are part of the overall cluster at the end. So my app-of-apps definition was very simple:

infrastructure/controllers/argocd-operator-apps
├── applications.yaml
├── kustomization.yaml
├── projects.yaml
└── repositories.yaml

Let's dive into it one, one by one.

applications.yaml

 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
   name: self-sync
   namespace: argocd
 spec:
   syncPolicy:
     automated:
       selfHeal: true
   project: non-core-namespaces
   source:
     repoURL: https://codeberg.org/3sky/argocd-for-home
     targetRevision: HEAD
     path: app-of-apps
   destination:
     server: https://kubernetes.default.svc
     namespace: argocd

That is a very simple definition of my main Application that controls all apps running on my cluster. Nothing very special here, I was experimenting with codeberg, but let's talk about that later.

kustomization.yaml

  ---
  apiVersion: kustomize.config.k8s.io/v1beta1
  kind: Kustomization

  namespace: argocd
  resources:
    - repositories.yaml
    - projects.yaml
    - applications.yaml

Kustomization is simple, just ordered in the logic path.

Repository -> Projects -> Application

projects.yaml

  ---
  apiVersion: argoproj.io/v1alpha1
  kind: AppProject
  metadata:
    name: non-core-namespaces
    namespace: argocd
  spec:
    description: Allow argo deploy everywhere
    sourceRepos:
      - 'https://codeberg.org/3sky/argocd-for-home'
    destinations:
      - namespace: '*'
        server: https://kubernetes.default.svc
    namespaceResourceWhitelist:
      - group: '*'
        kind: '*'
    clusterResourceWhitelist:
      - group: '*'
        kind: '*'

As I'm creating namespaces as part of my application setup, permission granted to Argo's Projects are very wide. In this case, I just trust the GitOps process.

repositories.yaml

  apiVersion: external-secrets.io/v1beta1
  kind: ExternalSecret
  metadata:
    name: gitops-with-argo-secret
    namespace: argocd
  spec:
    refreshInterval: 6h
    secretStoreRef:
      name: doppler-auth-argocd
      kind: ClusterSecretStore
    target:
      name: gitops-with-argo
      creationPolicy: Owner
      template:
        type: Opaque
        metadata:
          labels:
            argocd.argoproj.io/secret-type: repository
        data:
          type: git
          url: https://codeberg.org/3sky/argocd-for-home
          username: 3sky
          password: "{{ .password }}"
    data:
      - secretKey: password
        remoteRef:
          key: CODEBERG_TOKEN

This is probably the most interesting part of the configuration. As we're unable to inject our password directly into argocd.argoproj.io/secret-type: repository, which is a regular Kubernetes Secret. We need to generate the whole object
with ESO - details can be found here.

And that's it. Now let's talk about the actual bootstrap process.

Bootstraping environment

Spin up infrastructure with Terraform.

terraform apply -var="local_ip=$(curl -s ifconfig.me)"

Installing K3S with local fork of k3s-ansible.

ansible-playbook playbooks/site.yml -i inventory.yml

Load KUBECONFIG

export KUBECONFIG=/home/kuba/.kube/config.hetzner-prod
kubectl config use-context k3s-ansible

Doppler configuration (we need an initail secret somewhere)

kubectl create namespace external-secrets
kubectl create secret generic \
  -n external-secrets doppler-token-argocd \
  --from-literal dopplerToken=""

Bootstrap the cluster

flux bootstrap github \
  --owner=3sky \
  --repository=flux-at-home \
  --branch=main \
  --path=./clusters/cluster0 \
  --personal

Where GitHub is supported natively (via GitHub Apps). Solutions like Codeberg needs a more direct method.

flux bootstrap git \
  --url=ssh://git@codeberg.org/3sky/flux-at-home.git \
  --branch=main   \
  --path=./clusters/cluster0   \
  --private-key-file=/home/kuba/.ssh/id_ed25519_git

Get ArgoCD password if needed

kubectl --namespace argocd \
  get secret argocd-initial-admin-secret \
  -o json | jq -r '.data.password' | base64 -d

Summary

With this set of commands, I'm able to set fresh clusters, with Hetzner or whatever in literally 7 minutes. Starting from installing k3s to having applications exposed via Cloudflare tunnel, or Tailscale tailnet. To be honest I'm really satisfied with the result of this exercise. Besides a long-running cluster for my self-hosted apps. I can quite easily, fast, and with low costs test new versions of ArgoCD, or smb-driver operator, without harming my current setup! And that is great, to be honest. Especially for self-hosting, where it's good to have apps that are working in let's say production mode, even if there is only one user.