Until a couple of days ago, all my Kubernetes config has lived simply as yaml-file residing in a tree in my home directory. While I had all the configuration, and knew where to find it if I needed to change it, it became a bit difficult to keep overview over time, and especially was it difficult to see (without checking) which yaml-files was applied or not.

While I am generally a pretty metidious and values accuracy in what I do, I had to admit – I want more control over my Kubernetes cluster. I’ve been looking at continuous deploy systems for a while, regretting not doing it from the start, and dreading converting to it. A weekend with nothing else on the schedule gave me the time I needed to start on the journey, though, and while it was tedious work and I’m not 100% finished (there are things I’m not sure how to do best yet), it was extremely rewarding all the way, making me not lose motivation before reaching a pretty solid «almost everything as code» status where I am now. And it’s as always, time to write about my experience!

Selecting system components.

There exists a few different ones. The most prominent ones are probably Flux and ArgoCD. Apparently Flux is a bit more Kubernetes-native, so it was tempting in that way. ArgoCD, however, is more userfriendly and comes with a pretty nice user interface which gives you a nice visual overview of what’s installed in your cluster. I decided to go for ArgoCD, because there seemed to be a bit more stuff around it out there. It was an entirely subjective and maybe not well rationalized decision, but all in all I am pretty happy with the outcome. Also, the documentation for ArgoCD is pretty decent and well structured.

ArgoCD centers around «applications», which fit well with how I have been organizing my cluster. This blog is thus in my wordpress application in ArgoCD, where I have gathered all configuration needed to get wordpress up and going.

I also decided to split out a few common resources into the owning applications, namely some my externalnames service which just created a bunch if A-records to my publically available IPv4 address. I am still doing that, but now in the form of different services in their own applications, so that the configuration lives in the application they belong. I have also put some unifi firewall rule configurations into the applications, where I could do that cleanly.

While you can let the repo live in github, I decided to selfhost also the repository. However, an ultimate stretch goal I have is being able to bootstrap and reboot my whole cluster from scratch in a scripted way, and letting the full repo live inside the cluster would give me a solid chicken and egg problem. The solution I decided to go for was to have two ArgoCD repositiories: One bootstrap repository where I install enough to get to a point where I can deploy a working repository on-prem, and then another on-prem repository in the cluster, where the rest of my applications lives.

For internal repository I chose gitea, which is pretty similar to github, and as of recently even comes with GitHUB Action styles pipelines – which I’ll write about in another blog post.

I ended up with the following in the bootstrap repositorty:

• Some bootstrap scripts installing k3s and calico itself, plus the ArgoCD CRDs.
• ArgoCD repo containing:
  • Some calico resources like global network policies, ip pools etc which is needed to get other workload going
  • ArgoCD itself. Apparently Argo can self-deploy. That’s something I have yet to test, as I first installed it in a script outside ArgoCD. I still wanted ArgoCD to be able to handle its own configuration, so it needed to be an ArgoCD application too.
  • Storage definitions that the applications might depend on.
  • externaldns, both the internal and internal one. I was in doubt if this belongs in the bootstrap repo, but decided it could come in handy to get the UI up and running before the internal repo is accessible, if needed for troubleshooting
  • Traefik, for the same reason as above, I needed an ingress for ArgoCD up and running.
  • Gitea, to be able to reference the applications that live in gitea
  • …and any other resources that the bootstreap repo applications might depend on to deploy.

As earlier mentioned, ArgoCD centers around applications. You’ll configure an application through an Application custom resource, that define an application. The one for traefik can for example be configured like this:

kind: Application
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "2"
  name: traefik
  namespace: argocd
spec:
  destination:
    namespace: traefik-external
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: git@github.com:my-org/my-bootstrap-repo.git
    targetRevision: main
    path: apps/traefik
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

This just says that the application lives in apps/traefik of my boostrap repo, that it should selfHeal, that it should delete resources that is no longer valid if the configuration changes, and that the namespace should be created if it doesn’t exist.

There are multiple ways an application can be specified. I’ll not go through them all, but I am using the following types:

• You can have just a bunch of unordered yaml files in a directory.
• You can have use kustomize, which will happen automatially if there is a file kustomization.yaml in the specified directory
• You can reference a helm chart directly

With kustomize, you can also have a mix of helm charts and yaml resource files, and multiple helm charts if you want to group them. Since I had decided to create my monitoring setup (grafana, prometheus, alertmanager) like that logically, that is what I have used for my grafana application.

The above mentioned traefik application has the following kustomization.yaml:

resources:
  - calicopolicy-external.yaml
  - traefik-role.yaml
  - traefik-oidc.yaml
  - linode-token-sealed.yaml
  - traefik-stateful.yaml
  - internal-whitelist.yaml
  - middleware-proto.yaml
  - noop.yaml
  - middleware-websocket.yaml
  - traefik-external-svc.yaml
  - traefik-external-ipv4.yaml
  - traefik-external-name.yaml
  - traefik-dashboard.yaml
  - traefik-http-to-https-middleware.yaml

helmCharts:
  - name: traefik
    repo: https://traefik.github.io/charts
    releaseName: traefik
    version: 35.4.0
    namespace: traefik-external
    valuesFile: values-external.yaml

I’ll not go through all of it. I order the entries in roughly the order I need them to exist, but since these resources already existed when I created the applications, there’s no real way for me to test this in a non-destructive way, so I’ll leave that one for my DR test some time into the future….

Also, it wasn’t exactly recommended to install over existing helm installations, but if you create all the same resources and use the same helm file etc as you did with helm outside ArgoCD, it works. At least is has done so for me for my whole cluster. You can remove references to the helm installation by deleting the secrets that look like this:

sh.helm.release.v1.traefik.v1   helm.sh/release.v1   1      103d
sh.helm.release.v1.traefik.v2   helm.sh/release.v1   1      60d
sh.helm.release.v1.traefik.v3   helm.sh/release.v1   1      41d
sh.helm.release.v1.traefik.v4   helm.sh/release.v1   1      21d

This doesn’t guarantee there’s no leftover resources, so if you’re migrating, I found it wise to use the same values file and the same version of the helm chart, which made me pretty sure it was the same resources that were created and now is managed by ArgoCD.

Also, to migrate from a non-argoCD to an ArgoCD setup, you should try to be dead sure that you don’t forget anything in this list, but nothing will really break if you do, it’s just that you have stuff in your cluster that you forgot you had, and that is never good.

Secrets

In your cluster, secrets are actually unencrypted, it’s just that access to them is protected. If you want them checked in to a repo – and you probably do – there are ways around this!

The one I use is Sealed Secrets. It’s an operator that can can do two-way symmetric encryption between a Secret and a SealedSecret resource. When you install the operator, you get a set of encryption keys. In the form of secrets. These, you need to handle in a special way, but don’t check them into your repo! If you lose them, you’ll lose the ability to decrypt the SealedSecrets, though, and that’s not good at all….

You’ll create a sealed secret by running a yaml for a secret through the kubeseal tool, which will create a yaml file with the secret encrypted. Now, this can’t be decrypted unless you have the keys from the sealed secret operator. These are considered safe to check into a repo, even an external one, as the encryption is considered strong enough.

In traefik, I have one secret I have sealed, the API key that traefik use towards my linode DNS for ACME DNS validation records. I had my secret in the cluster as linode-token, non-encrypted. I encrypted it like this:

kubectl get -n traefik-external secret linode-token | kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets > linode-token-sealed.yaml

Then I need to install the sealed secret instead of the secret. The operator will immediately decrypted a sealed secret to a secret, and a sealed secret will not overwrite a secret it doesn’t manage, so I need to do:

kubectl delete -n traefik-external secret linode-token
kubectl apply -f linode-token-sealed.yaml

It’ s a good idea to do this outside ArgoCD before submitting the application, as it’s a bit harder and will make the cluster not have the secret installed for a little longer if you don’t.

Applications of Applications

You can also create hierarchies of applications, i.e. an application can itself contain other applications. I have two such parent applications, one for the bootstrap repo and one for the repo that lives in gitea.

The application simply points to <repo>/apps, which in turn contains the yaml files defining the applications in it, and the directories of the resources they define. So my bootstrap repo apps directory contains this:

drwxr-xr-x 2 vegard vegard  17 Jun 24 19:34 argocd
-rw-r--r-- 1 vegard vegard 491 Jun 20 17:27 argocd.yaml
drwxr-xr-x 2 vegard vegard  16 Jun 22 00:49 calico
-rw-r--r-- 1 vegard vegard 498 Jun 21 22:13 calico.yaml
drwxr-xr-x 2 vegard vegard   2 Jun 21 20:02 externaldns
drwxr-xr-x 2 vegard vegard  13 Jun 22 21:28 gitea
-rw-r--r-- 1 vegard vegard 489 Jun 20 17:26 gitea.yaml
-rw-r--r-- 1 vegard vegard 391 Jun 20 14:14 gitea-argocd-app.yaml
drwxr-xr-x 2 vegard vegard   7 Jun 21 22:13 internaldns
-rw-r--r-- 1 vegard vegard 502 Jun 22 17:22 internaldns.yaml
drwxr-xr-x 2 vegard vegard   6 Jun 21 22:13 priorityclasses
-rw-r--r-- 1 vegard vegard 514 Jun 21 22:13 priorityclasses.yaml
drwxr-xr-x 2 vegard vegard   5 Jun 21 22:13 tigera-operator
drwxr-xr-x 2 vegard vegard  21 Jun 23 15:13 traefik
-rw-r--r-- 1 vegard vegard 503 Jun 20 20:38 traefik.yaml
drwxr-xr-x 2 vegard vegard  11 Jun 21 22:13 zfs-storage
-rw-r--r-- 1 vegard vegard 494 Jun 21 22:13 zfs.yaml

As you understand, if the goal is to use this to bootstrap your cluster, you need to consider a whole lot of chicken and egg situations, and it’s probably a whole lot easier to do this from the start than what I did, fitting in ArgoCD way after creating my cluster.

Internally hosted repo

I’ll not really describe installing and configuring gitea, but it wasn’t that hard to do. I have chosen to create one application referencing my gitea-hosted applications, in an apps-of-apps setup:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kubernetesapps
  namespace: argocd
spec:
  project: default
  source:
    repoURL: git@gitea-ssh.example.com:my-org/argocd.git
    targetRevision: main
    path: apps
  destination:
    server: https://kubernetes.default.svc
    namespace: apps
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Both for this and for the bootstrap-repository, I need to provision SSH private keys. You can do this with a secret – or rather a sealed secret, since you want to check it into the repo:

{ 
  "kind": "SealedSecret",
  "apiVersion": "bitnami.com/v1alpha1",
  "metadata": {
    "name": "gitea-repo-creds",
    "namespace": "argocd",
    "creationTimestamp": null
  },
  "spec": {
    "template": {
      "metadata": {
        "name": "gitea-repo-creds",
        "namespace": "argocd",
        "creationTimestamp": null,
        "labels": {
          "argocd.argoproj.io/secret-type": "repo-creds"
        },
        "annotations": {
          "argocd.argoproj.io/credential-repository": "git@gitea-ssh.example.com:"
        }
      },
      "type": "Opaque"
    },
    "encryptedData": {
      "sshPrivateKey": "AgAqxDq7DaXiHk9prR4qZRYEzxRpYysBFMBMf9+r3q0m5Zjn/WeX80WpYmJKc+lVBUuujedGqyy2XKqzEk/BqI2vgpCJDAcwCzhEnDx+zjy1Td/fKR0sSpfZ/FaxJTYkqW0vcQBRmTisGvhqUuSBMO+IEan9m/VtEU11iFW0qJq8hkkL43eAtaWG63dhHgNnVvQ1ZlEOcgxPsrtyRmxT1C2gt7hkX6BxiUBQFF9AhHzhKKy2Y2s3vedoW2drj4380UP3f+TMLHOdF/EKBmg/UMB3TzX1Vt9z5nAnvkua/v/j1hoxD7ufP6AWhK7VNRQp6EkBgw8LnJ3I4qtUXVp1JS9DjwmDrS48/qgSSQniZ6jGscAt0ZnSizdqnVoc9XPrrVb9O7F6m/PmDehBUD9ql6vXKchXTnCXJ9H19cU7/++vKbomW5jwfwuEMiuIYG0lSSkrtg7AM4EN5N5QS5mHDhQTunbwXX/e+3HH1AaiUwCGLPTqTMczXC+JSFfjJTw+CNcKqF9xN1VIBs/zng2trFWn3r/RQjN7ONjRz7w9X3uArdo2krhIdwk3j5VJGHGpQxuBjek3Fo3XvxnKeoSVup2pP7h7RUJuUubWYl8ddc4vvY5Z74MZgkfxNUVP8exRRsKidIE0n1CUjwkKVD3n+4LL5cPzl3Br4EagzVHJCuI8Wf1yLtvpLfLWmqg3ZDATpfsE2bL2oepj8gEUiKb6vrcSuPVOhwXEswupNFfWbOdOSy6lohrqVYfe9BJi4bAGoPV3AI2e6vt6nk3kx+hrJjm0imbflkgT0fYZWYH5vzUTKeXX+v7fGioquF2lokqXCjJ+zrFHSeDFO9+vzuAdSMHBictp2HBqJxUQIgWbP4uQyh+6qTMtdL+C3H5XigIO3QdGCaJ9nXzwbzuaLQXASjs/SqJOufAlz84u3uinxjQi+uLiipt+oFklB2nQR+e4OHy/AKI1y1RIK4Eid1Fhh3x02wEMmlUQRimdxgpt1DAHswx0tAfk0VZWBRrk5kcga0JBWhzjBZoPhoIRfvzpQakTF89VVT0GM2fNe6xHts08MEkgAid+OX784PEVIN9tUBeRcDQqZXI2GRk04pG34z7XXRTwNOu6GVdHoXAP84YbYZtwm0yL45pqAbc/njJk8iJeoc3vsOw3R6lE2Aa+MtyLLW1HiJc5JhkVBmTO2lm0gxqbFkCXv9wFxC6gttbTJozYYk064nDwFocfFMKCmNlAIcR2LKUi9tnf1rUDhojOajRDqqTIXJuPx2e+4+3rbhXn/4hs39nKSE9VQsRSicU3B5tOmMylVCPn2iZoYpzZ509UT69zmOABqisT8TaspPp6xmcaFgceGSd4VEXDG57vRLFkxIU4LziYi5O81IPaaZIva+eOrV5KC1RCFOtopxTzqXvh5aFmpniBl5sz2/MS7PvR6Pn4BcHFBSYVhdUKRDN0PdU368r4XEygNEt/Y6ploljBBszlCOJGFGzjofxsr+UZOvB54sPg46aXG4R0Bakh+mQOspbjZz5wa7g61HR6kJaEruoDW6aI+or3mRUNu4qWeJebZYK1riT06nqMRTaJZpuE4/eKYwNFdSXXg8qwqHCZ7FN0DZ1+2+WjVhQVpsJMSVdvWWP+CGg4ZYPK750Qu++fEpcEcVIDeGCuAxNC8LjRtzONsIy2eTNk4wTw+nPIU8ihwI8GYI2Zgq2dZNTRoDDMm2v7uv2tzZouTySSyx4eTQYoSCs5RLRfdAXFQ3fNv/jqhYgCnTr9uuyWUTOd9ZmtSr3HImRcXRiLMSA44Zpyz6M2IDbfhAg2N+vCasUGfr5pAYNiVS40m32XH1kc8u8tCd02OZ3CjM4c6H+P997mci+Q0WP/PgUs7u+GC/h6gzTLf8V3pO/F1Xi4eLghUbsNuS8UWTjHObJUZPRZrA1YW08t50wu1l5JbELgXWl6uj1BH1SbWPhXICcd+R83txtDcq6VXFpV7416gdiV5KYfAXhuwGI41DYQxhqiM4gYHs1qdf8krgdi739R+EciU/Puebqwfgumn41saEo3yMjdh9/PfqghK7nq/QrRRwUgMJxSs52LUzJcY0M3hu+IG0wOSR4ZSYc4suldm5GFMqEVVEuevFu771hsY75wlV9uwIUWEMag4sCPguPtsC7OM9cJvgIyo8whe6YRYiD0g1lBq6eoGlr1qhavrrwlmEcu2hJMtSarpYI6z6SSB2MCa1uCfkXAZYA5v4tjpg1FQV4Fk5afhPcYzWntV3IvaN5q5vd4Do4qZgwKSQ3RayESaoO7wIAqHfww1xhpWMppM9Ccsq/lhCrCVphOk+p6wtSzKXsw5RSOVGwdR9e18c3rX6HcCDmB+w7fe2hYYCfGRXC6WpDeJ+hTcmihuXA93MHQ8N/ZjBqAxUX/rz6gYKHehPjO55Y6BkOTVyzObhOCbiq6k/c2ascU1OtQtfkWqCzKibNZj/oZ8pR0cnxGNTuBSP2JEj6hHgVrvztM/L/UvJmUMkv6TyJGhcmEilljeGKgym28mMdNuVOEjNKT3fJfynUWREv2Mgy2SkYF1dMg7PJHO4YsVy4M5BKNJLrjD6pic6DmT9cvYasxuKdkuuUnHgADPc/Y2E0FlsqfrrsOndC8Su26i1fvaXkp4+MXZufYaprJyD1kzI5l8bvm7aXvQ8Zpd/A1QmZQtWhdPQt9hB9LG+qIZTqYGLq5xpiFqEZ/jIIP3QqgmIhvHNT6DcSg0AFPrEoOxlwByNqfGr2QaMX/+mi15tzTYglH2bss7TNNPPQRXqpP+At9zXsg2oghmHfbi9Pjp52ev2/WInVM9YD+v1fE70II7AgVrVZa+K9a6C1rmiqWZLY7q90MpFEKJjANq+6xYvpfr2nb2ZtVCfqv5fTqTn6QLFti7ZhJ6o1oLJZh8RUAj+KfaWXdUyZ96rc7vycv1WZfP75D4IZv3af7JFQ0MqCuKduvHItE+qiS3w7Hal9UYXW9zpYLh+ygyUS72/R+tIkmKGy5bZwgVfyK1ih0cvAiVdl0QFc5Ga27OdvwFXQ6iVm2M9z1vvP4qxXTr2PFdlk3X9zyFKXRPNvFUhILEXx2UPv5fAG2qwZKqXrxCmwzXtr6R6x2POI6kBArU5fxkMBzD98Rep2nvoeAp3N7y0pwUmwXJady9fOHEKS7frwDHHG1R4JiDlxE2+xyzoMFMvonIBmPcoSWKxvvq0lw5nrtQscNhwkNyAd8S5JOIBVFwA2b3fwDMobJ4nQiojxF1m8MzOJ1+i+18sWoKKgLDrZEec5YBuo2SPu14D+y7zggb5q8o3mNAEjflWZClijX6CuJ9XhsdL0w8v877YmH1eklmhzBf1M7+NCuNSstUhrSE4oIXWPHsGRcIWxJUY7j/CM9h3rnHYPOD+R1B62hVScEuD4fGZgdSB0KkHoi2ReSE48naY3njGdvEomIcWfVXvmKaQLFri9pVCRl4QHdXruuLgVF5/d1N32cfjDydTAzmD+FMNnCCe3cbgLiKT31xumbxXxG7sfX+0sJD0aVAxTYgQ5nh5slr6AHvy5KzY6R9kzccIk5FeeOjLVRjwRN6If+yFll5+fWPS5EpoUxS8mqv00n1u69MCwdfTMdXiIyehOqWRJfKh4O87WUybzLXeGludeN9T1iObkVt0pqpW1PI4CU6N0aippt8OgeZh8oKoIBR96ApsHuuRGnFaKg326u6OebWpLZpsOwG/wCRmoXhu/h66ezSs9w7Vggn6fAMYqtRx+q/pSXZR1d7VxJ2VwihhUJs0hp8TcZySewH7U3VO/AfqNnU34d2juWAQziVnguF5lcBIzbK1tT0HK1aTEwrETiDpt7+ND21ib1Z6r8GHW5oRjCTwn4XqNd2uzoS1rIVb6G/oDhzfwOf9fZifmII3Ihbm9QdPZ3aZiBkVqlLL3CEBEeIEfdDBUr3ecvp45VqLU+Qh3S6KIJbQ060840tyrTZFhhfqp4meU8xhilR0pA+NhiigkVYEUiKlJ3x06of/GFOLZIpCWXzkuhyUfFRVb9BMv8o14fhhcff8oJYpD8A8OcIG0hJ+FA4Y/JfZBoOaR9yMHMFlB6sKf8GcmHv4K4qupIgkYdqyyKKN12AbvOxvqxGBsptwH0UmbfoeMwHAvI6jQ9omoFmOlLg6GCdSWjUqxBq3RjYNPYkebJ",
      "url": "AgAvkdLLt9ozg0A+oZ8J/aKeyBlbbOivkjeRdr3+ypCt7lzyjkrY3VjoV97RWPa6e4ORQW9tmJGoHweopvEDlgOZIsO+BJBx764lWncB5u/25ThzFgZ1/SYQL0tPONr6+UghZKCI+vqkHDbUEf9h2u3IzzWBlNFS7ekuzMNikJPY7Fxthg1n4KgHg9g6OAnQIEsu2J8jmbWVnTGSUshYJ27RiCCnaaHLsTyytApwuf6LWk9GRnH8sdwlBTOW/v3Rl3aoXfJgybI5r6QZ7K3okBkmEu6j5JYoAMi97BlWMWLqi5mrlI4Zm8R+g017fo5Xxw6KY+XA6L+exYYAcdvNJatNC3quizSGvZCHA6cv0C8KKuH70rKGkfwvO4b4r+ZJzIRBxe6P3hvnyz6RO6GpdZC92nh2Qo2Ja6Kjnhj0jQS02WdEIdrESjTFQMgFHfg+yxTEd0yuuy34nQ9t9e98xIkehoxu/Wjuo5HVfK38Usu6qBv68GavmFDB4tUjaDin38QdEAF6U9rHi3B6FKbtVyNLopNzHXKWi3ULSK6taOYJbXv08TXWkTesqw2uD2uqZ0XhxF/D3JwZwIMwBCnceVQzCRCrCBl4NuKLM4iB5l+clXouYdP7BlT0K+CYQhSmUW1ITBrys3n2ibn4dvgo2/FSxRCc4cL4cL6NliJVT4vIAkTWeFWFpzWqslKl5AA1Vtq6ylE4xAEX4jLKfknJH/pFI19dG8MrA/9yvZ10fGg5ZFg8N4v8uIHAnAiOVltcMv0hIoI="
    }
  }
}

The URL here is the URL to the repository again. ArgoCD will find this key when trying to access this repo. I of course need to put the public key in the repo and make sure I get at least read access to the repo with this key, but this is a bit outside the scope of this blog post.

Using and managing ArgoCD

Once you have everything in argoCD, you should do all your changes through checking in changes to the repositories. In fact, if you try to change the resources manually, ArgoCD will detect the difference and change it back!

You can also do some operations (like syncronization, restarting applications, …) through the ArgoCD applicaton – or the argocd command line application, which is handy for command line freaks like me.

The UI looks like this:

You can look at single applications. Here’s an excerpt from traefik.

You can look at and resync and operate on single element, although changes are best done through changing the repository. Triggering a sync is handy, it will be quicker than waiting for ArgoCD to discover that it needs to be resynced. It can also be handy to do restarts of applicatons from here.

Summary

Keeping your code in ArgoCD or a similar tool is a good idea when you’re growing, and crucial to make your setup reproducible. That’s always something to work towards. To me, it gave me a much cleaner and much more structured configuration that the just a bunch of yaml files setup I had before this.