Bitnami, which was once regard as a readily available and reliable source of containers and helm chart, was recently bought by Broadcom.

Broadcom has made the decision to host their containers and Helm chart behind a subscription paywall, and no longer provide a helm chart repository or their full docker image catalog for free.

While I am in no way opposed to paying for services I use, their starting fee of $50.000 per year was a little bit beyond my home tinkering budget, so I needed to make some changes…

While things would not immediately break, and they will move over their current batch of charts and images to a free «legacy» repository without promises of maintaining it, going that route would effectively be running abandonware and acquiring technical debt. Technical debt is when you know you need to do something, you just haven’t bothered to do so, because at the moment things run fine.

So, I could buy myself some time buy changing to the legacy repo, or I could decide to not have that technical debt and just migrate away from bitnami.

I chose the latter, because technical debt tends to not get any easier to fix over time. Often you build on it, and you end up with even more technical debt.

I started off scanning the argocd repo for bitnami charts. There were a few:

• Keycloak
• Redis
• Some explicitly defined postgresql charts
• Some implicit postgresql helm charts.
• Minio (my S3-compatible store I use mainly for logs)
• A mariadb helm chart in my bookstack
• My grafana-operator helm chart was for some reason from bitnami

PostgreSQL

I started off in the order of importance, and decided to get rid of bitnami postgresql helm chart. Fortunately, I do have a few DR (which I tend to also use as test) applications that had PostgreSQL, so I could start in non-active production.

A lot of the helm charts I am using have a need for a database. PostgreSQL helm chart conveniently is easily embeddable inside other helm charts. Often, all you need to do, is:

postgresql:
  enabled: true

…and off it goes, installing a postgresql for you, fully embedded into the application you are really installing. I, as many others likely, was tempted by the convenience.

Cloud-Native PostgreSQL

The remedy is to install PostgreSQL separately by other means, and point the application towards that PostgrSQL instead. Enter https://cloudnative-pg.io/, which is an operator that manages PostgreSQL for you. It can set up HA solutions, and even help you creating a DR setup. So I decided to go for it.

CNPG is easily installed with helm (no, not a bitnami helm chart), so I created an ArgoCD Application for it:

kind: Application
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "4"

  name: cnpg
  namespace: argocd
spec:
  project: default
  source:
    repoURL: git@github.com:vegardengen/kubernetes-bootstrap.git
    targetRevision: main
    path: apps/cnpg
  destination:
    server: https://kubernetes.default.svc
    namespace: cnpg-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

In apps/cnpg, I have

total 10
-rw-r--r-- 1 vegard vegard 949 Aug 22 20:36 calicopolicy.yaml
-rw-r--r-- 1 vegard vegard 326 Aug 22 20:36 kustomization.yaml
-rw-r--r-- 1 vegard vegard  62 Aug 22 20:36 namespace.yaml
-rw-r--r-- 1 vegard vegard  61 Aug 22 20:36 values.yaml

And kustomization.yaml simply contains:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: cnpg-system

resources:
  - namespace.yaml
  - calicopolicy.yaml

helmCharts:
  - name: cloudnative-pg
    repo: https://cloudnative-pg.github.io/charts
    version: 0.26.0
    releaseName: cnpg
    namespace: cnpg-system
    valuesFile: values.yaml

I’ll spare you a lot of the details here, once it runs, I can create a PostGRESQL instance by basically creating a resource

kind: Cluster
metadata:
  name: keycloak-db
  namespace: keycloak
  annotations:
    # Avoid first-sync dry-run errors in Argo CD before CRDs are present
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
    # (Optional) apply DB slightly before your app; adjust if you like
    argocd.argoproj.io/sync-wave: "1"
spec:
  # Single-instance (adjust to >1 if you later want HA)
  instances: 1

  # Pin a PG image version you want to run
  imageName: ghcr.io/cloudnative-pg/postgresql:16

  # Storage: adjust class/size to your environment
  storage:
    size: 10Gi
    storageClass: zfs-storage-znvm


  # Set the postgres superuser password from a basic-auth secret
  enableSuperuserAccess: true
  superuserSecret:
    name: keycloak-superuser   # type: kubernetes.io/basic-auth (username/password)

  # Bootstrap the application database and owner
  bootstrap:
    initdb:
      database: keycloak       # DB name
      owner: keycloak          # role name
      secret:
        name: keycloak-app     # type: kubernetes.io/basic-auth (username/password)
  priorityClassName: normal-priority

I still need to provision the secrets first, of course, which should (since I intend to migrate data) contain the same secrets as the old installation. The de-facto standard seems to be to have two secrets, one -app and one -sysasdmin, where app is for the application password and sysadmin for the postgres password. So, I put these in sealed secrets in the applications where I need postgresql.

If you like me have a strict network policy, you also need to allow CNPG to talk to the postgresql to manage it:

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: allow-cnpg-operator-to-keycloak-db
  namespace: keycloak
spec:
  # If you use Calico "order", put a small number so it evaluates before any broad deny
  order: 100
  selector: 'cnpg.io/cluster in {"keycloak-db-dr", "keycloak-db"}'
  types:
    - Ingress
  ingress:
    - action: Allow
      source:
        namespaceSelector: 'kubernetes.io/metadata.name == "cnpg-system"'
      protocol: TCP
      destination:
        ports: [8000, 9187]

That done, we get a fully provisioned keycloak postgresql database, which I wire into keycloak like this:

env:
- name: KC_DB
  value: postgres
- name: KC_DB_URL
  value: jdbc:postgresql://keycloak-db-rw.keycloak.svc.cluster.local:5432/keycloak
- name: KC_DB_USERNAME
  valueFrom: { secretKeyRef: { name: keycloak-app, key: username } }
- name: KC_DB_PASSWORD
  valueFrom: { secretKeyRef: { name: keycloak-app, key: password } }

For other applications with an embedded postgresql chart, the process is pretty similar.

Now, before doing that last change, you might want to copy the data from the old to the new database:

kubectl exec -it -n keycloak keycloak-postgresql-0 -- sh

cd /tmp
pg_dump -U postgres -h keycloak-postgresql -d bitnami_keycloak > keycloak.sql
psql -U postgres -h keycloak-db-rw -d postgres
    CREATE ROLE bn_keycloak LOGIN;
    \e
psql -U postgres -h keycloak-db-rw -d keycloak < keycloak.sql
psql -U postgres -h keycloak-db-rw -d postgres
     REASSIGN OWNED BY bn_keycloak TO keycloak;
     DROP OWNED BY bn_keycloak;
     DROP ROLE bn_keycloak;

You’ll be asked for the passwords a few time. I like to just cut&paste them from portainer, where I can find them in clear text (if I have access to portainer, of course…)

A bit of this was to get rid of the bitnami naming, which probably wasn’t strictly necessary, but it might be confusing down the road without, so I wanted just plain naming.

Having done this, we’re ready to fire up keycloak, and everything should just work as before.

Keycloak

Keycloak itself was a bitnami helm chart. I decided to go with plain yaml for this. Often, it makes it more maintainable.

I have a custom spi in keycloak to allow me to trust a device, effectively turning off two-factor from that computer until further notice. I sometimes like to do that, because it still gives me the protection against external intruders, while letting my login flow be a bit easier. For this reason, I decided to roll my own image. I had resorted to all kinds of tricks that the bitnami helm chart supported to get it into keycloak, but at the end of the day, especially if you have a build pipeline for container images anyways (and of course I do!), building your own image is easy enough. It also allows me to do somethings pre-built into the image that keycloak else have to do on startup, so there is some other benefits to it.

Other than that, it was pretty straightforward:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  namespace: keycloak
spec:
  replicas: 2
  minReadySeconds: 10
  selector:
    matchLabels: { app: keycloak }
  template:
    metadata:
      labels: { app: keycloak }
      annotations:
        spi-restart-trigger: "v1"
    spec:
      priorityClassName: normal-plus-priority

      containers:
        - name: keycloak
          image: registry.engen.priv.no/keycloak:26.3.3
          env:
            - name: KC_DB
              value: postgres
            - name: KC_DB_URL
              value: jdbc:postgresql://keycloak-db-rw.keycloak.svc.cluster.local:5432/keycloak
            - name: KC_HTTP_ENABLED
              value: "true"
            - name: KC_DB_USERNAME
              valueFrom: { secretKeyRef: { name: keycloak-app, key: username } }
            - name: KC_DB_PASSWORD
              valueFrom: { secretKeyRef: { name: keycloak-app, key: password } }
            - name: KC_PROXY
              value: edge
            - name: KC_PROXY_HEADERS
              value: xforwarded
            - name: KC_HOSTNAME_URL
              value: https://keycloak.engen.priv.no
            - name: KC_HOSTNAME_ADMIN_URL
              value: https://keycloak.engen.priv.no
            - name: KC_HOSTNAME_STRICT
              value: "false"
            - name: CUSTOM_SPI_ENABLED
              value: "true"
          ports:
            - { name: http,    containerPort: 8080 }
            - { name: metrics, containerPort: 9000 }
          startupProbe:
            httpGet: { path: /health/ready, port: 9000 }
            initialDelaySeconds: 10
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 60
            
          readinessProbe:
            httpGet: { path: /health/ready, port: 9000 }
            initialDelaySeconds: 5
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 12
          
          livenessProbe:
            httpGet: { path: /health/live,  port: 9000 }
            initialDelaySeconds: 90
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 6

And of course I need also service, ingressroutes etc to expose it.

…and more

These were just examples. But I am not sorry, I have switched out my bitnami-provided postgresql with something more kubernetes-like, and I have in many cases simplified by either going to yaml or to switch to official helm charts. There was also an excellent mariadb-operator I could switch to for my mariadb databases. Both of them (postgres and mariadb-operator) supports managing databases via CRs, and conveniently also makes it easy to configure backups etc. Backup, however, I’ll cover in a future blog post.

And I avoided gathering technical debt, something that one should try hard to avoid.