DR activation is about injecting state into the ArgoCD applications. One state I am already injecting globally is the cluster I am on, via the in-cluster secret, and based on that I set the env variable to either prod or DR. This is done through the cluster generator. I basically set the cluster based on a label selector on this secret.

There is really nothing that stops me from using another label selector on the same secret and thus getting a new cluster with basically a new set of properties. But even better for my use case is to combine these two with the merge generator, and get properties based on both labels, where the last one defined will override properties from the first one.

I decided to use completedr=true as the label selector for activating complete DR. But first, as part of this I learnt a tiny bit about the go templating in ArgoCD, and was able to simplify my current setup a bit. Previously, I was using a matrix generator to combine a cluster generator with a list generator. Since my matrix doesn’t have many dimensions at all, it’s basically just a list of values I have in each cluster, I could simplify it like this:

Before, I had:

  generators:
    - matrix:
        generators:
          - clusters:
              selector:
                matchLabels:
                  env: prod
          - list:
              elements:
                - appName: wordpress-prod
                  appPath: application/wordpress/prod
                  namespace: wordpress
                  enabled: "true"
                  repoURL:  "git@gitea-ssh.engen.priv.no:klauvsteinen/argocd.git"
                  env: prod

    - matrix:
        generators:
          - clusters:
              selector:
                matchLabels:
                  env: dr
          - list:
              elements:
                - appName: wordpress-dr
                  appPath: application/wordpress/dr
                  namespace: wordpress
                  enabled: "false"
                  repoURL: "git@gitea-ssh-dr.engen.priv.no:klauvsteinen/argocd.git"
                  env: dr

Then, in the application template I could use .appName, .appPath etc directly. But I simplified:

 generators:
    - clusters:
        selector:
          matchLabels:
            env: prod
        values:
          appName: wordpress-prod
          appPath: application/wordpress/prod
          namespace: wordpress
          enabled: "true"
          repoURL:  "git@gitea-ssh.engen.priv.no:klauvsteinen/argocd.git"
          env: prod
    - clusters:
        selector:
          matchLabels:
            env: dr
        values:
          appName: wordpress-dr
          appPath: application/wordpress/dr
          namespace: wordpress
          enabled: "false"
          repoURL: "git@gitea-ssh-dr.engen.priv.no:klauvsteinen/argocd.git"
          env: dr

So, I only have clusters and set variables under values. THis means I need to prefix the reference with .values, thus .values.appName, .values.appPath. It’s still simpler, and made the next step much simpler.

What I want, is to flip the DR enabled value to true whenever I set the label completedr=true on the cluster secret.

This, I do by converting the second cluster generator in this setup to a merge between two cluster generators, like this:

  generators:
    - clusters:
        selector:
          matchLabels:
            env: prod
        values:
          appName: wordpress-prod
          appPath: application/wordpress/prod
          namespace: wordpress
          enabled: "true"
          repoURL:  "git@gitea-ssh.engen.priv.no:klauvsteinen/argocd.git"
          env: prod
    - merge:
        mergeKeys: [ "name" ]
        generators:
          - clusters:
              selector:
                matchLabels:
                  env: dr
              values:
                appName: wordpress-dr
                appPath: application/wordpress/dr
                namespace: wordpress
                enabled: "false"
                repoURL: "git@gitea-ssh-dr.engen.priv.no:klauvsteinen/argocd.git"
                env: dr
          - clusters:
              selector:
                matchLabels:
                  completedr: "true"
              values:
                enabled: "true"

With this setup, I get values from both cluster generators, where the last one overrides the first one, and I get the values from my DR cluster, except I override the enabled setting.

If I don’t set the completedr label, the value will be fetched from the first cluster in the merge generator, so I can activate it both there by flipping enabled to true, or if I add the label, it will also be set to true.

Of course, I need to do this on all my workloads I want DR for, but then I can just set this label and activate DR for my whole workload. Or, possibly only a selected set, I can totally decide that I can have a possibility to do DR on a single application selectively, but drop activating it if enabling the complete DR. The reason for this can for example be performance, if your DR node isn’t powerful enough to run your full application portfolio.

With this, my DR setup is more or less complete, with simple activation procedures. There’s still some rough edges around data synchronization back to prod should I need that, but it will more or less be a reverse DR activation, where I create DR volumes in production based on the backups from DR. Simple but a bit tedious, so I will likely be on the lookout for neater solutions.

Oh btw, if you’re confused about these generators in ArgoCD, you’re probably not alone. They can be a bit confusing. But it’s just a way of generating data fields into a template that generates ArgoCD applications, with multiple sources of truths. There’s more generators than I have used, if you have different needs than me.

With this, I’m going to leave the DR track for a bit, and focus on stability. I’ve been hunting for features to test for around half a year now, jumping onto the next change whenever the previous one is done, so it’s time for some serious look at solving rough edges in my cluster. At times, I have had I/O running haywire, leading to serious stability issues. But don’t worry. It will likely be worth a blog post soon. And hopefully, you’re going to be able to read it too, before my cluster goes poof again.