Recently, a Kubernetes version upgrade led to issues in my “prod” lab environment. After troubleshooting and restoring the cluster to a healthy state. By healthy state meaning, I needed to restore from backup.

This restore was the perfect time to reconfigure the cluster setup to leverage ArgoCD’s App of Apps model.

Why? such a big change?

If you are supporting a Kubernetes cluster, adding IaC (Infrastructure as Code) code to manage deployment lifecycles can be highly beneficial. Using IaC opens up CI (Continious Integration) options.

The initial setup may take longer, however the manability improves greatly.

While I have been using ArgoCD to handle Continuous Delivery (CD), I will now be using App of Apps, which allows management of app life cycles and apply similar configurations to multiple apps using code or manifests.

To start, I implemented this approach in my “dev” lab environment and then apply changes in the “prod” lab environment.

My dev environment consists of three Raspberry Pi devices while the prod environment is hosted in the cloud using ARM compute instances.

Although some familiarity with ArgoCD is recommended for this overview, don’t worry if you’re not familiar with Helm or ArgoCD - you can check out the resources below.

Take advantage of the hands-on, in-browser labs and engaging lectures in these KodeKloud courses (Affiliate links) to enhance your Kubernetes knowledge:

Why App of Apps?

You can create an app that creates other apps, which in turn can create other apps. This allows you to declaratively manage a group of apps that can be deployed and configured in concert. - ArgoCD docs

In the screenshot provided, you can see how the app of apps is structured in the ArgoCD UI.

App life cycles can be managed in one place and similar configurations can be applied to the apps, much like Helm’s values.yaml.

The ArgoCD Application configurations are stored in a single Git repo, and changes can be synced to the cluster automatically after pulling from the latest commit.

Using Git as the source of truth allows this configuration to be applied to potentially any environment. More info on this can be found in Argo’s Docs

App of Apps Git structure

If you’re familiar with Helm, the Git layout for App of Apps will make sense as it’s a typical structure you’d see in Helm.

The templates directory is where each ArgoCD application manifest will be stored. (Please note other kubernetes resources can be inside the templates directory. Like secrets)

├── Chart.yaml
├── templates
│   ├── argocd.yaml
│   ├── vault.yaml
│   └── heimdall.yaml
└── values.yaml

Here’s a manifest to install HashiCorp Vault (Open source secret management software):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vault
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: argocd
    server: {{ .Values.spec.destination.server }}
  project: default
source:
  repoURL: https://helm.releases.hashicorp.com
  targetRevision: 0.24.0
  chart: vault

Like Helm, you can use variables and reference them from values.yaml

spec:
  destination:
    server: https://kubernetes.default.svc

Sync the root application via the CLI or UI. The parent “app” will appear in sync, and based on whatever sync policy you’ve chosen the child apps will sync or be ready to sync.

$ argocd app create ROOTAPPNAME \
    --dest-namespace argocd \
    --dest-server https://kubernetes.default.svc \
    --repo <your repo url include .git> \
    --path <path in the git repo>  
$ argocd app sync apps

Utilizing values.yaml you can make your changes in one place. If you’re updating repositiories, destination servers, and all other applicational parameters can take place there.

For more information read “Cluster Bootstrapping doc”

ArgoCD Application Manifests

App of Apps is a useful feature that allows you to group multiple applications together. I personally use it to group all the Kubernetes utility apps in one group, and self-hosted services in another App of Apps group.

As far as I know, there is no limit to the number of child apps that can be included in an App of Apps group. Although the image above shows only three manifests in the templates directory, I have since added a fourth app.

Final notes

ArgoCD provides a application.yaml file with extensive comments, which can be found on the argo-cd github repository: https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/application.yaml. It’s advisable to keep this file handy while developing your applications, or you can create a custom template with Helm.

Although the initial setup of this system can be time-consuming, it ultimately makes managing the applications easier, making the investment worth it.

In conclusion, I have successfully implemented the App of Apps approach as described in the blog post and I have included two screenshots below to showcase the setup.

Dev Environment:

Prod Environment: