This week, I implemented a CI/CD pipeline to deploy Podman container quadlets to RHEL 9.5 servers. This approach combines the power of containers with the familiarity of systemd, creating a solution that’s both powerful and accessible. In this post, I’ll walk through how to set up this system and highlight key insights from the implementation process.

What Are Podman Quadlets?

Podman is a daemonless container engine that serves as a lightweight alternative to Docker, particularly well-suited for RHEL environments. Quadlets are a relatively new feature that allow you to define containers as Systemd unit files, simplifying container management through familiar Systemd commands.

For example, instead of using podman commands, you can simply use standard systemd commands:

systemctl --user start app-container
systemctl --user status app-container

This creates a more intuitive approach for sysadmins who are already comfortable with systemd.

Why Use Podman and Quadlets?

Several factors make Podman with Quadlets an excellent choice for container deployments on RedHat Servers:

  1. Daemonless architecture: Podman’s fork-exec model enhances security and uses fewer resources than Docker’s daemon-based approach
  2. Root-less containers: Run containers without root privileges, significantly improving security posture
  3. Red Hat support: First-class support in RHEL 9.5 with regular updates and long-term support
  4. Systemd integration: Leverage existing systemd knowledge for container management
  5. Portability: Quadlet files are easy to version control and deploy across environments
  6. SELinux compatibility: Better integration with SELinux than other container runtimes

Setting Up the CI/CD Pipeline

The CI/CD pipeline automates the entire process from building container images to deploying them as Quadlets on RHEL servers. This implementation uses GitLab CI/CD and Ansible to create a fully automated workflow.

The GitLab CI/CD Pipeline

The pipeline consists of three primary stages:

  1. Build: Creates container images and pushes them to a registry (Not shown below for brevity)
  2. Test: Runs automated tests against the built images (Not shown below for brevity)
  3. Deploy: Uses Ansible to deploy containers as Quadlets on target servers

Here’s a simplified version of the .gitlab-ci.yml:

# Template for the app build jobs

.ansible-deploy-setup:
  stage: deploy
  when: manual
  variables:
    ANSIBLE_HOST_KEY_CHECKING: False
    ANSIBLE_FORCE_COLOR: True
  before_script:
    - eval $(ssh-agent -s)
    - chmod 400 "$ANSIBLE_SSH_PRIVATE_KEY" 
    - ssh-add "$ANSIBLE_SSH_PRIVATE_KEY" 

# Deploy job using Ansible to provision Podman Quadlets
deploy-containers:
  extends: [.ansible-deploy-setup]
  script:
    - cd devops/ansible
    - > 
      ansible-playbook 
      -u app-deploy 
      -i inventory playbooks/deploy-containers.yaml 
      -e "CI_REGISTRY=${CI_REGISTRY} 
      CI_REGISTRY_USER=app-deploy
      CI_REGISTRY_PASSWORD=${REGISTRY_PASSWORD} 
      app_image=${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG}-latest
      # Additional variables passed to Ansible
      "
      --private-key $ANSIBLE_SSH_PRIVATE_KEY

This pipeline builds the application container, pushes it to the registry, and triggers the Ansible playbook for deployment as a Quadlet.

The Ansible Playbook

The Ansible playbook handles the actual deployment of containers as Quadlets. Key responsibilities include:

  1. Setting up required directories
  2. Authenticating with the container registry
  3. Creating Podman networks
  4. Generating Quadlet files for each container
  5. Restarting containers when configuration changes

Here’s a simplified example:

- name: Setup App Containers
  hosts: all
  vars:
    username: app-deploy
    quadlet_dir: "/home/{{ username }}/.config/containers/systemd"
  tasks: 
    - name: Create App Network
      containers.podman.podman_network:
        name: appnet
        state: quadlet
        quadlet_dir: "/home/{{ username }}/.config/containers/systemd"

    - name: Create Quadlets for the App Deployment
      containers.podman.podman_container:
        name: "{{ item.name}}-container"
        image: "{{ item.image }}"
        state: quadlet
        quadlet_dir: "{{ quadlet_dir }}"
        ports: "{{ item.ports | default(omit) }}"
        env: "{{ item.env | default(omit) }}"
        volume: "{{ item.volumes | default(omit) }}"
        network: appnet.network
        quadlet_options:
          - "AutoUpdate=registry"
          - "Pull=newer"
          - |
            [Install]
            WantedBy=default.target            
      loop:
        - name: database
          image: "docker.io/library/postgres:16"
          env:
            POSTGRES_PASSWORD: "{{ DB_PASSWORD }}"
            POSTGRES_USER: "{{ DB_USER }}"
          volumes:
            - "db-data:/var/lib/postgresql/data:Z,U"
        - name: app-server
          image: "{{ app_image }}"
          env_file: "/home/{{ username }}/.env"
          ports: 8000:3000

This playbook leverages the Podman Ansible collection to create Quadlet files, which define how containers run as systemd units. The magic is in the loop section. The syntax in the loop section may remind you of Docker Compose. This snippet could be useful in migrating to Podman Quadlets from Docker Compose.

Understanding Generated Quadlet Files

After running the Ansible playbook, Quadlet files are created in the .config/containers/systemd directory. Here’s an example:

[Unit]
Description=Podman app-server-container container

[Container]
Image=registry.example.com/my-project/app:1.0.0
PublishPort=8000:3000
EnvFile=/home/app-deploy/.env
Network=appnet.network
AutoUpdate=registry
Pull=newer

[Service]
Restart=always

[Install]
WantedBy=default.target

This configuration creates a container that:

  • Runs the specified image
  • Publishes port 8000 to 3000
  • Uses environment variables from a file
  • Connects to the application network
  • Automatically updates when a newer image is available
  • Restarts on failure
  • Starts automatically with the user’s session

Viewing Container Status with Systemd

One major advantage of Quadlets is the ability to use familiar systemd commands to check container status:

[app-deploy@app-server ~]$ systemctl --user status app-server-container
● app-server-container.service - Podman app-server-container container
     Loaded: loaded (/home/app-deploy/.config/containers/systemd/app-server-container.container; generated)
     Active: active (running) since Thu 2024-07-09 14:15:47 EDT; 1 day ago
       Docs: man:podman-systemd.unit(5)
   Main PID: 1826 (conmon)
      Tasks: 16 (limit: 48921)
     Memory: 246.2M
        CPU: 11min 32.337s
     CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/app-server-container.service
             ├─container
             │ └─1829 app server process
             └─runtime
               ├─1763 rootlessport
               ├─1789 rootlessport-child
               └─1826 /usr/bin/conmon --api-version 1 -c ...

This familiar output format helps sysadmins understand container status even without deep container expertise.

Key Technical Advantages

This CI/CD pipeline with Podman Quadlets offers significant technical benefits:

  1. GitLab CI/CD Integration: Streamlined container build and deployment process
  2. Ansible Automation: Infrastructure-as-code approach to container management
  3. Security-Enhanced Containers: Rootless containers with SELinux integration
  4. Version-Controlled Configuration: All container definitions in Git
  5. Simplified Operations: Standard systemd commands for management
  6. Resource Efficiency: Podman’s daemonless architecture minimizes overhead
  7. Cross-Environment Consistency: Works across development, testing, and production
  8. Auto-Healing Capabilities: Automatic container restarts and updates

Important Technical Considerations

When implementing this solution, several technical details require attention:

User Permissions and Lingering

For user services to start automatically on boot, enable lingering for the user:

# As root
loginctl enable-linger app-deploy

This ensures that user services start even without active login sessions.

SELinux Context for Volume Mounts

When mounting volumes, use the :Z,U suffix to set correct SELinux context:

volumes:
  - "db-data:/var/lib/postgresql/data:Z,U"

The Z flag indicates a private unshared label, while U preserves ownership.

Container Dependencies

For containers that depend on others, use the After directive in Quadlet files:

[Unit]
Description=Podman app-server-container container
After=database-container.service

This ensures proper startup order for interdependent services.

Environment Variables Management

For secure environment variable handling:

  1. Store sensitive variables in GitLab CI/CD variables or Ansible Vault
  2. Pass them to Ansible during deployment
  3. Write them to a .env file on the target server
  4. Reference the .env file in the Quadlet configuration

This approach keeps sensitive information out of Git repositories.

Monitoring and Logging

Access container logs through the standard systemd journal:

# View logs for a specific container
journalctl --user -u app-server-container.service

# Follow logs in real-time
journalctl --user -f -u app-server-container.service

This integrates with existing monitoring tools and log aggregation systems.

Horizontal Scaling

For applications requiring horizontal scaling, deploy multiple containers:

- name: Create Quadlets for the Multiple Containers
  containers.podman.podman_container:
    name: "worker-{{ item }}"
    image: "{{ worker_image }}"
    env_file: "/home/{{ username }}/.env"
    state: quadlet
    quadlet_dir: "{{ quadlet_dir }}"
    network: appnet.network
  loop:
    - worker1
    - worker2
    - worker3
    - worker4

Conclusion

Deploying Podman container quadlets with GitLab CI/CD on RHEL 9.5 creates a powerful but accessible container orchestration solution. This approach combines modern containerization with familiar systemd management, making it ideal for organizations transitioning to containerized workloads.

The automated CI/CD pipeline ensures consistent deployments, while Quadlet files simplify day-to-day operations. For environments running RHEL that need container capabilities without the complexity of Kubernetes, this solution offers an excellent middle ground.

Whether you’re a DevOps engineer, system administrator, or developer, this approach provides a solid foundation for containerized applications on RHEL 9.5 with built-in automation, security, and scalability.

Additional Resources