Tier-1

BigConfig and Tier 1 Workflow

Core Problem

The primary problem big-config addresses is the significant repetition encountered when curating configuration files (e.g., for Terraform, Kubernetes, Ansible). This repetition arises from the need to express similar logic (such as importing data structures or iterating) within the specific configuration language of each tool.

BigConfig Solution: A “Zero-Cost” Build Step

• Concept: BigConfig introduces a build step for configuration files, drawing inspiration from software-development workflows (e.g., JavaScript projects that compile TypeScript).

• Process:

  1. Users maintain configuration source files (often Clojure code) in a src/ directory.
  2. A build step transforms these sources into output files in a dist/ directory, using the syntax required by the target tool (Terraform, Kubernetes, etc.).

• Benefits:

  • Reduced repetition: Common logic is handled in Clojure rather than duplicated in each tool’s DSL.
  • Abstraction: Clojure can manipulate data structures, making configuration more dynamic and less verbose.
  • Templating: Two template engines are supported—deps-new (via build.tools) and Selmer (similar to Jinja)—to generate config dynamically.
  • “Zero-Cost” ideal: The build step aims to feel invisible, letting developers work as if editing the final files directly.

Tier 1 Workflow Language

• Purpose: A small Domain-Specific Language (DSL) for composing command-line steps—an alternative to Makefiles.

• Motivation: Makefiles were found insufficient for dynamic workflows.

• Key features:

  • CLI tool: babashka provides commands to define and run workflows.

  • Composed steps:

    • build — run the BigConfig build step
    • exec — execute any command (e.g., ansible-playbook, terraform plan) inside dist/
    • git-check — verify the local branch isn’t behind origin
    • git-push — push changes
    • lock — acquire a pessimistic lock
    • unlock-any — release the lock

  • Sequential execution: Steps run in order; a failure stops the workflow.

  • Integration with BigConfig: The build step is fundamental, enabling a change → build → run loop.

  • “Change and run” ideal: Embedding the build step minimizes perceived overhead.

Coordination and Locking

• Problem: Multiple developers modifying shared resources (e.g., an AWS account) need coordination to prevent conflicts. Tools like Atlantis handle this for Terraform.

• BigConfig / Tier 1 solution:

  • Uses Git tags as an exclusive locking primitive—generic across tools.
  • lock / unlock-any steps manage these locks within a workflow.
  • Provides a more general alternative to specialized services like Atlantis.

Key Takeaways

• BigConfig adds a powerful abstraction layer by leveraging Clojure and a build step to reduce repetition and add dynamic logic.
• The Tier 1 Workflow language supplies a flexible CLI for defining and executing configuration workflows—integrating the BigConfig build step.
• Combined, they offer a generic Git-based coordination mechanism for shared resources, potentially replacing specialized tools such as Atlantis.
• The overall goal is to improve developer experience by simplifying both the creation and execution of complex infrastructure configurations.
• Compared to atlantis, big-config enables a faster inner loop. Only two accounts are needed, prod and dev. The lock step enables developers and CI to share the same AWS account for development and integration. Refactoring the code that generates the configuration files is trivial because the dist dir is committed and we can track with git any change made by mistake in it.
• Compared to cdk, BigConfig supports only clojure and tofu. The problem of generating json files should not be blown out of proportion.

Tier-1 workflow language deep dive

The tier-1 workflow language is a simple DSL that allows developer to compose different steps into a workflow to make the build step a zero-cost operation. Other steps available in the tier-1 workflow language are:

• Acquire/release the lock
• Check if the working directory is clean and if we have pulled all commits from origin
• Push the changes inside a transaction

These primitives are necessary to enable multiple developers to work at the same time on the same infrastructure without any further coordination.

Example

bb build lock git-check tofu:init tofu:apply:-auto-approve git-push unlock-any -- alpha prod

is a tier-1 workflow defined in the command line using big-config and invoked using babashka. The build step will use deps-new to generate the alpha module using the prod profile. The lock step will acquire a lock to make sure that we are the only one running (same capability of atlantis). The git-check step will make sure that our working directory is clean and not behind origin. The tofu:init step will run tofu init in the target-dir. The tofu:apply:-auto-approve step will run tofu apply -auto-approve in the target-dir. The git-push step will push our commits. The unlock-any step will release the lock.