workflow

Note

big-config.workflow has replaced big-config.step.

big-config.workflow

The goal of the BigConfig Workflow is to enable independent development of automation units while providing a structured way to compose them into complex pipelines.

Workflow Types

• tool-workflow: The fundamental unit. It renders templates and executes CLI tools (e.g., Terraform/OpenTofu, Ansible).
• comp-workflow: A high-level orchestrator that sequences multiple tool-workflows to create a unified lifecycle (e.g., create, delete).

Usage Syntax

BigConfig Workflow can be used as a library or as a CLI using Babashka.

# Execute a tool workflow directly
bb <tool-workflow> <step|cmd>+ [-- <raw-command>]

# Execute a composite workflow
bb <comp-workflow> <step>+

Examples

# Individual development/testing
bb tool-wf-a render tofu:init -- tofu apply -auto-approve
bb tool-wf-b render ansible-playbook:main.yml

# Orchestrated execution
bb comp-wf-c create

In this example, comp-wf-c composes tool-wf-a and tool-wf-b. Development and debugging happen within the individual tool workflows, while the composite workflow manages the sequence.

Available Steps

tool-workflow

Step	Description
render	Generate the configuration files.
git-check	Verifies the working directory is clean and synced with origin.
git-push	Pushes local commits to the remote repository.
lock	Acquires an execution lock.
unlock-any	Force-releases the lock, regardless of the current owner.
exec	Executes commands provided.

comp-workflow

Step	Description
create	Invokes one or more tool-workflows to create a resource.
delete	Invokes one or more tool-workflows to delete a resource.

Core Logic & Functions

• run-steps: The engine for dynamic workflow execution.
• prepare: Shared logic for rendering templates and initializing execution environments.
• parse-args: Utility function to normalize string or vector-based arguments.
• select-globals: Utility function to copy the global options across workflows.

Options for opts and step render

• ::name (required): The unique identifier for the workflow instance.
• ::path-fn (optional): Logic for resolving file paths.
• ::params (optional): The input data for the workflow.

Options for prepare-opts and step render

• ::name (required): The unique identifier for the workflow instance.

Options for prepare-overrides and step render

• ::path-fn (optional): Logic for resolving file paths.
• ::params (optional): The input data for the workflow.

Options for opts and step create or delete

• ::create-fn (required): The workflow to create the resource.
• ::delete-fn (required): The workflow to delete the resource.

Naming Conventions

To distinguish between the library core and the Babashka CLI implementation:

• [workflow name]: The library-level function. Requires step-fns and opts.
• [workflow name]*: The Babashka-ready task. Accepts args and optional opts.

(ns wf)

(defn tofu
  [step-fns opts]
  (let [opts (prepare {::name ::tofu
                       ::render/templates [{:template "tofu"
                                            :overwrite true
                                            :transform [["tofu"
                                                         :raw]]}]}
                      opts)]
    (run-steps step-fns opts)))

(ns wf)

(defn tofu*
  [args & [opts]]
  (let [opts (merge (parse-args args)
                    opts)]
    (tofu [] opts)))

; bb.edn
{:deps {group/artifact {:local/root "."}}
 :tasks
 {:requires ([wf :as wf])
  tofu {:doc "bb tofu render tofu:init tofu:apply:-auto-approve"
        :task (wf/tofu* *command-line-args* {:big-config/env :shell})}
  ansible {:doc "bb ansible render -- ansible-playbook main.yml"
           :task (wf/ansible* *command-line-args* {:big-config/env :shell})}
  resource {:doc "bb resource create and delete a resource"
            :task (wf/resource* *command-line-args* {:big-config/env :shell})}}}

Decoupled Data Sharing

Standard Terraform/HCL patterns often lead to tight coupling, where downstream resources must know the exact structure of upstream providers (e.g., the specific IP output format of AWS vs. Hetzner).

BigConfig Workflow solves this through Parameter Adaptation:

1. Isolation: tool-wf-b (Ansible) never talks directly to tool-wf-a (Tofu).
2. Orchestration: The comp-workflow acts as a glue layer. It uses path to discover outputs from the previous workflows (e.g., via tofu output --json) and maps them to the ::params required by the next.
3. Interchangeability: You can swap a Hetzner workflow for an AWS workflow without modifying the downstream Ansible code. Only the mapping logic in the comp-workflow needs to be updated.

Note: Resource naming and state booking are outside the scope of BigConfig Workflow.

Composite workflow

This is an example of composite workflow that contains two tool workflows. Tool workflows must be isolated by using distinct opts maps.

• extract-params: The function to extract outputs from a tool workflow and use them as parameters for another tool workflow.
• resource-create: The composite workflow to create the resource.
• resource-delete: The composite workflow to delete the resource.
• resource: The composite workflow to expose the steps interface.

(defn extract-params
  [opts]
  (let [ip (-> (p/shell {:dir (workflow/path opts ::tofu)
                         :out :string} "tofu show --json")
               :out
               (json/parse-string keyword)
               (->> (s/select-one [:values :root_module :resources s/FIRST :values :ipv4_address])))]
    {::workflow/params {:ip ip}}))

(defn resource-create
  [step-fns {:keys [::tofu-opts ::ansible-opts] :as opts}]
  (let [globals-opts (workflow/select-globals opts)
        tofu-opts (merge (workflow/parse-args "render tofu:init tofu:apply:-auto-approve")
                         globals-opts
                         tofu-opts)
        ansible-opts (merge (workflow/parse-args "render ansible-playbook:main.yml")
                            globals-opts
                            ansible-opts)
        opts* (atom opts)
        wf (core/->workflow {:first-step ::start
                             :wire-fn (fn [step step-fns]
                                        (case step
                                          ::start [core/ok ::tofu]
                                          ::tofu [(partial tofu step-fns) ::ansible]
                                          ::ansible [(partial ansible step-fns) ::end]
                                          ::end [identity]))
                             :next-fn (fn [step next-step {:keys [::bc/exit] :as opts}]
                                        (if (#{::tofu ::ansible} step)
                                          (do
                                            (swap! opts* merge (select-keys opts [::bc/exit ::bc/err]))
                                            (swap! opts* assoc step opts))
                                          (reset! opts* opts))
                                        (cond
                                          (= step ::end)
                                          [nil @opts*]

                                          (> exit 0)
                                          [::end @opts*]

                                          :else
                                          [next-step (case next-step
                                                       ::tofu tofu-opts
                                                       ::ansible (merge-with merge ansible-opts (extract-params @opts*))
                                                       @opts*)]))})]
    (wf step-fns opts)))

(defn resource-delete
  [step-fns opts]
  (let [opts (merge (workflow/parse-args "render tofu:destroy:-auto-approve")
                    opts)]
    (tofu step-fns opts)))

(defn resource
  [step-fns opts]
  (let [opts (merge {::workflow/create-fn resource-create
                     ::workflow/delete-fn resource-delete}
                    opts)
        wf (core/->workflow {:first-step ::start
                             :wire-fn (fn [step step-fns]
                                        (case step
                                          ::start [(partial workflow/run-steps step-fns) ::end]
                                          ::end [identity]))})]
    (wf step-fns opts)))

parse-args

(parse-args str-or-args)

Function.

Utility functions to normalize string or vector-based arguments. See the namespace big-config.workflow.

_Source

path

("path[{:keys [::prefix]} name]")

Function.

Find the path of a previous workflow to extract the outputs.

_Source

prepare

(prepare opts overrides)

Function.

Prepare opts. See the namespace big-config.workflow.

_Source

print-step-fn

(print-step-fn step opts)

Function.

Print all steps of the workflow. See the namespace big-config.workflow.

_Source

run-steps

(run-steps step-fns opts)

Function.

A dynamic workflow that takes a list of steps, a create function, and a delete function. See the namespace big-config.workflow

_Source

select-globals

(select-globals {:keys [globals], :as opts})

Function.

_Source