How to circumvent input size restrictions in Terragrunt

[yhakbar]

The Problem

When you reach a certain scale with Terragrunt, and inputs are too large, you might encounter an error like the following:

$ terragrunt plan
08:46:20.895 ERROR  tofu invocation failed in .
08:46:20.903 ERROR  error occurred:

* Failed to execute "tofu init" in .

  fork/exec ../../.local/share/mise/installs/opentofu/1.10.2/tofu: argument list too long

08:46:21.007 ERROR  Unable to determine underlying exit code, so Terragrunt will exit with error code 1

The root cause of this error is that the size of the inputs being set in a terragrunt.hcl has exceeded the operating system limit for how long environment variables can be.

The following terragrunt.hcl and main.tf files reproduce that error:

# terragrunt.hcl

locals {
  length     = 100000000
  long_input = run_cmd("--terragrunt-quiet", "bash", "-c", "head -c ${local.length} /dev/urandom | base64 | tr -d '\n' | head -c ${local.length}")
}

inputs = {
  long_input = local.long_input
}

# main.tf

variable "long_input" {
  type = string
}

output "shortened_output" {
  value = substr(var.long_input, 0, 100)
}

The reason for this is that Terragrunt is going to try to set the environment variable TF_VAR_long_input to a random string of characters 100000000 long.

The Workaround

The workaround for this is to write the contents of the input to an .auto.tfvars or .auto.tfvars.json file instead of using inputs.

# terragrunt.hcl

locals {
  length     = 100000000
  long_input = run_cmd("--terragrunt-quiet", "bash", "-c", "head -c ${local.length} /dev/urandom | base64 | tr -d '\n' | head -c ${local.length} | jq --raw-input '{\"long_input\": .}' > .auto.tfvars.json")
}

$ terragrunt plan
09:07:22.738 INFO   tofu: Initializing the backend...
09:07:22.738 INFO   tofu: Initializing provider plugins...
09:07:22.738 INFO   tofu: OpenTofu has been successfully initialized!
09:07:22.738 INFO   tofu:
09:07:22.738 INFO   tofu: You may now begin working with OpenTofu. Try running "tofu plan" to see
09:07:22.738 INFO   tofu: any changes that are required for your infrastructure. All OpenTofu commands
09:07:22.738 INFO   tofu: should now work.
09:07:22.738 INFO   tofu: If you ever set or change modules or backend configuration for OpenTofu,
09:07:22.738 INFO   tofu: rerun this command to reinitialize your working directory. If you forget, other
09:07:22.738 INFO   tofu: commands will detect it and remind you to do so if necessary.
09:07:26.147 STDOUT tofu: Changes to Outputs:
09:07:26.147 STDOUT tofu:   + shortened_output = "eKh1bL6V6HwzFtvhVJVR03cxt6CrVlIwxkkg6dQYUMZx8+1L3MsyrsH5KxYue/yTofzgPC17gN4ncYJNx6q8I4jWUorEOGhrLEkq"
09:07:26.147 STDOUT tofu: You can apply this plan to save these new output values to the OpenTofu
09:07:26.147 STDOUT tofu: state, without changing any real infrastructure.
09:07:26.147 STDOUT tofu:
09:07:26.147 STDOUT tofu: ─────────────────────────────────────────────────────────────────────────────
09:07:26.147 STDOUT tofu: Note: You didn't use the -out option to save this plan, so OpenTofu can't
09:07:26.147 STDOUT tofu: guarantee to take exactly these actions if you run "tofu apply" now.

$ jq '.long_input | length' .auto.tfvars.json
100000000

There's also the workaround of not having inputs that large, but we can't always control that 😁 .

The Solution

The longer term solution is going to be something like the following, where Terragrunt can handle this automatically on behalf of users:
#4402

The community has largely pushed back on that because it's a breaking change to the inputs attribute (no matter how long backwards compatibility is preserved), so we'll need more feedback and design iteration before a simple solution like that is built into Terragrunt.