# Large builds

Source: https://www.testifysec.com/docs/cilock/guides/large-builds/

A large build should never fail just because it is large. cilock applies a few limits to keep

A large build should never fail just because it is large. cilock applies a few limits to keep attestations readable by the things that verify them. When a build crosses one, cilock keeps the evidence, moves the bulk into a signed companion file bound to the attestation by digest, and records what happened in the attestation itself. It refuses only when there is no way to keep the evidence under the limit, and then it says exactly what filled it.

 

Every limit message cilock prints names the setting to change (its environment variable and its flag), says what the overflow cost, and links the section of this page for that limit.

 

## The limits at a glance

 

| Limit | Flag | Environment variable | Default | On overflow |
| --- | --- | --- | --- | --- |
| Product count | `--attestor-product-max-products` | `CILOCK_MAX_PRODUCTS` | 10000 (0 = no limit) | Compact profile (default): build succeeds, every product kept in the signed product inventory, gap `products-over-limit` recorded. Legacy profile: refused, because there is nowhere to keep the overflow. |
| Attestation size | `--max-attestation-bytes` | `CILOCK_MAX_ATTESTATION_BYTES` | 4MiB (0 = no limit) | A large command-run trace moves to a signed `command-run-trace` companion. If the statement is still over, the step is refused with a per-attestor breakdown. |
| Trace split threshold | `--attestor-command-run-trace-split-threshold` | none | 4194304 bytes (0 = never split) | The process trace moves out of the statement into the trace companion. |
| Product inline budget | none (build-time) | none | 128 KiB | Product leaves past this go to the detached product inventory instead of inline. |

 

The flag always wins over the environment variable, and the environment variable wins over the default. A malformed value is refused before the build runs, never ignored. Both environment variables apply to `cilock run` and `cilock attest`.

 

## Product count limit

 

The product attestor records every file the build produced. Past 10,000 files the run is almost always recording a dependency tree (`node_modules`, `.venv`, `vendor`) as if it were the build's output.

 

**What happens over the limit.** With the default **compact** evidence profile the build succeeds and every product is kept. The per-file list lives in the signed product inventory written beside `--outfile` as `<outfile>-product-inventory.json`, bound to the attestation by its sha256, and the product predicate carries a named coverage gap a policy can read:

 

```json
"coverageGaps": [
  {"name": "products-over-limit", "limit": 10000, "matched": 11115}
]
```

 

A gap is only written when every matched product is recorded. With the **legacy** profile there is no inventory to hold the overflow, and signing a subset would claim a complete product set that is not, so the run is refused with the same message. A set whose inventory would pass the inventory's own 64 MiB ceiling is refused too, with a message that gives the product count and inventory size. Each entry costs about 130 bytes plus the length of its path, so the ceiling arrives at roughly 380,000 products with short absolute paths and fewer with long ones, well before the 1,000,000-entry cap (see [Ceilings that are not configurable](https://www.testifysec.com/docs/cilock/guides/large-builds/#ceilings-that-are-not-configurable)).

 

**What to do.** Exclude the dependency tree, so the product set is what the build actually ships:

 

```bash
cilock run --attestor-product-exclude-glob '{**/,}{node_modules,.venv,vendor}/**' -- npm ci
```

 

The exclude flag takes ONE pattern; repeating it keeps only the last. Put every tree in one brace alternation.

 

With `--trace`, products are recorded by absolute path. Name each tree with a leading `**/` so it matches wherever it sits, including outside the working directory (npm's `~/.npm`, Maven's `~/.m2`):

 

```bash
cilock run --trace --attestor-product-exclude-glob '{**/node_modules/**,**/.npm/**}' -- mvn -B install
```

 

Do not write a single name as `{**/,}node_modules/**`: the glob library compiles that form to a pattern that matches nothing. Never exclude a directory that holds the working directory (a home directory such as `/home/jenkins/**` on a CI agent): that drops the build's own output too. The over-limit message never suggests one; when the largest directory holds the working directory, it names the dependency folders it found instead, or offers no command.

 

If the build really does ship more files than the limit, raise it:

 

```bash
export CILOCK_MAX_PRODUCTS=50000      # or --attestor-product-max-products=50000
```

 

A policy that wants to flag a product set this large (usually a dependency tree recorded as output) can deny on the gap:

 

```rego
deny[msg] {
  gap := input.coverageGaps[_]
  gap.name == "products-over-limit"
  msg := sprintf("%d products matched, over the limit of %d", [gap.matched, gap.limit])
}
```

 

## Attestation size limit

 

`--max-attestation-bytes` bounds the in-toto statement cilock signs. The platform parses every envelope matching a commit on each push evaluation at about 0.4 s per MB against a 25 s budget, so the 4 MiB default keeps a commit with a few envelopes near two seconds.

 

**What happens over the limit.**

 

1. A command-run trace over the trace split threshold moves to a signed `command-run-trace` companion (`<outfile>-command-run-command-run-trace.json`). The statement keeps a `command-run/v0.3` skeleton that names the companion's sha256, byte count, process count and opened-file count. A verifier loads the companion, checks the digest, and evaluates the policy against the full trace. This happens for a failed command too, so a failed build still leaves evidence.
 2. Product and material per-file details already live in their detached inventories under the compact profile.
 3. If the collection is still over the limit, cilock moves the command-run trace and the product leaves into their companions regardless of their own thresholds, and measures again. An explicit `--attestor-command-run-trace-split-threshold=0` (never split) is respected.
 4. If the statement is still over the limit, the step is refused before anything is signed, written or uploaded, and cilock lists what it already moved and the largest contributors that remained, with a remedy for each.

 

**What to do when it is refused.** Read the contributor list. A command-run trace is split automatically, so a refusal usually means stdout or stderr is very large (redirect it to a file under the working directory so it is attested as a product digest) or several attestors together crossed the limit. Raise the limit only if the evidence store can take it:

 

```bash
export CILOCK_MAX_ATTESTATION_BYTES=16MiB   # or --max-attestation-bytes=16MiB
```

 

## Where the companions go

 

| Companion | File beside `--outfile` | Uploaded when |
| --- | --- | --- |
| Product inventory | `<outfile>-product-inventory.json` | `--upload-inventories` (default on in compact-chain builds) |
| Material inventory | `<outfile>-material-inventory.json` | `--material-manifest` retains it; `--upload-inventories` uploads it |
| Command-run trace | `<outfile>-command-run-command-run-trace.json` | with the collection, whenever Archivista upload is enabled |

 

Each companion is its own signed envelope. The attestation names it by sha256, and a verifier refuses a companion whose digest does not match.

 

A run that splits writes the companion beside `--outfile`, so a traced build large enough to split needs `--outfile` even when it only uploads. Without one, cilock refuses the run and says why.

 

## The trace companion format (`command-run-trace/v0.2`)

 

The trace companion's predicate type is `https://aflock.ai/attestations/command-run-trace/v0.2`. Its body is the full `command-run/v0.2` document with one change: writes are coalesced. The skeleton's `_meta.trace.type` names the companion type, and a verifier refuses a body whose `_meta.version` disagrees with it.

 

- `_meta.version` is `trace/v0.2`.
 - `_meta.writeColumns` is `["pathId", "dev", "ino", "calls", "bytes"]`.
 - Each process's `fileOps.writes` holds one five-integer array per process, path and file, instead of one object per `write(2)` call. `pathId` indexes the document's `paths` table, the same interned table the rest of the trace uses. `dev` and `ino` are the file the tracer saw written, both `0` when it saw only a name. `calls` counts the writes merged into the record, and `bytes` sums them; a write of unknown size adds 0.
 - `renames`, `deletes` and `permChanges` keep their v0.2 form.

 

A verifier expands each record back into the `fileOps.writes` entries a Rego policy reads, so a policy written against inline v0.2 traces reads a split trace without changes. A dangling `pathId`, a record with zero calls or a malformed column refuses the whole companion; nothing is dropped quietly. On a cold build of our own `jade` tool, per-call write records were 63.7 MB of an 81.6 MB trace, which is why this version exists.

 

Verifiers read both `command-run-trace/v0.1` and `v0.2` companions. cilock built from source after v4.8.0 signs v0.2 and is not yet in a release on cilock.dev; earlier builds that split sign v0.1. A verifier that predates the v0.2 reader refuses a v0.2 body instead of evaluating it, so upgrade verifiers before producers.

 

## Ceilings that are not configurable

 

| Ceiling | Value | Why |
| --- | --- | --- |
| File inventory entries | 1,000,000 | bounds what a verifier decodes |
| File inventory size | 64 MiB | same |
| Trace companion size | 383 MiB | the largest companion a verifier will download |

 

A build over one of these is past what any verifier will read. Split it into several `cilock run` steps, one per module or phase.

 

## Recommended settings

 

**Large monorepo.** Keep the defaults and exclude generated trees in one pattern. Split very long builds into steps per module so each trace stays readable.

 

```bash
cilock run --step build \
  --attestor-product-exclude-glob '{**/,}{node_modules,dist/cache,.turbo}/**' -- make all
```

 

**node\_modules-heavy (npm, yarn, pnpm).** Install and build in separate steps. Exclude `node_modules` from the build step's products; record it as material of the install step instead.

 

```bash
cilock run --step install -- npm ci
cilock run --step build --attestor-product-exclude-glob '{**/,}node_modules/**' -- npm run build
```

 

**Go with vendor or a large module cache.** Builds that compile hundreds of packages produce large process traces; the trace companion handles that automatically. Exclude `vendor` from products when the vendored sources are inputs, not outputs.

 

```bash
cilock run --step build --attestor-product-exclude-glob '{**/,}vendor/**' -- go build ./...
```

 

**Detection and other unattended probes.** The defaults already always produce an attestation for these two limits under the compact profile: an oversized product set is kept whole with a `products-over-limit` gap, and an oversized trace moves to its companion. To silence the gap for builds known to ship many files, raise the product limit once in the environment:

 

```bash
export CILOCK_MAX_PRODUCTS=100000
```

Reference generated from the product documentation. Match commands and support details to your installed release.
