!5217odin/ci-deep-tier → develop2e8fe5ff21

Run the deep test-quality jobs in every pipeline

AI-generatedEverything on this page was written by an AI agent from the diff: the summaries, the narration, the diagrams, the flags and the glossary. It can be wrong or incomplete, so check it against the code before you rely on it.

4commits
4files
23lines changed
19lines to read
2deep jobs now run
exit 76lock timeout code

Where the diff is

3-minute explainer

What changed

Never ran

Schedule-only rules, and the project has no schedules.

schedule0 schedules0 runs

rules: *rules

Same anchor as consensus-determinism: forks skipped, all else runs.

  • security-fuzz-deep
  • accounting-properties-deep

Fuzz lock: exit 76

A lock timeout warns; a real crash still fails.

75retry76allowed to failotherjob fails

Go pin fixed

Accounting script moves to go1.26.10, matching develop.

go1.26.9→ go1.26.10

Quiet blockscanner test

No-op logger so log spam can't hide the real failure.

scanner.logger = zerolog.Nop()

Look here first

Deep jobs share the one small slotLine 206: one small slot today, so fuzz-deep and accounting-deep queue behind gate jobs; check against the sim's 30-45 min..gitlab-ci.yml:1270 ↗verify
12561263 timeout: 25m
12571264 variables:
12581265 FUZZ_TIME: 1000x
12591266 FUZZ_CORPUS_DIR: "$CI_PROJECT_DIR/.security-fuzz-corpus"
12601267
12611268 security-fuzz-deep:
12621269 extends: .security-fuzz-base
1263− rules:
1264− - if: '$CI_PIPELINE_SOURCE == "schedule"'
1270+ rules: *rules
12651271 timeout: 1h
12661272 variables:
12671273 FUZZ_TIME: 20s
12681274 FUZZ_CORPUS_DIR: "$CI_PROJECT_DIR/.security-fuzz-corpus"
12691275
12701276 accounting-properties-deep:
12711277 tags: !reference [.small-runner, tags]
1272− rules:
1273− - if: '$CI_PIPELINE_SOURCE == "schedule"'
1278+ rules: *rules
12741279 stage: long-test
12751280 needs:
12761281 - job: unit-tests
Commit says fast mutation job droppedCommit 07e0ba34 claims removal, but security-mutations-fast stays and develop has no deep mutation job: description and diff disagree..gitlab-ci.yml:1157 ↗verify
unchanged in this MR · context
11481148 artifacts:
11491149 when: always
11501150 reports:
11511151 junit: test-quality-reports/root-mocknet-race-bifrost/junit.xml
11521152 paths:
11531153 - test-quality-reports/root-mocknet-race-bifrost/
11541154
11551155 # Curated mutations fail when an assertion survives, an anchor changes, or a
11561156 # baseline cannot run. The runner owns the shared build lock and time budget.
11571157 security-mutations-fast:
11581158 tags: !reference [.small-runner, tags]
11591159 rules:
11601160 - if: '$CI_MERGE_REQUEST_EVENT_TYPE == "merge_train" && $CI_MERGE_REQUEST_TARGET_BRANCH_SHA == $CI_MERGE_REQUEST_DIFF_BASE_SHA'
11611161 when: never
11621162 # The observer baseline imports tss-lib-private, which fork MRs cannot fetch.
11631163 - if: $CI_MERGE_REQUEST_SOURCE_PROJECT_PATH && $CI_MERGE_REQUEST_SOURCE_PROJECT_PATH != $CI_MERGE_REQUEST_PROJECT_PATH
11641164 when: never
11651165 - if: '$CI_MERGE_REQUEST_IID && $CI_PIPELINE_SOURCE != "schedule"'
11661166 changes:
11671167 paths:
11681168 - bifrost/observer/**/*
11691169 - bifrost/pkg/chainclients/shared/signing/**/*
11701170 - common/**/*
Exit 76 allowed on both fuzz jobsInherited by security-fuzz-fast too; a busy runner can pass the pipeline with a yellow, unfuzzed job..gitlab-ci.yml:1239 ↗careful
12301230 key: security-fuzz-corpus-v2-$CI_COMMIT_REF_SLUG
12311231 fallback_keys:
12321232 - security-fuzz-corpus-v2-$CI_DEFAULT_BRANCH
12331233 paths:
12341234 - .security-fuzz-corpus/
1235+# Every pipeline's fast and deep fuzz jobs on a runner host queue on one lock
1236+# in the shared build cache. Exit 76 means the wait for it timed out: runner
1237+# load, not a finding. It is a distinct code from 75 so the default retry does
1238+# not requeue the job onto the same contended lock.
1239+allow_failure:
1240+ exit_codes:
1241+ - 76
12351242 before_script:
12361243 - !reference [.pkg-install, before_script]
12371244 - !reference [.go-private-auth, before_script]
12381245 - !reference [.stable-mtimes, before_script]
12391246 script:
12401247 - mkdir -p test-quality-reports
12411248 - sh -c 'scripts/test-security-fuzz.sh > test-quality-reports/security-fuzz.log 2>&1; rc=$?; cat test-quality-reports/security-fuzz.log; exit "$rc"'
12421249 artifacts:
12431250 when: always
12441251 paths:
12451252 - test-quality-reports/security-fuzz.log
Fast and deep share one corpus cacheBoth now run per pipeline with one cache key; the last upload wins and can drop the other's new entries..gitlab-ci.yml:1225 ↗careful
12161216
12171217 .security-fuzz-base:
12181218 tags: !reference [.small-runner, tags]
12191219 stage: long-test
12201220 needs:
12211221 - job: unit-tests
12221222 optional: true
12231223 artifacts: false
12241224 variables:
12251225 PKGS: util-linux
12261226 FUZZ_CORPUS_DIR: "$CI_PROJECT_DIR/.security-fuzz-corpus"
12271227 # Each branch keeps its own corpus, seeded from the default branch's, so a
12281228 # crasher saved by one branch is never replayed on another.
12291229 cache:
12301230 key: security-fuzz-corpus-v2-$CI_COMMIT_REF_SLUG
12311231 fallback_keys:
12321232 - security-fuzz-corpus-v2-$CI_DEFAULT_BRANCH
12331233 paths:
12341234 - .security-fuzz-corpus/
1235+ # Every pipeline's fast and deep fuzz jobs on a runner host queue on one lock
1236+ # in the shared build cache. Exit 76 means the wait for it timed out: runner
1237+ # load, not a finding. It is a distinct code from 75 so the default retry does
1238+ # not requeue the job onto the same contended lock.
20 min lock wait vs 25 min timeoutThe fast fuzz job's wait counts against its 25m timeout, so long contention ends in a hard timeout, not 76.test-security-fuzz.sh:43 ↗careful
3434 # fuzzing and back out on any exit. Clear it first: the build cache volume is
3535 # shared by every branch, and only the corpus from $corpus_dir may be replayed.
3636 # The lock lives on that volume, so jobs in other containers sharing it wait
3737 # instead of swapping the subtree out from under each other.
3838 gocache=$(GOTOOLCHAIN=go1.26.10 go env GOCACHE)
3939 mkdir -p "$gocache"
4040 exec 9>"$gocache/.thornode-security-fuzz.lock"
41+# Exit 76 on a lock timeout: CI allows that code to fail, since it reflects
42+# runner load rather than a fuzz finding.
4143 flock -w 1200 9 || {
4244 echo 'Timed out waiting for the shared fuzz cache lock' >&2
43− exit 1
45+ exit 76
4446 }
4547 fuzz_cache=$gocache/fuzz/gitlab.com/thorchain/thornode/v3
4648 rm -rf "$fuzz_cache"
4749 mkdir -p "$fuzz_cache"
4850 cp -a "$corpus_dir/." "$fuzz_cache/"
4951 trap 'cp -a "$fuzz_cache/." "$corpus_dir/"' EXIT
5052 printf 'Fuzzing %d target(s), each for %s; corpus cache: %s\n' "$selected" "$budget" "$corpus_dir"
5153 for spec in "${targets[@]}"; do
5254 name=${spec%%|*}
5355 package=${spec#*|}
No -run filter or -timeoutNot in this diff, but now per pipeline: x/thorchain's plain Test funcs run too, under the default 10m timeout.test-accounting-property.sh:20 ↗careful
1212
1313 cd "$(dirname "$0")/.."
14−export GOTOOLCHAIN=go1.26.9
14+export GOTOOLCHAIN=go1.26.10
1515 export GOCACHE="${GOCACHE:-${TMPDIR:-/tmp}/thornode-accounting-gocache}"
1616 export GOMAXPROCS=4
1717 export THORNODE_ACCOUNTING_SEEDS="${THORNODE_ACCOUNTING_SEEDS:-$seeds}"
1818
1919 flock -w 600 /tmp/thornode-test-quality-build.lock \
2020 go test -p 2 ./x/thorchain -check.f AccountingPropertySuite -count=1
Campaign seeds are fixed 1..32Not in this diff: each pipeline replays the same 32 cases, a regression check rather than exploration.test-accounting-property.sh:6 ↗careful
11 #!/usr/bin/env bash
22 set -euo pipefail
33
44 case "${1:-fast}" in
55 fast) seeds=4 ;;
66 campaign) seeds=32 ;;
77 *)
88 echo "usage: $0 [fast|campaign]" >&2
99 exit 2
1010 ;;
1111 esac
1212
1313 cd "$(dirname "$0")/.."
14−export GOTOOLCHAIN=go1.26.9
14+export GOTOOLCHAIN=go1.26.10
1515 export GOCACHE="${GOCACHE:-${TMPDIR:-/tmp}/thornode-accounting-gocache}"
1616 export GOMAXPROCS=4
1717 export THORNODE_ACCOUNTING_SEEDS="${THORNODE_ACCOUNTING_SEEDS:-$seeds}"
1818
Logger swapped after constructionTest-only; set before scanner starts, so no race with the scan goroutine.blockscanner_extra_test.go:1043 ↗look
10331034 HTTPRequestTimeout: time.Second,
10341035 BlockHeightDiscoverBackoff: 100 * time.Millisecond,
10351036 BlockRetryInterval: 100 * time.Millisecond,
10361037 ChainID: common.ETHChain,
10371038 MaxHealthyLag: time.Second, // very short lag threshold
10381039 }, mss, m, bridge, fetcher)
10391040 c.Assert(err, IsNil)
1041+ // An unhealthy scanner logs every block; a second of that at full speed
1042+ // overflows the CI job log and hides whichever test actually failed.
1043+ scanner.logger = zerolog.Nop()
10401044
10411045 globalChan := make(chan types.TxIn, 100)
10421046 nfChan := make(chan common.NetworkFee, 5)
10431047 scanner.Start(globalChan, nfChan)
10441048 // health is set on every scanned block
10451049 waitForScanner(c, func() bool { return scanner.PreviousHeight() > 1 }, "a scanned block")
10461050 scanner.Stop()
10471051
10481052 // Scanner should be unhealthy when significantly behind
10491053 c.Check(scanner.IsHealthy(), Equals, false)
10501054 }
10511055
10521056 // TestStartWithPersistedPos tests that Start reads the persisted scan position.

Ask an AI about this change

Chat assistantClaude, ChatGPT, … (reads a link)

Coding agentClaude Code, Cursor, … (reads the repo)

The context file is context.md. Use Copy with context for an AI that can't open links; it pastes the whole file in.

Glossary

Accounting property campaign
Seeded tests that drive swaps, streams, refunds and withdrawals and check an outside ledger balances to the unit. Campaign mode runs seeds 1 to 32. handler_accounting_property_test.go ↗ test-accounting-property.sh ↗
allow_failure: exit_codes
GitLab marks a job that exits with a listed code as a warning instead of a failure. Here 76 means the fuzz lock wait timed out. GitLab allow_failure ↗
Exit 75 (EX_TEMPFAIL)
The repo's code for an infrastructure failure; the default retry block re-runs jobs that exit 75. 76 is chosen so lock timeouts are not retried. .gitlab-ci.yml ↗
flock
A util-linux tool that holds a file lock around a command. The fuzz lock lives on the shared GOCACHE volume, so it works across job containers. test-security-fuzz.sh ↗
Fuzz corpus
The saved inputs Go's fuzzer replays and mutates. Each branch keeps its own in the GitLab cache, seeded from the default branch's. docs/security-fuzz.md ↗ Go fuzzing ↗
GOTOOLCHAIN
Tells Go which toolchain to use. The accounting script's pin moves from go1.26.9 to go1.26.10, the version develop uses. Go toolchains ↗
Merge train
GitLab queues approved MRs and tests each on top of the ones ahead of it. Train jobs can't be retried by hand here, so flaky jobs block merges. GitLab merge trains ↗
Shared rules anchor
The YAML anchor at .gitlab-ci.yml:387, used as rules: *rules. It skips merge requests from forks and runs every other pipeline; the deep jobs now use it. .gitlab-ci.yml ↗ GitLab YAML anchors ↗