More › status

More

What proves each requirement

A model says what the system promises: its requirements. archec status answers the next question. What proves each promise today?

For each requirement it shows what satisfies it, where that is built in the generated code, and every piece of evidence: a test case, a property the model checker proved, timing measured on a board, a log of the real system. When something the evidence depended on has changed since, it says so. When nothing proves a requirement, it says that too.

How it works#

The commands that prove things record what they found:

Command What it records
archec test each case that ran, passed or failed, on the simulator
archec verify each formal property NuSMV or nuXmv proved or refuted
archec timing each requirement a chain, a thread or a monitor of the traces speaks for
archec replay each requirement checked on the real system's logs
archec live --replay the final state of each requirement
archec test --against <ref> each case that fails on an older build and passes on this one: a fix
archec cad each limit and clearance a requirement puts on a part's geometry (CAD as code)
archec synth each limit a requirement puts on what synthesis measures of an FPGA component (limit Pwm.luts <= 500), and each FPGA component placed and routed for its device (FPGA components)

Each record goes in .arche/evidence/, beside your main file, one file per command. You do nothing more: run your tests as usual.

Then ask:

$ archec status
REQ-FAN   Fan on when hot
  The fan shall start when the air is above 28.
  satisfied by  Thermostat (ctl)
  built into    gen/c/thermostat.c:11-19,97-128, gen/c/thermostat.h:16-20
  ✓ test    TP/TC-HOT    web sim              2 h ago
  ✓ verify  REQ-FAN      NuSMV                2 h ago  ctl in Idle leads to ctl in Cooling
  ✓ timing  chain react  board nucleo-f446re  3 h ago  chain react max 212ms ≤ budget 1s (40 run(s))

REQ-STOP  Fan stops when cool  (out of date)
  ! test    TP/TC-COOL   web sim   out of date: Thermostat: Cooling -> Idle on READING changed

REQ-SAFE  Never both  (nothing proves it)
  ✗ nothing proves it   add a test case that verifies it, or a formal property for archec verify

3 requirement(s): 1 proved, 1 out of date, 1 unproved, 0 failing

A requirement is in one of four states:

  • proved: it has evidence, all of it current and passing.
  • out of date: something a piece of evidence depended on changed since it ran. Run it again.
  • failing: a current piece of evidence failed.
  • nothing proves it: no evidence at all. A case that verifies it but never ran is listed, with never ran.

What "out of date" means#

Each record keeps a fingerprint (a short hash) of every model element the run depended on. When you run archec status, it hashes the model again and compares.

What a run depends on:

  • A test case: the case itself, the requirements it verifies, the parts it names, the parts that satisfy those requirements, and every part upstream of them (a part that sends them messages).
  • A formal property: the requirement, the parts it names, and the parts upstream of them.
  • A timing run or a replay: the requirement, the deployment, the parts measured, and the code that ran (see below).

A part is hashed piece by piece: each function, port, variable, state and transition apart, then the rest. So the message names the piece that changed: Thermostat.update changed, or Thermostat: Idle -> Cooling on READING changed for a guard. A part with no link to what a case checks is not in its fingerprints: editing it changes nothing. Comments and layout do not count. A requirement is hashed by its wording only, so changing its status or owner keeps its evidence current.

Runs on a board or an emulator#

archec timing, replay and live take three options to say where they ran:

  • --board <id>: on that board (board nucleo-f446re in the report).
  • --place <where>: anywhere else, such as renode or host.
  • --firmware <file>: the firmware that ran. Its hash is kept. When the file changes, the evidence is out of date.

The environment can say the same: ARCHE_BOARD, ARCHE_EVIDENCE_PLACE and ARCHE_FIRMWARE. A script that flashes a board can set them once. Without them, the place is trace or log.

The generated files that hold the code of the measured parts are hashed too. archec status finds them through the genmaps archec gen writes (<out>/.archec/genmap.json), up to three folders down. The same genmaps give the built into lines.

The build a board runs#

Firmware generated with --trace (--target embassy or c) says which build it is, in the first line of its trace:

arche-trace 0 build 79bbe5250c9e 3f2a1c9b0d

The first word is the model's build fingerprint: one short hash of everything code is generated from (components, messages, systems, deployments, types, the .py files of bodies). Editing a requirement's wording or a test case does not change it. The second is the git commit the code was generated at: 3f2a1c9b0d-dirty when tracked files had changes, nogit outside a repository (ARCHE_BUILD_COMMIT says it instead, in CI).

archec replay, timing and live --replay read that line before anything else:

  • The trace is of this model's build. One grey line says so, and each record keeps the build ("build": {"fingerprint", "commit"}). --firmware is no longer needed to know what ran. When the model changes later, the record is out of date: the model changed since build 79bbe5250c9e ran (rebuild and flash).
  • The trace is of another build. The first line of the report is ✗ the board runs an older build: rebuild and flash, before any verdict. Nothing is recorded, and the command exits 1. --any-build judges the trace anyway (an old log you want to read); it still records nothing.
  • The trace has no build line (a CAN log, a firmware generated before): nothing is said, and everything works as before.

To ask only that question:

$ archec status --build                       # the model's build now
build 79bbe5250c9e 3f2a1c9b0d
$ archec status --build build/board/firmware.trace
✓ build 79bbe5250c9e, commit 3f2a1c9b0d: the firmware is this model's

It exits 1 when the trace is of another build, 2 when the trace does not say. A script that flashes a board asks it first, as examples/greenhouse/board.sh does two seconds after the reset.

The test ladder#

archec test main.arche --ladder runs the same proof on three rungs (four for a model with FPGA parts: ghdl, between renode and board), lowest first, and stops at the first rung that fails:

$ archec test main.arche --ladder --board 0483:374b
  ✓ sim     4 case(s) passed on the browser simulator
  ✓ renode  board.sh --renode held (212 s)
  ✓ board   board.sh held (95 s)
✓ ladder: held up to board
Rung What runs It runs when
sim the selected cases (--plan, --case) on the simulator always
renode the project's hook with --renode the deployment has a binary on a board, renode is installed, and there is a hook
ghdl archec test --target vhdl: the cases on the generated VHDL, in GHDL (fpga.md) the model has FPGA parts, and GHDL is here; the rung is named only for such a model
board the hook with no argument the same, a flash tool of the chip (probe-rs…) is installed, and you asked with --board <probe>

For an FPGA there is one rung more, above ghdl: synth, the design synthesised, placed and routed for its device. The ladder does not run it: archec synth does (fpga.md), and archec status shows it. It says a component is built for its device and meets its clock. Its requirements are still proved by its tests.

The board rung flashes, so it never runs unless --board asks for it. A rung whose tool is missing is skipped, with the reason, not failed: – renode skipped: renode not installed. Nothing is downloaded: Renode and probe-rs are yours to install.

The hook is a script beside the model that builds the firmware with its trace, runs it, and hands the trace to archec timing and archec replay: board.sh, else --hook <script> or ARCHE_LADDER_HOOK. examples/greenhouse/board.sh is one to copy. Its contract:

  • hook --renode runs the firmware in Renode; hook flashes the board and records its trace.
  • It exits 0 when everything held, 2 when it could not run (a tool or the board is missing: the rung is skipped, with the hook's last ✗ line as the reason), anything else when something did not hold.
  • It is given ARCHEC (this archec), ARCHE_LADDER (renode or board), ARCHE_EVIDENCE_PLACE=renode on the Renode rung, and ARCHE_PROBE and ARCHE_BOARD (the value of --board) on the board rung. So what its archec timing and archec replay record says which rung it came from.

Each rung's output is kept in <out>/.archec/ladder/<rung>.log (--verbose prints it as it runs), and the ladder in <out>/.archec/ladder.json (--json prints it).

Every record has a place, and a place is on a rung: sim and web sim are the simulator, renode is Renode, board <id> is the board. archec status shows the highest rung where everything current passed, once a rung above the simulator has evidence:

REQ-SAFE  No short cycling
  highest rung  renode (sim ✓, renode ✓, board –)
  ✓ test    TP-GREENHOUSE/TC-SAFE  web sim  2 h ago
  ✓ replay  REQ-SAFE               renode   2 h ago  1 checked

– is a rung with no evidence, ✗ one where something current failed, ! one that is out of date. --json gives rung and rungs per requirement.

A test that proves a fix#

A regression test is worth something when it fails on the build that has the bug and passes on the one that fixes it. archec test --against <git-ref> runs both:

$ archec test main.arche --against HEAD
archec test --against HEAD  1 case(s) on 3f2a1c9b0d (the model as it was, the cases as they are), then here
  ✓ TP/TC-WARM  proves the fix: it fails on HEAD (3f2a1c9b0d) and passes here
✓ 1 case(s) prove the fix

The older build is a checkout of the ref in a temporary folder (a git worktree, removed afterwards). The cases run there as they are written now, added to the model as it was, then on the working tree. Both runs are on the simulator. With no --plan or --case, the cases are the ones that are new or changed since the ref.

On the ref Here What it says
fails passes proves the fix: a fix record
passes passes this test does not catch the bug: recorded too, in yellow
any fails fails here: the fix is not in

The command exits 0 only when every case proves the fix. archec status shows a fix line under the requirements the case verifies:

  ✓ test    TP/TC-WARM  web sim  just now
  ✓ fix     TP/TC-WARM  web sim  just now  proves the fix: it fails on HEAD (3f2a1c9b0d) and passes here

A fix record says what a test is worth, not whether the requirement holds: it never changes a requirement's state. It depends on the case alone, so it goes out of date only when the case is edited.

In the editor and in the documents#

In VS Code, a line above each requirement says the same in short: proved by 3 tests · timing on board nucleo-f446re ✓ · 2 h ago, out of date: … or nothing proves it. A requirement proved above the simulator, or by a test that proves a fix, says so: … · proves a fix · up to renode. Click it for the full status in the Arche Status output.

Three commands of the palette run the checks above with the archec the extension ships, and show their report in the same output:

  • Arche: Run Test Ladder…: it asks how far up, and for the probe when the board is included.
  • Arche: Prove a Fix Against a Git Ref…: it asks for the ref and the case.
  • Arche: Check Board's Build from Trace…: it asks for the trace file.

They work with no internet. Renode and probe-rs are not part of the extension: a rung whose tool is not installed is reported skipped. See the VS Code extension.

archec doc adds a chip to each requirement (proved, out of date, failing, nothing proves it) and a "Proved by" line with each piece of evidence, its place and its date. It does so only when the project has evidence.

In CI#

archec status exits 1 when a requirement fails, is out of date, or has nothing that proves it. That makes a gate:

archec test main.arche
archec verify main.arche
archec status main.arche

--lenient exits 1 only when a requirement fails. --req REQ-FAN looks at one requirement. --json prints the whole report for another tool.

Commit the evidence, or not#

The files in .arche/evidence/ are small JSON lines, sorted, one line per subject and place. A new run replaces its own lines. Commit them, and the repository shows what proved the model at each commit, CI's runs included. To keep them on your machine only, add .arche/evidence/ to .gitignore.

Any of the commands takes --no-record to record nothing (or set ARCHE_NO_RECORD=1).

Limits#

  • The dependencies are found from the model's text and links, not from what a run really did. A case that names no part depends on the whole system. A part reached only through a function call, not a link, is not seen as upstream.
  • Types and constants a part uses are not fingerprinted, only the messages of its ports. A body in a .py file is hashed as a whole file; a TypeScript handler file is not hashed.
  • A firmware is compared only where it is: on a machine without it, its record stays current.
  • archec live records its final state; a live session under podman is recorded when it stops, not as it runs.
  • The cases that run on arche-cli (archec test without --sim web) are recorded as sim; the tests of the generated C (gen --target c-tests --run) are not recorded yet.
  • One record is kept per subject, place and requirement: the history is git's.
  • The build fingerprint is of the model's text. A change in a TypeScript handler file, a driver's source or the toolchain gives the same fingerprint: the commit beside it tells those apart.
  • The ladder's rungs above the simulator are what the project's hook does. archec does not build or flash the firmware itself, and does not check that the hook ran the selected cases.
  • --against carries a case to the older build as text. A case made from a cases template, or written in another file than its plan, is not carried. A case that names a part the older model does not have cannot run there: the command says so and proves nothing.
  • Nothing here has run on a real board yet: the board rung is tested with a stand-in hook.