How-to guides › Generate integration files

How-to guides

Integration targets

A firmware lives among other tools: protocol tools, build systems, flashers and test benches. archec gen --target <t> can write what those tools read, from the same model as the code. Each target writes one kind of file, and its notes say what the model did not give it.

A first target#

examples/can-bench has three boards on one CAN bus. The bench target writes the configuration of the test rack for its deployment:

cd examples/can-bench
archec gen main.arche --target bench -o gen
note: platform BenchHw: no power link — the rack's supplies are not in the model (link a supply's `out v: power` to each board's `in pwr`)
  gen/bench/OnBench.yaml
  gen/bench/setup_can_bus.sh
  gen/bench/bench_can_bus.py

The start of gen/bench/OnBench.yaml:

deployment: OnBench
buses:
  - id: can_bus
    name: "CAN bus"
    type: can
    bitrate: 500000
    interface: can0   # the rack's adapter on it
    nodes:
      - part: pi
        port: can0
        binaries: [gw]
      …
    frames:
      - id: 0x100
        message: FAN_CMD
        length: 1
        sender: gw
        receiver: fan_fw
      - id: 0x200
        message: ROOM_TEMP
        length: 2
        sender: gw
        receiver: ui_fw
        period_ms: 500
      …

The note says what the model lacks: here, how the boards are powered. The other targets work the same way.

How the targets work#

Each target is one generator, like the others that archec gen --list shows. It is a pure function of the model and the deployment: the same model always gives the same files. The deployment is the one --deployment names, or else the active one.

Each target writes only what the model says. Some protocols need a value the language has no word for, such as a J1939 SPN, a MAVLink id or a CANopen vendor id. The target reads that value from an attr on the declaration it belongs to. When the attr is missing, the notes say so. A value is never made up silently.

Target Writes From
canopen canopen/<bus>/<binary>.eds: a CiA 306 EDS per binary on a CAN bus the routes over the bus, the messages' layout
j1939 j1939/<bus>.json, .md: the parameter groups and SPNs the routes with a 29-bit id
mavlink mavlink/<project>.xml: a MAVLink 2 dialect the message catalog
someip someip/<deployment>.json: SOME/IP services, methods, events, instances interfaces and Ethernet ports
openapi openapi/<Component>.json: OpenAPI 3.1 the ports' HTTP handlers
asyncapi asyncapi/<system>.json: AsyncAPI 3.0 the links that carry messages, their topics
cmake cmake/: a CMake project per C binary, a toolchain per microcontroller the binaries, their processors' cores
west west/<binary>/: a Zephyr application with its west manifest the binaries on Zephyr boards
platformio platformio/<binary>/platformio.ini the binaries on boards PlatformIO knows
ci ci/.github/workflows/archec.yml, ci/.gitlab-ci.yml the targets the deployment uses
flash flash/<binary>/flash.sh, monitor.sh each microcontroller binary's chip and board
bench bench/<deployment>.yaml, setup_<bus>.sh, bench_<bus>.py the platform: buses, boards, power and IO links

There is one target per format, rather than one target with sub-kinds. So each format has its own --check, its own notes and its own row in archec gen --list. A CI job regenerates only the formats it uses.

Each section below ends with how the output is checked in archec's own tests (test/integration.test.js).

Protocol descriptions#

CANopen (--target canopen)#

It writes an electronic data sheet (EDS, CiA 306, version 4.0) for each binary that sends or receives on a CAN bus of the deployment: canopen/<bus>/<binary>.eds. The EDS has:

  • the mandatory objects: device type (0x1000), error register (0x1001) and identity (0x1018);
  • a transmit PDO for each frame the binary sends on the bus (0x1800 + n, with its mapping at 0x1A00 + n);
  • a receive PDO for each frame it takes (0x1400 + n, and 0x1600 + n);
  • an object for each message, in the manufacturer area (from 0x2000). The index is the same in every node's EDS. Each signal is a sub-index, named as in the model (TEMPERATURE.value), with its CANopen type.

For each PDO:

  • The COB-ID is the route's id, with bit 29 set for a 29-bit id.
  • A message with a period is sent on the event timer: transmission type 0xFE, with the period in ms.
  • The other messages are sent when the application sends them.

The signals follow the layout that the DBC and the generated C and Rust use, so the PDO carries the same bytes. A message with a wire format maps its fields at their bits, with dummy entries over the gaps.

What a classic PDO cannot carry is left out, and the notes say so:

  • a frame over 8 bytes (CANopen FD, CiA 1301, is not written);
  • a big-endian field;
  • a field whose bit count no CANopen type has, such as a 4-bit field (there is no UNSIGNED4).

The model has no CANopen identity. Give it on the binary, or on its processor:

binary pump_fw "pump-fw" on pumpb {
  attr canopen_node_id = 5      // [DeviceComissioning] NodeID
  attr vendor_id = 0x1234       // 0x1018 sub 1, VendorNumber
  attr product_code = 42        // 0x1018 sub 2
  attr revision = 1             // 0x1018 sub 3
  …
}

Without them the EDS says 0, and the notes name what is missing. For example, on examples/can-bench:

note: can_bus/gw: no vendor_id, product_code, canopen_node_id in the model — the EDS says 0; write `attr vendor_id = …` on binary gw

A bit rate that is not one of CANopen's is noted too. CANopen's rates are 10, 20, 50, 125, 250, 500, 800 and 1000 kbit/s.

Checked: each file is read with the canopen Python package, both its object dictionary and the PDO maps it builds from it.

J1939 (--target j1939)#

For each CAN bus that carries frames with a 29-bit id, it writes j1939/<bus>.json, and the same as Markdown tables in j1939/<bus>.md.

In J1939-21, the identifier holds everything about a parameter group. So the target reads it from the route's id:

  • the priority;
  • the PGN, with the data page and the extended data page;
  • the PDU format: PDU 1 with its destination address, or PDU 2 with its group extension;
  • the source address.

Then it adds the group's length, its rate (the message's period), the binary that sends it and the one that receives it.

Each signal is an SPN, with the columns of J1939-71:

  • its start position, as byte.bit, both counted from 1;
  • its length in bits;
  • its resolution and offset (a wire format's scale and offset);
  • its unit and range.

The SPN number is not in the identifier, so you write it on the field:

message EEC1 "Electronic engine controller 1" {
  attr pgn = 61444             // checked against the route's id
  period 10ms
  size 8
  torque_mode: uint8 bits 4 range 0..15 { attr spn = 899 }
  speed:       float32 unit rpm range 0..8031 at 24 bits 16 scale 0.125 { attr spn = 190 }
}
…
route pump.engine -> gateway.engine over can_bus { id 0x0CF00405 }

A signal without attr spn is listed with "spn": null, and the notes name it. The notes also name:

  • a frame over 8 bytes (it goes by the transport protocol, TP.BAM or TP.CMDT);
  • a big-endian field;
  • a binary that sends from more than one source address, or from another address than its attr j1939_address;
  • a PGN that two binaries send;
  • the frames with 11-bit ids on the bus, which are left out.

Checked against cantools.j1939: it reads the same priority, PGN and source address from the same id.

It writes the message catalog as a MAVLink message-definition file, the format mavgen reads: mavlink/<project>.xml. The file is a dialect of its own. It includes nothing, so the model's HEARTBEAT does not clash with the one in common.xml.

  • Each message with a fixed layout is a <message>. Each signal of that layout is a <field>:
    • a struct is flattened (header_seq);
    • an array of scalars is one array field (float[4]);
    • an enum is a uint8_t with enum="MODE".
  • Each enum the messages use is an <enum>, with entries such as MODE_RUN.
  • A field's unit is written when MAVLink's schema knows it: m/s, degC, rad, %. The model's °, °/s and m/s² become deg, deg/s and m/s/s. Another unit goes into the field's description.
  • The message's description holds its name or description, and its period.

The MAVLink id is not in the model. Give it with attr mavlink_id = 30 on the message. Messages without one are numbered from 65536, in the catalog's order. That is above every range the MAVLink registry gives to dialects, so they clash with none. The notes list them: give them their real ids before two systems rely on them.

A payload over 255 bytes, or with more than 64 fields, is left out, and the notes say so.

Checked with mavgen --strict-units, which checks the schema and the units. Its C headers are compiled and a message is packed. Its Python round-trips a message.

SOME/IP (--target someip)#

It writes a service description as JSON: someip/<deployment>.json. It holds what FIBEX and ARXML hold: the services, their methods, events and event groups, and where each instance runs. It also holds the data types the payloads use.

  • A service per interface that a port provides:
    • Each operation is a method. It is request/response when it returns something, and fire-and-forget when it does not.
    • The interface's messages that the provider sends are events, in one event group. An event is cyclic at the message's period, or else sent on change.
    • The messages the provider only receives are fire-and-forget methods.
  • A service per out port on an Ethernet link that sends messages and takes part in no interface. Its messages are events.
  • Instances, for each service: the part that provides it, the binary and processor it runs on, and the parts at the other end of its links.
  • dataTypes: the structs, enums and unions the payloads use. Scalars are named as SOME/IP names them (sint16, float32, boolean).

The ids are not in the model. Give them with:

attr on
someip_service_id the interface (or the port)
someip_instance_id the providing port
someip_method_id an operation
someip_event_id a message
someip_major_version, someip_minor_version the interface

Without them, services are numbered from 0x1000, methods from 0x0001 and events from 0x8001, in the model's order. The notes list each one. An interface provided on a port that is not on IP (a CAN port) is noted.

SOME/IP serializes big-endian. The codecs archec generates for CAN are little-endian, so they are not this wire format.

OpenAPI (--target openapi)#

It writes an OpenAPI 3.1 document for each component definition whose ports have HTTP handlers, such as handler GET "/api/v1/readings/{node}" -> reading. The file is openapi/<Component>.json.

  • Each handler is an operation:
    • its operationId is the function's name;
    • its summary is the function's name in quotes;
    • its description is the handler's, or else the function's;
    • it is tagged with the port.
  • {node} in the path is a path parameter, typed by the function's parameter of that name.
  • The other parameters are the query (GET, DELETE, HEAD), or a JSON body (POST, PUT, PATCH). They are required unless optional.
  • The function's return type is the 200 answer, as JSON. With no return type, the answer is 204.
  • servers lists each part of that definition with prop hostname = "…" (or attr url = "…").

The schemas are JSON Schema, shared with AsyncAPI:

  • A struct is an object with its fields required (an optional one is not), and no other fields.
  • An enum is its literal names. x-values gives their numbers.
  • A fixed array has its size as both min and max items.
  • An integer has its type's bounds, or its range. A float has its range.
  • A unit is x-unit.
  • An alias type (type Speed = float32 unit m/s range 0..70) is its scalar, with its unit and range.
  • A union is a oneOf of {case, value}.

Checked with openapi-spec-validator.

AsyncAPI (--target asyncapi)#

It writes an AsyncAPI 3.0 document for each system, or for the deployment's system when there is one: asyncapi/<system>.json.

  • A channel per port that sends over a link. Its address is the link's or the port's topic. Without one, it is <part>/<port>, as the ROS 2 nodes name it. Its messages are what the links carry.
  • A send operation for the part that publishes, and a receive operation for each part at the other end.
  • A server per bus a route takes, when there is a deployment. Its protocol comes from the port type (can, serial, udp, mqtt…), with its bit rate in the description. The channel is on that server.
  • The messages are in components.messages, and their payloads in the shared schemas.

Checked against the official AsyncAPI 3.0.0 JSON schema (@asyncapi/specs with ajv), when ARCHE_NODE_TOOLS names a folder where they are installed.

Build systems#

The build targets wrap the C that --target c writes, where it is. They refer to ../../c from their own folder, so write both into the same output folder:

archec gen main.arche --target c -o gen
archec gen main.arche --target cmake -o gen
cmake -S gen/cmake -B build && cmake --build build      # every C binary, on this host

A binary in Rust (--target embassy) or Go (--target go) has its own build, with cargo or go. These targets note it and leave it out.

CMake (--target cmake)#

  • cmake/<binary>/CMakeLists.txt builds two things:

    • the binary's model, as a static library <binary>_model: the types, its parts, its wiring and its drivers;
    • the executable, with its main.c and its runtime. The runtime is POSIX on a host, and FreeRTOS when cross-compiled. ARCHE_RT chooses.
  • cmake/<binary>/toolchain.cmake, for a binary on a microcontroller:

    • GCC, by its prefix (arm-none-eabi-, riscv32-esp-elf-, xtensa-esp32-elf-);
    • the core's flags, from its core-type. For example Cortex-M4F gives -mcpu=cortex-m4 -mthumb -mfpu=fpv4-sp-d16 -mfloat-abi=hard.
  • cmake/<binary>/port.cmake is written once, and then it is yours. It names:

    • the FreeRTOS kernel (FREERTOS_KERNEL_PATH), and its port for the core (GCC_ARM_CM4F…);
    • the chip's startup code and linker script.

    Without a kernel, only the model library is built, with a warning.

  • cmake/CMakeLists.txt builds every C binary for this host. The microcontrollers' binaries are included too: their logic runs on the POSIX runtime, as their Makefile builds it. This is what CI builds and runs.

Some cores have no upstream FreeRTOS port, such as Xtensa (the ESP32's). This is noted. ESP-IDF brings its own port, which --target platformio uses.

Checked:

  • the rig's three C binaries are configured, built and run with CMake on the host;
  • the sensor node's model library is cross-compiled for its STM32G4 with the toolchain file, when an arm-none-eabi-gcc with newlib is installed.

Zephyr (--target west)#

For each C binary on a board Zephyr knows, it writes a Zephyr application in west/<binary>/. The boards are the store's Nucleo, nRF52840 DK, Pico and ESP32 boards.

  • CMakeLists.txt adds the binary's C to Zephyr's app, on Zephyr's POSIX API (arche_rt_posix.c).
  • prj.conf is written once, and then it is yours. It turns on CONFIG_POSIX_API and the drivers that the overlay's nodes need (CONFIG_CAN, CONFIG_SERIAL…).
  • boards/<board>.overlay is what --target devicetree writes for that board: the linked CAN, UART, I2C and SPI on their pins, and an alias per bus that the C opens.
  • west.yml is a manifest. It pins Zephyr (v4.1.0), and allows only the modules the board's HAL needs (cmsis, hal_stm32…).
west init -l gen/west/node_fw && west update && west build -b nucleo_g474re gen/west/node_fw

Checked: west's own Python package validates and loads the manifest.

PlatformIO (--target platformio)#

For each C binary on a board PlatformIO knows, it writes platformio/<binary>/platformio.ini. The file gives:

  • the platform and board: ststm32/nucleo_g474re, espressif32/esp32dev, nordicnrf52/nrf52840_dk, raspberrypi/pico;
  • a framework with threads under the runtime:
    • Zephyr on STM32 and nRF52, with zephyr/CMakeLists.txt, prj.conf and app.overlay;
    • ESP-IDF's FreeRTOS on ESP32;
    • the FreeRTOS of the Earle Philhower Arduino core on the RP2040;
  • the generated C as its sources: src_dir = ../../c, and a build_src_filter of the binary's files.

The notes say what is left to do by hand. For example, ESP-IDF starts app_main, not main.

Checked: pio project config reads each file. The files are not built, because that downloads each platform's toolchain.

CI (--target ci)#

It writes a pipeline for GitHub Actions (ci/.github/workflows/archec.yml), and the same for GitLab CI (ci/.gitlab-ci.yml). Copy ci/ to the root of the repository. The jobs:

  • model: archec check, then archec ci with its JUnit report. The report is uploaded on GitHub, and is a test report on GitLab.
  • generated: archec gen --check for each target the deployment uses, in a matrix. That is c and cmake for C binaries, embassy for Rust ones, go, and dbc for its CAN buses. It shows that the generated code committed is what the model gives.
  • build-c: the CMake project of --target cmake, built on the host. Each binary runs for a second.
  • build-<binary>, for each Rust binary: cargo build --release for its chip's target (thumbv7em-none-eabihf…, or the Espressif toolchain for an Xtensa ESP32).

Three variables at the top set where things are. Edit them when the project is not at the root of the repository:

  • ARCHE_MAIN: the model's main file, from the repository root;
  • ARCHE_GEN: the folder the generated code is committed in (generated);
  • the deployment.

Checked: both files are valid YAML, and valid against SchemaStore's schemas for GitHub workflows and GitLab CI (check-jsonschema).

The lab#

Flash and monitor (--target flash)#

For each binary on a microcontroller, it writes two executable scripts in flash/<binary>/.

flash.sh [image] flashes the binary. It uses the first of its chip's tools that is installed, or the one TOOL= names:

Chip Tools, in order
STM32 probe-rs (--chip STM32G474RETx), OpenOCD over the ST-LINK (target/stm32g4x.cfg)
nRF52 probe-rs (nRF52840_xxAA), nrfjprog (the ELF made a HEX), OpenOCD over a J-Link
RP2040 picotool (load -x), probe-rs, OpenOCD over a CMSIS-DAP probe
ESP32 espflash, esptool (the ELF made an image, written at 0x10000)

The image is $1. Without it, the script takes the image where the build puts it:

  • for Rust, cargo's embassy/<binary>/target/<triple>/release/<crate>;
  • for C, build-<binary>/<binary>.elf: the cross build of --target cmake, from the folder archec gen wrote into.

monitor.sh reads the binary's trace for ARCHE_DURATION seconds (10 by default) into ARCHE_TRACE (<binary>.trace). Then it runs archec timing and archec live --replay on the trace, with the deployment.

  • A C binary's trace comes from the board's UART: its virtual COM port when it has one. ARCHE_PORT names it (/dev/ttyACM0 by default). It is read with pyserial, at the bitrate of the link on it (or else 115200).
  • A Rust binary's trace is defmt over RTT, through probe-rs attach.
  • The model is ARCHE_MAIN: the main file's name, from where the script runs.

The trace is what archec gen --trace makes the firmware write:

  • the C, built with -DARCHE_TRACE_LIVE, with its c/arche_trace_port.c writing to that UART;
  • the Rust, built with --features trace.

archec timing skips whatever comes before arche-trace on a line, so a log prefix does no harm.

A binary on Linux is not flashed, and this is noted: copy it there.

Checked:

  • each script parses (sh -n);
  • flash.sh, run against a stand-in probe-rs, gets the chip and the image;
  • OpenOCD reads the interface, transport and target of each OpenOCD line;
  • monitor.sh reads arche-trace lines written to a pseudo-terminal, and archec timing and archec live measure the thread from them.

It has not been run on a board here.

The bench (--target bench)#

It writes the configuration of the deployment's hardware test rack, from its platform. A first target shows it on examples/can-bench.

bench/<deployment>.yaml has:

  • buses: each bus, with:
    • its type (can, can-fd, uart, ethernet);
    • its bitrate, and its data_bitrate from attr data_bitrate on a CAN FD bus (the language has no word for the data phase);
    • the rack's interface on it: can0, can1… in order, or /dev/ttyUSB0… for a serial bus;
    • the nodes on it, with their binaries;
    • for a CAN bus, every frame it carries: id, message, length, sender, receiver and period.
  • duts (devices under test): each binary on a microcontroller, with its board, its chip, its flash and monitor scripts (--target flash) and its console UART.
  • hosts: the binaries on Linux.
  • power: each supply (an out … : power port) and what it feeds.
  • io: the GPIO, PWM and analog links.

What the model cannot say is null, such as the channels of the supplies and the IO on your rack. The notes say what else is missing, such as a bitrate or a power link.

bench/setup_<bus>.sh, for each CAN bus, sets up its SocketCAN interface at the bus's bitrate (restart-ms 100, and dbitrate … fd on for CAN FD). When the interface given is named vcan…, it makes a virtual bus instead.

bench/bench_<bus>.py, for each CAN bus, uses python-can:

  • It lists the frames that go by, named after the model.
  • It fails (exit 1) when a periodic frame is missing, or off its period by more than half.
  • --as <binary> stands in for a binary that is not on the rack: it sends that binary's frames at their period.
  • --send <MESSAGE> sends one message.
  • --interface virtual runs it all in one process, without hardware.
sh gen/bench/setup_can_bus.sh can0
python3 gen/bench/bench_can_bus.py --channel can0 --duration 10 --as gw_app

Checked:

  • the YAML loads, and the setup script parses;
  • the python-can script on its virtual bus sees the sensor's and the pump's frames at their periods when it stands in for them;
  • it fails with the pump's frame missing when it does not.

Limits#

  • Each target writes only what the model says. A value the language has no word for (a J1939 SPN, a MAVLink id, a CANopen identity, SOME/IP ids) comes from an attr. Without it the target writes 0, null or a numbered stand-in, and the notes say so.
  • CANopen: no CANopen FD (CiA 1301). A frame over 8 bytes, a big-endian field, or a field whose bit count no CANopen type has, is left out of the PDOs.
  • J1939: frames with 11-bit ids are left out. A frame over 8 bytes needs the transport protocol, which is noted but not written.
  • MAVLink: a payload over 255 bytes or with more than 64 fields is left out.
  • SOME/IP: the CAN codecs archec generates are little-endian, so they are not SOME/IP's wire format.
  • The build targets build C only. A Rust or Go binary is left out, with a note. A core with no upstream FreeRTOS port (Xtensa) is noted.
  • PlatformIO files are read by pio project config in the tests, not built. ESP-IDF starts app_main, not main: that is left to do by hand.
  • The flash and monitor scripts have not been run on a board in archec's tests. A binary on Linux is not flashed.
  • The bench file cannot say the channels of your rack's supplies and IO: they are null.