βββ MyProject.Checkout.Empty_Cart/
β βββ ...
```
Test names are sanitised to filesystem-safe directory names (spaces become underscores, path separators are removed). External file references are resolved relative to the directory containing the result file; if the referenced file doesn't exist, the artefact is recorded as skipped with reason `missing-source` and nothing is copied. References that would escape the base directory via `..` are blocked β there is no way to overwrite files outside `DIR`.
### `--raw-html` vs default
By default, log messages tagged as HTML (Robot Framework's `Log ... HTML`) are converted to a plain-text approximation: `
` becomes a placeholder like `_(embedded image β see artefacts: no alt text)_` and the actual bytes are decoded and stored separately for `--extract`. `link` becomes a markdown-style link.
With `--raw-html` the conversion is skipped β the original HTML markup is preserved verbatim and no artefacts are extracted. Use this when you want to feed the output into something that renders HTML itself.
### `--keyword-info` and `--suite-info`
Both flags add **structured metadata** to the log output and are independent β you can pass either or both. They default to off so the standard `log` view stays compact.
**`--keyword-info`** β for every executed `KEYWORD` / `SETUP` / `TEARDOWN`, the keyword's own `[Documentation]`, `[Tags]` and `[Timeout]` (taken from the keyword definition, not the call) are emitted. In TEXT the renderer prints them as indented `[Documentation] / [Tags] / [Timeout]` lines under the keyword header; in JSON they appear as `doc` / `tags` / `timeout` keys on the matching body-item entry. Fields whose underlying setting is empty are dropped, so `BuiltIn.Log` (which has a docstring but no tags) shows only `doc`.
**`--suite-info`** β tests get grouped under one `Suite:` header per parent suite. The header carries the suite's `fullName`, source path, executed status, plus the suite's `Documentation` (with `...` continuation lines for multi-line docs) and one `[Metadata]` line per key. In JSON, `LogResult.suites` is populated with one entry per surviving suite, and every test gains a `suite` field cross-referencing the parent suite by `fullName`.
Combine both for a maximal view that mirrors the structure of the original `.robot` files:
```bash
robotcode results log --suite-info --keyword-info
```
Example TEXT output:
```
Suite: MyProject.Login (tests/login.robot) PASS
[Documentation] Exercises the login flow against the staging environment.
[Metadata] OwnerTeam = identity-squad
Test: MyProject.Login.Bad Password (tests/login.robot:42) FAIL
[SETUP] Open Browser PASS
[Documentation] Launches a fresh Chromium with our shared profile.
[Tags] browser setup
...
```
### Recipes
```bash
# Drill into one failing test
robotcode results log -bl MyProject.Login.Bad_Password
# Walk only the FAIL-level messages of every failed test
robotcode results log --failed --level FAIL
# Top-level keyword calls only, with everything nested collapsed
robotcode results log --max-depth 1
# Pull screenshots out of a failed run
robotcode results log --failed --extract ./extracted
# Diagnose suites that broke before they even ran
robotcode results log --execution-messages --level WARN
# Full structured view β suite headers + per-keyword doc/tags/timeout
robotcode results log --suite-info --keyword-info
```
## `stats` β aggregate by tag, suite, or status
`stats` answers questions like "which tag has the worst pass rate" or "which suite is consuming most of my CI time". You pick one or more aggregation dimensions and `stats` emits one table per dimension.
```bash
robotcode results stats # default: by status
robotcode results stats --by tag
robotcode results stats --by suite
robotcode results stats --by tag --by suite # two sections, in the requested order
```
Output:
```text
By Tag:
NAME TOTAL PASS FAIL SKIP ELAPSED
regression 24 19 5 0 4m 12s
smoke 18 18 0 0 47s
slow 7 5 1 1 2m 03s
bug-1842 2 0 2 0 8s
By Suite:
NAME TOTAL PASS FAIL SKIP ELAPSED
MyProject.Checkout 14 10 4 0 3m 11s
MyProject.Login 12 12 0 0 1m 02s
MyProject.Search 8 7 0 1 52s
```
### Flag reference
| Flag | Effect |
|---|---|
| `--by tag\|suite\|status` | Aggregation dimension. Repeatable for multiple sections. Default: `--by status`. |
| `--sort name\|total\|failed\|elapsed` | Order groups within each section. Default: `failed` descending β most painful first. |
| `--top N` | Keep at most `N` groups per section. The dropped count is reported as a footer line. |
### Aggregation rules
- **`--by tag`**: a test with N tags counts in N buckets. Tags that differ only in case, whitespace or underscores (`Bug 1`, `bug_1`, `Bug1`) are treated as the same tag and merge into one bucket, displayed in normalised form. Tests without tags drop out β there is **no** "(untagged)" bucket. The elapsed time per group is the **sum** across all matching tests (not the average).
- **`--by suite`**: groups are keyed by the test's parent suite **full longname**, not the leaf name. `MyProject.Login` and `MyProject.Checkout.Login` are distinct groups.
- **`--by status`**: groups are the literal status strings `PASS`, `FAIL`, `SKIP`, `NOT RUN`. Empty buckets are omitted.
Filters and `--search` are applied **before** aggregation. That makes questions like "for the smoke subset, which suite has the most failures?" a single command:
```bash
robotcode results stats --include smoke --by suite --sort failed
```
### Recipes
```bash
# Worst tags by failure count (default sort)
robotcode results stats --by tag
# Five worst tags only
robotcode results stats --by tag --top 5
# Where is the CI time going?
robotcode results stats --by suite --sort elapsed
# Status breakdown for one specific tag
robotcode results stats --include flaky --by status
```
## `diff` β compare two runs
`diff` matches tests across two result files by **full longname** and classifies the differences. Use it for regression triage, to spot flaky tests, or to confirm that a fix has actually landed.
```bash
robotcode results diff baseline/output.xml # current is auto-discovered
robotcode results diff baseline/output.xml current/output.xml # explicit pair
```
### Output sections
| Section | Tests that β¦ |
|---|---|
| `new failures` | passed in baseline, fail in current β regressions |
| `new passes` | failed or skipped in baseline, pass in current β fixes |
| `status changes` | otherwise changed status (e.g. `SKIP β FAIL`, `FAIL β SKIP`) |
| `added` | exist only in current (added since baseline) |
| `removed` | exist only in baseline (deleted/renamed since baseline) |
Tests that have the same status in both runs are **not** reported β `diff` shows changes only.
### Flag reference
| Flag | Effect |
|---|---|
| `--only new-failures\|new-passes\|status-changes\|added\|removed` | Restrict output to these categories. Repeatable. |
| `--message-chars N` | Truncate the per-entry messages to `N` chars. Default: `120`. |
| `--full-paths` | Absolute source paths. |
The standard `--status / -i / -e / -s / -t / -bl / -ebl / --search / --search-regex` filters apply to **both** baseline and current before matching. This lets you scope the comparison to a tag, sub-suite, or pattern without re-running anything:
```bash
robotcode results diff baseline.xml --include smoke
robotcode results diff baseline.xml --suite "*.Checkout.*"
robotcode results diff baseline.xml --search Timeout
```
### Why `diff` always exits 0
`diff` exits 0 even when there are new failures, so it composes cleanly with other tools. For a regression gate, drive the exit code from the structured output β see [CI recipes](#ci-recipes) in the JSON reference.
### When tests have been renamed
`diff` matches on `fullName`. A rename therefore looks like one `removed` entry plus one `added` entry β even if the test bodies are identical. There is currently no fuzzy-match mode; if rename detection is important to you, keep test names stable or post-process the JSON.
### Recipes
```bash
# Regression triage: every new failure, full message, with sources
robotcode results diff baseline.xml --only new-failures --message-chars 0 --full-paths
# Diff only the smoke subset
robotcode results diff baseline.xml --include smoke
# Compare two CI runs (e.g. main vs feature branch)
robotcode results diff main/output.xml branch/output.xml
```
## Filters
Every subcommand accepts the same filter set. Filters combine with **AND** β every test must satisfy every filter to make it through. Within a single repeatable flag (`--status`, `--include`, etc.) the matches combine with **OR** in the way Robot Framework normally treats those options.
| Flag | Behavior |
|---|---|
| `--status pass\|fail\|skip\|not-run` | Repeatable; tests whose status is in the chosen set. |
| `--failed` / `--passed` / `--skipped` | Shorthands for `--status fail` / `--status pass` / `--status skip`. Additive with `--status` and with each other (OR semantics). Available on `show`, `log`, `stats`. |
| `-i/--include TAG_PATTERN` | Robot tag-pattern syntax (see below). Repeatable. |
| `-e/--exclude TAG_PATTERN` | Same syntax, exclusion side. Repeatable. |
| `-s/--suite GLOB` | Match against the suite's full longname. Repeatable. |
| `-t/--test GLOB` | Match against the test's full longname. Aliased as `--task`. Repeatable. |
| `-bl/--by-longname NAME` | Exact longname match (no glob expansion). Repeatable. |
| `-ebl/--exclude-by-longname NAME` | Exclusion variant of `--by-longname`. Repeatable. |
### Tag-pattern syntax (`-i`/`-e`)
Tag patterns follow Robot Framework's own rules:
- A plain tag (`smoke`) matches tests carrying that tag.
- `AND` (uppercase) requires multiple tags: `smokeANDregression` matches only tests that have **both** tags.
- `NOT` excludes: `smokeNOTslow` matches `smoke` minus `slow`.
- Tags are matched case-insensitively, and whitespace and underscores are ignored β `smoke_test`, `Smoke Test` and `smoketest` all refer to the same tag. Other punctuation, including hyphens, is significant: `bug-123` and `bug123` are distinct.
- Glob characters `*` and `?` work inside a tag pattern: `bug-*` matches any tag starting with `bug-`.
You can repeat `-i` to OR multiple patterns: `-i smoke -i regression` selects tests that carry **either** `smoke` **or** `regression`.
### Suite / test globs
`-s` / `-t` accept shell-style glob patterns matched against the full longname:
- `*` matches any sequence of characters (including `.`, so `*.Login.*` matches at any depth).
- `?` matches a single character.
- Matching is case-insensitive and applied to the full longname (`MyProject.Login.Bad Password`).
- Repeat for OR: `-s "MyProject.Login.*" -s "MyProject.Checkout.*"`.
Quote globs in the shell β `--suite "*.Login.*"` and `--test "Test ?"` need quotes so the shell doesn't expand them against the local filesystem.
### `--by-longname` vs `--suite` / `--test`
- `-s` / `-t` are **glob** matches and accept patterns.
- `-bl` / `-ebl` are **exact** matches against the full longname β no glob expansion. They mirror `robotcode robot --by-longname` so you can hand the same names to `robot` and `results`.
Use `-bl` when you have a precise name to filter against (often pasted from the failure list); use `-s/-t` when you want pattern matching.
## Search
`--search` and `--search-regex` are **mutually exclusive** (passing both is a usage error). They apply across:
- the test's `name` and full longname
- the test's failure message
- the test's tags
- the test's `[Documentation]`, `[Template]` and `[Timeout]`
- every keyword call's name, arguments and assignments within the test body
- every executed keyword's `[Documentation]`, `[Tags]` and `[Timeout]` (taken from the keyword definition; result-tree only)
- every log message's text
- any **ancestor suite**'s `Documentation` or `Metadata` (a hit on a suite-level field keeps every test underneath it)
A test matches if **any** of those targets matches. Once `log` decides a test matches, all of its body items are emitted as usual β the search is a test-level filter, not a per-message filter.
| Flag | Semantics |
|---|---|
| `--search TEXT` | Case-insensitive substring match. No metacharacters; `[` and `*` are literal. |
| `--search-regex PATTERN` | Regular expression, **case-sensitive** by default. Use `(?i)` at the start of the pattern for case-insensitive matching. |
When matching against **tags** specifically, both the pattern and each tag are normalised the way Robot Framework normalises tags (lowercase, whitespace and underscores ignored). So `--search "bug 1"` matches tests tagged `bug_1`, `Bug1`, or `BUG 1`. Other targets β name, message, keyword arguments, log text β are matched literally without normalisation.
```bash
robotcode results log --search "TimeoutError"
robotcode results log --search-regex 'AssertionError.*expected: \d+'
robotcode results log --search-regex '(?i)login' # case-insensitive
```
### Anchors and alternation
The regex is matched against each target string individually, **not anchored** to the whole string β a pattern like `Login` matches any target that contains `Login` somewhere. Use `^` and `$` if you want anchored matches. Note that anchors apply per target string, so `^Login` matches a test whose full longname is `Login.Bad Password` but also a keyword named `Login With Token` somewhere inside the test body.
### Highlighting in the terminal
On `log` (and `show`), matches are highlighted inline in the TEXT output with a yellow background so you can scan visually. Structured output (JSON/TOML) is unchanged β searching only filters, it doesn't add markup.
### Search on `summary` and `stats`
`--search` works on these too, but its effect is subtle: the matching tests narrow the input set, and the counts / aggregations are computed against that narrowed set. So `robotcode results summary --search Login` is "how is the Login subset doing" and `robotcode results stats --by tag --search Login` is "which tags are involved in the Login subset".
### Invalid patterns
An unparseable regex (e.g. `[unclosed`) yields a usage error pinpointing where the pattern failed to compile. The command exits non-zero before reading the result file.
## Tips for terminal use
- **TEXT output is markdown.** On a coloured TTY the markdown is rendered to themed ANSI via `rich` (and paged if longer than your terminal); on pipes or with `--no-color` the raw markdown goes through verbatim β paste it into a PR description, a Slack message, or feed it straight to an LLM.
- **Quote your globs.** `--suite "*.Login.*"` and `--test "Test ?"` need quotes so the shell doesn't expand them against the local filesystem first.
- **Filters apply before aggregation.** `stats`, `summary` and `diff` all run the filter pipeline first, so `--search Login` followed by `--by tag` gives you tag stats over the Login subset, not all tags whose name happens to contain "Login".
- **`-bl` is for exact names, `-t` is for patterns.** When you copy a failing-test name out of the failure list, `-bl` is usually what you want β no need to escape glob characters.
- **Save your baselines.** Archive `output.xml` from your green main branch (or main-CI run) and feed it to `diff` from feature branches β that's the fastest way to catch regressions before review.
- **`diff` doesn't fail the build.** It always exits 0. If you want a regression gate, see the [CI recipes](#ci-recipes) in the JSON reference.
---
# JSON reference
When you set `-f json` (or `-f json_indent` for pretty-printed, or `-f toml`) between `robotcode` and the subcommand, every result command emits structured data instead of the terminal-formatted text. This is what CI pipelines, scripts and AI agents consume.
```bash
robotcode --format json results summary
robotcode --format json results show --failed --sort elapsed
robotcode --format json results log --failed
```
This part of the guide is the schema reference: what fields each subcommand emits, when they're present, and how to consume them safely.
## Schema rules
A few rules hold across every subcommand:
- **camelCase keys.** `fullName`, `elapsedSeconds`, `messagesCount`, etc.
- **ISO-8601 timestamps** with microsecond resolution.
- **Optional fields are omitted, not `null`.** A `null` literal never appears in the output. If a field has no value (e.g. `failed` without `--failed`, `tags` on an untagged test), the key is simply absent from the JSON object.
- **Scalar fields with a meaningful zero are always emitted.** `counts.failed = 0`, `truncated = 0`, `elapsedSeconds = 0.0` all stay in the object. Only optional fields disappear when unset.
- **Empty array β missing field.** Sections that "exist but are empty" come through as `[]`; sections that were never requested (e.g. anything excluded by `diff --only`) are absent. The two are different and your queries should handle both, typically via `// []` in jq.
- **Stability.** Fields are appended over time, never renamed or removed in place. New fields are additive; existing consumers keep working.
## `summary` JSON
```json
{
"file": {
"source": "results/output.xml",
"relSource": "results/output.xml"
},
"status": "FAIL",
"counts": { "total": 42, "passed": 37, "failed": 2, "skipped": 3, "notRun": 0 },
"elapsedSeconds": 83.4,
"startTime": "2026-05-15T08:11:02",
"endTime": "2026-05-15T08:12:25",
"failed": [
{
"name": "Bad Password",
"fullName": "MyProject.Login.Bad Password",
"suite": "MyProject.Login",
"status": "FAIL",
"message": "AssertionError: Expected 'Login failed' but got 'Internal error'",
"tags": ["smoke"],
"elapsedSeconds": 0.234,
"startTime": "2026-05-15T08:11:04",
"source": "tests/login/test_login.robot",
"relSource": "tests/login/test_login.robot",
"lineno": 42
}
],
"messagesCount": { "INFO": 318, "WARN": 7, "FAIL": 2 }
}
```
Field notes:
- `failed` only appears when `--failed` was passed.
- `messagesCount` aggregates log messages by level (`TRACE` / `DEBUG` / `INFO` / `WARN` / `ERROR` / `FAIL`). Only levels with at least one message appear β empty buckets are omitted, not emitted as `0`.
- `executionMessagesCount` (parser / discovery errors that fired outside of test execution) appears **only** when there were any.
- `filtersApplied` (see [below](#filtersapplied)) appears when any filter was passed.
## `show` JSON
```json
{
"file": { "source": "results/output.xml" },
"counts": { "total": 42, "passed": 37, "failed": 2, "skipped": 3, "notRun": 0 },
"tests": [
{
"name": "Bad Password",
"fullName": "MyProject.Login.Bad Password",
"suite": "MyProject.Login",
"status": "FAIL",
"message": "AssertionError: Expected 'Login failed' β¦",
"tags": ["smoke", "regression"],
"elapsedSeconds": 0.234,
"startTime": "2026-05-15T08:11:04",
"source": "tests/login/test_login.robot",
"lineno": 42
}
],
"truncated": 0,
"elapsedSeconds": 83.4,
"startTime": "2026-05-15T08:11:02",
"endTime": "2026-05-15T08:12:25"
}
```
Field notes:
- `tests[]` is always present, possibly empty.
- `tags` is always emitted for tests that have any (and absent for untagged tests), in normalised form (`Bug 1`, `bug_1`, `Bug1` all come through as `"bug1"`). The `--tags` flag in TEXT mode is render-only β it doesn't affect the JSON.
- `truncated` is the number of tests dropped by `--top N`; `0` when nothing was dropped.
- The order of `tests[]` reflects `--sort` and `--reverse`.
## `log` JSON
```json
{
"file": { "source": "results/output.xml" },
"tests": [
{
"fullName": "MyProject.Login.Bad Password",
"status": "FAIL",
"message": "AssertionError: Expected 'Login failed' β¦",
"body": [ /* recursive tree of body items, see below */ ],
"artifacts": [ /* aggregated artifact refs for the whole test */ ],
"source": "tests/login/test_login.robot",
"lineno": 42,
"elapsedSeconds": 0.234,
"startTime": "2026-05-15T08:11:04",
"suite": "MyProject.Login"
}
],
"suites": [
{
"fullName": "MyProject.Login",
"name": "Login",
"status": "FAIL",
"doc": "Exercises the login flow against staging.",
"metadata": { "OwnerTeam": "identity-squad" },
"source": "tests/login.robot",
"elapsedSeconds": 12.7,
"startTime": "2026-05-15T08:11:02"
}
],
"executionMessages": [ /* only with --execution-messages */ ],
"extractDir": "./extracted",
"extractedCount": 7
}
```
The `body` array of each test is a recursive tree of body items. Field notes:
- `extractDir` / `extractedCount` appear only with `--extract DIR`.
- `executionMessages` appears only with `--execution-messages`.
- `suites` and the per-test `suite` cross-reference appear only with `--suite-info`. One `suites` entry per parent suite that has at least one surviving test, in traversal order. Empty `metadata` / `doc` are dropped.
- Artefact entries carry `kind` (`"image"` | `"file"`), `src`, and β when extraction happened β `extractedTo` (absolute path of the written file). Failed extractions get a `skippedReason` instead (e.g. `"missing-source"`, `"target-traversal"`).
### Body-item types
Every body-item entry has a `type` field. The vocabulary tracks Robot Framework's own body-item taxonomy:
| `type` | What it represents | Notable fields |
|---|---|---|
| `KEYWORD` | A keyword call | `name`, `owner`, `args`, `assign`, `body` |
| `SETUP` / `TEARDOWN` | Suite/test setup or teardown keyword | same as `KEYWORD` |
| `FOR` | `FOR` loop | `flavor` (`IN`/`IN RANGE`/`IN ENUMERATE`/`IN ZIP`), `body` (iterations) |
| `ITERATION` | One iteration of a `FOR` / `WHILE` loop | `assign` (the loop variable values), `body` |
| `WHILE` | `WHILE` loop | `condition`, `body` |
| `IF` / `ELSE IF` / `ELSE` | Conditional branches | `condition` (on `IF` / `ELSE IF`), `body` |
| `TRY` / `EXCEPT` / `FINALLY` | Exception handling | `patterns`, `patternType` (on `EXCEPT`) |
| `VAR` | `VAR` assignment statement | `assign` (variable name), `args` (value), `scope` |
| `RETURN` / `BREAK` / `CONTINUE` | Control-flow exits | `args` (values for `RETURN`) |
| `GROUP` | `GROUP` block | `name`, `body` |
| `ERROR` | Recorded execution error | `args` |
| `MESSAGE` | A log message | `level`, `text`, `timestamp`, `isHtml`, `artifacts` |
Every entry carries `status`, `elapsedSeconds` and `startTime` where applicable. The recursive `body` field is what makes the tree walkable β you can drill into a `FOR` loop's `ITERATION` and then into the `KEYWORD` calls inside each iteration.
With **`--keyword-info`**, every `KEYWORD` / `SETUP` / `TEARDOWN` entry additionally carries the keyword's own definition metadata:
| Field | When it appears |
|---|---|
| `doc` | Keyword has a `[Documentation]` setting (library keywords always do; user keywords only when written). |
| `tags` | Keyword has at least one `[Tags]` entry. |
| `timeout` | Keyword has a `[Timeout]` setting. |
Empty entries are dropped β a user keyword without `[Tags]` will not have a `tags` key.
## `stats` JSON
```json
{
"file": { "source": "results/output.xml" },
"sections": [
{
"dimension": "tag",
"groups": [
{
"name": "regression",
"counts": { "total": 24, "passed": 19, "failed": 5, "skipped": 0, "notRun": 0 },
"elapsedSeconds": 252.1
}
],
"truncated": 0
}
]
}
```
Field notes:
- `sections[]` has one entry per `--by` dimension, in the order they appeared on the command line.
- `dimension` is `"tag"` | `"suite"` | `"status"`.
- `groups[]` is sorted by `--sort` (default: `failed` descending) and capped by `--top N`.
- `elapsedSeconds` per group is the **sum** of test elapsed times in that group.
- `truncated` is the number of groups dropped by `--top N`.
## `diff` JSON
```json
{
"baseline": { "source": "baseline/output.xml" },
"current": { "source": "current/output.xml" },
"newFailures": [
{
"fullName": "MyProject.Login.Bad Password",
"baselineStatus": "PASS",
"currentStatus": "FAIL",
"currentMessage": "AssertionError: Expected 'Login failed' β¦",
"source": "tests/login/test_login.robot",
"relSource": "tests/login/test_login.robot",
"lineno": 42
}
],
"newPasses": [],
"statusChanges": [],
"added": [],
"removed": []
}
```
Field notes:
- Without `--only`, all five section arrays (`newFailures`, `newPasses`, `statusChanges`, `added`, `removed`) are always present β empty `[]` when nothing matched.
- With `--only`, sections you didn't list are **omitted entirely** from the JSON. Use `// []` in jq when accessing fields that may be missing.
- Each entry carries `fullName`, the relevant status fields, and (when they exist) the relevant messages. `baselineMessage` / `currentMessage` are omitted when the corresponding run had no failure message.
## `filtersApplied`
Every subcommand's JSON output includes a `filtersApplied` field when **any** filter was passed. It echoes back the filters the command actually saw:
```json
"filtersApplied": {
"status": ["fail"],
"include": ["bug1"],
"exclude": ["smokeANDregression"],
"suite": ["*.Login.*"]
}
```
Most filters come back verbatim. The two exceptions are `include` and `exclude`:
- A plain single-tag pattern is normalised the way Robot would (`BUG 1` β `bug1`, `Smoke_Test` β `smoketest`), so you see how the filter is actually being compared.
- A pattern containing an uppercase `AND` / `OR` / `NOT` operator is echoed verbatim, because each operand would need to be normalised individually and the surrounding structure carries deliberate meaning.
This is useful when piecing together a complex CI command β you can assert that the filter set ended up where you expected. When no filters were passed, the field is absent.
## CI recipes
A grab-bag of jq-based recipes. Pin `-f json` between `robotcode` and the subcommand to lock the format.
### Pass/fail the build
```bash
# Fail on any test failure
robotcode --format json results summary | jq -e '.counts.failed == 0'
# Fail only on new failures vs a checked-in baseline
robotcode --format json results diff baseline/output.xml \
| jq -e '(.newFailures // []) | length == 0'
# Allow a fixed budget of failures
FAILED=$(robotcode --format json results summary | jq '.counts.failed')
test "$FAILED" -le 3
```
### Slow-test report
```bash
# Top 20 slowest tests, CSV
robotcode --format json results show --sort elapsed --top 20 \
| jq -r '.tests[] | [.fullName, .elapsedSeconds] | @csv'
# Tag-level elapsed budget (minutes)
robotcode --format json results stats --by tag --sort elapsed \
| jq -r '.sections[].groups[] | [.name, (.elapsedSeconds / 60)] | @csv'
```
### Notification payloads
```bash
# One-line Slack-style message
robotcode --format json results summary \
| jq -r '"\(.status): \(.counts.passed)/\(.counts.total) passed in \(.elapsedSeconds / 60 | floor)m. \(.counts.failed) failed."'
# Diff summary against main
robotcode --format json results diff baseline.xml \
| jq -r '"+ \((.added // []) | length) added Β· - \((.removed // []) | length) removed Β· \((.newFailures // []) | length) regressions Β· \((.newPasses // []) | length) fixes"'
```
### Artefact gathering
```bash
# After a failed CI run: pull screenshots out for upload
robotcode results log --failed --extract ./extracted
# Then: zip the lot
zip -r artefacts.zip ./extracted
```
### Flakiness signal
```bash
# Tests that flipped state between two runs (either direction)
robotcode --format json results diff run-1.xml run-2.xml \
| jq '((.newFailures // []) + (.newPasses // [])) | map(.fullName)'
```
## Tips for scripting
- **Pin the output format.** Always say `-f json` (between `robotcode` and the subcommand). The TEXT format is meant for humans and may evolve.
- **Empty array β missing field.** Sections that exist but are empty come through as `[]`. Sections excluded by `diff --only` are absent. Use `// []` in jq to handle both.
- **Optional fields are absent, never `null`.** Test for presence of the key rather than for a `null` value; in jq use `(.failed // empty)`.
- **Trust `filtersApplied`.** If you're not sure your filters are being honoured, the field echoes them back β tag patterns in their canonical (normalised) form, everything else verbatim.
- **For AI agents:** when feeding tool output into an LLM, prefer the narrowest subcommand call you can make (`summary` over `show`, `show --top 5` over a full listing, `log --failed --level FAIL` over an unfiltered log). Each layer of filtering saves tokens and reduces the amount of reasoning the model has to spend on parsing.
---
URL: "/03_reference/cli"
LLMS_URL: "/03_reference/cli.md"
---
# Command Line Interface (CLI)
The `robotcode` CLI tool enables seamless interaction with Robot Framework through the command line, offering a comprehensive set of features for project analysis, debugging, and configuration management.
The CLI tool is designed to be straightforward and user-friendly, with a broad range of commands to support various use cases, including test execution, documentation generation, and code analysis. It also supports configuration profiles, allowing users to define and quickly switch between different project setups as needed.
In most cases, not all CLI components are required within a single project. For example, in a CI (Continuous Integration) environment, only the `runner` package is typically necessary to execute tests, while the `language-server` package is generally not needed. The `analyze` package is mainly useful in a development environment to detect syntax errors and validate code, but it may be unnecessary in production or deployment environments. Similarly, the `debugger` package is essential for local development when troubleshooting test cases but isnβt usually required in production or CI pipelines.
To accommodate these varied needs, `robotcode` is organized into separate packages that each focus on specific functions. The core package, `robotcode`, provides foundational support for working with `robot.toml` configuration files and profile management. Hereβs a more detailed breakdown of each package and its capabilities:
- **`robotcode` Core Package**:
Always installed, the core package provides the top-level `robotcode` command together with the global options (config files, profiles, output format, paging, β¦) shared by every other command. It also includes commands to inspect the resolved configuration and the profiles defined for a project.
- **Commands**:
- `config`: Shows the merged `robot.toml` configuration and which files contributed to it, making it easy to understand how settings are resolved.
- `profiles`: Lists the profiles defined for the project and shows their resulting configuration, helping verify which profile applies in a given context.
- **`runner` Package**:
This package is essential for users who need to run and manage tests within Robot Framework projects. It includes commands for executing tests, generating documentation, and discovering test elements. This package is especially important in CI/CD pipelines, where automation of test execution is a primary focus.
- **Commands**:
- `robot`, `rebot`, `libdoc`, `testdoc`: Enhanced versions of the standard Robot Framework tools, with support for `robot.toml` configuration files and profiles, allowing customized setups for different environments or testing requirements.
- `discover`: Searches the project for tests, suites, tags, tasks, and other elements, providing a quick overview of available test cases and project structure.
- `results`: Inspects a finished run's `output.xml`/`output.json` β summaries, failures, the per-test execution log, and diffs between two runs β without re-executing the tests.
- **`analyze` Package**:
This package provides tools for detailed inspection and validation of Robot Framework code, helping users identify errors and improve code quality. The `analyze` package is typically more useful in development environments where code quality checks and error detection are needed before moving tests to a CI or production environment.
- **Commands**:
- `analyze`: Analyzes Robot Framework scripts for syntax errors, undefined keywords, and other potential issues, allowing early detection of problems and ensuring adherence to best practices.
- **`debugger` Package**:
The debugger package enables powerful debugging capabilities for Robot Framework tests by providing a Debug Adapter Protocol (DAP)-compatible debugger. This package is most beneficial in development or local testing environments where developers need to diagnose and troubleshoot test issues. A DAP client, such as Visual Studio Code, can be connected to initiate and control debug sessions, enabling features like setting breakpoints, stepping through code, and inspecting variables.
- **Commands**:
- `debug`: Starts a DAP-compatible debug session for Robot Framework tests. This tool requires a DAP client to connect to the debug session, such as Visual Studio Code, which can then interface with the debugger and provide interactive debugging tools to analyze code behavior and troubleshoot issues effectively.
- **`repl` Package**:
The REPL (Read-Eval-Print Loop) package provides an interactive, real-time environment for executing Robot Framework commands. Itβs ideal for experimenting with keywords, testing ideas, and performing quick debugging without needing to create full test files. It also ships a command-line debugger for Robot Framework runs. This package is mainly used in local development or testing environments where users can quickly prototype or troubleshoot commands.
- **Commands**:
- `repl`: Launches an interactive Robot Framework shell where users can execute commands line-by-line, ideal for quick testing and experimentation.
- `robot-debug`: Runs a real `.robot` suite with the command-line debugger attached, pausing at breakpoints to step through execution, inspect the call stack and variables, and run keywords in the paused context.
- **`language-server` Package**:
This package provides language server capabilities, supporting IDE integration for Robot Framework with real-time code insights. Compatible with editors that support the Language Server Protocol (LSP), such as Visual Studio Code, it enables enhanced productivity and convenience. It is most useful in local development environments where interactive IDE support aids in code writing and refactoring but is generally not needed in CI or production environments.
- **Commands**:
- `language-server`: Starts the RobotCode Language Server, which provides features like syntax highlighting, auto-completion, and code analysis, designed to improve the Robot Framework development experience within IDEs.
## Installation
To install the core `robotcode` CLI tool, use `pip`:
```bash
pip install robotcode
```
This command installs only the main package. For specific functionality, additional packages can be installed as needed:
```bash
pip install robotcode[runner]
pip install robotcode[analyze]
pip install robotcode[debugger]
pip install robotcode[repl]
pip install robotcode[languageserver]
```
To install all packages, including optional dependencies, use:
```bash
pip install robotcode[all]
```
This includes additional tools, such as [`robocop`](https://robocop.readthedocs.io) for linting and formatting, which further enhance the development experience with Robot Framework.
## Commands
The following sections outline all available commands, their usage, and the corresponding options.
Options with and asterisk (*) can be specified multiple times.
### robotcode
A CLI tool for Robot Framework.
**Usage:**
```text
robotcode [OPTIONS] COMMAND [ARGS]...
```
**Options:**
- `-c, --config PATH *`
Config file to use. If not specified, the default config file is used. [env var: ROBOTCODE_CONFIG_FILES]
- `-p, --profile TEXT *`
The Execution Profile to use. If not specified, the default profile is used. [env var: ROBOTCODE_PROFILES]
- `-r, --root DIRECTORY`
Specifies the root path to be used for the project. It will be automatically detected if not provided. [env var: ROBOTCODE_ROOT]
- `--no-vcs`
Ignore version control system directories (e.g., .git, .hg) when detecting the project root. [env var: ROBOTCODE_NO_VCS]
- `-f, --format [toml|json|json_indent|text]`
Set the output format.
- `-d, --dry`
Dry run, do not execute any commands. [env var: ROBOTCODE_DRY]
- `--color / --no-color`
Force or disable colored output. Default (no flag): auto-detect β colors only when stdout is a TTY, disabled if `NO_COLOR` is set, forced if `FORCE_COLOR` is set. [env var: ROBOTCODE_COLOR]
- `--pager / --no-pager`
Force or disable the pager. Default (no flag): auto-page when the rendered output exceeds the terminal height. [env var: ROBOTCODE_PAGER]
- `-v, --verbose`
Enables verbose mode. [env var: ROBOTCODE_VERBOSE]
- `--log`
Enables logging. [env var: ROBOTCODE_LOG]
- `--log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL]`
Sets the log level. [env var: ROBOTCODE_LOG_LEVEL; default: CRITICAL]
- `--log-format TEXT`
Sets the log format. See python logging documentation for more information. [env var: ROBOTCODE_LOG_FORMAT; default: %(levelname)s:%(name)s:%(message)s]
- `--log-style [%|{|$]`
Sets the log style. See python logging documentation for more information. [env var: ROBOTCODE_LOG_STYLE; default: %]
- `--log-filename FILE`
Write log output to a file instead to console. [env var: ROBOTCODE_LOG_FILENAME]
- `--log-calls`
Enables logging of method/function calls. [env var: ROBOTCODE_LOG_CALLS]
- `--log-config FILE`
Path to a logging configuration file. This must be a valid Python logging configuration file in JSON format. If this option is set, the other logging options are ignored. [env var: ROBOTCODE_LOG_CONFIG]
- `--version`
Show the version and exit.
- `--help`
Show this message and exit.
**Commands:**
- [`analyze`](#analyze)
The analyze command provides various subcommands for analyzing Robot Framework code.
- [`config`](#config)
Shows information about the configuration.
- [`debug`](#debug)
Starts a Robot Framework debug session and waits for incomming connections.
- [`debug-launch`](#debug-launch)
Launches a robotcode debug session.
- [`discover`](#discover)
Commands to discover informations about the current project.
- [`language-server`](#language-server)
Run Robot Framework Language Server.
- [`libdoc`](#libdoc)
Runs `libdoc` with the selected configuration, profiles, options and arguments.
- [`profiles`](#profiles)
Shows information on defined profiles.
- [`rebot`](#rebot)
Runs `rebot` with the selected configuration, profiles, options and arguments.
- [`repl`](#repl)
Run Robot Framework interactively (alias `shell`).
- [`repl-server`](#repl-server)
Start a REPL server, client can connect to the server and run the REPL scripts.
- [`results`](#results)
Inspect a finished run's `output.xml` / `output.json` β counts, failures, and per-test execution tree, without re-running.
- [`robot`](#robot)
Runs `robot` with the selected configuration, profiles, options and arguments.
- [`robot-debug`](#robot-debug)
Run a real Robot Framework suite with the debugger attached (alias `run-debug`).
- [`testdoc`](#testdoc)
Runs `testdoc` with the selected configuration, profiles, options and arguments.
**Aliases:**
- [`shell`](#repl)
Run Robot Framework interactively (alias `shell`).
- [`run`](#robot)
Runs `robot` with the selected configuration, profiles, options and arguments.
- [`run-debug`](#robot-debug)
Run a real Robot Framework suite with the debugger attached (alias `run-debug`).
#### analyze
The analyze command provides various subcommands for analyzing Robot
Framework code. These subcommands support specialized tasks, such as code
analysis, style checking or dependency graphs.
**Usage:**
```text
robotcode analyze [OPTIONS] COMMAND [ARGS]...
```
**Options:**
- `--version`
Show the version and exit.
- `--help`
Show this message and exit.
**Commands:**
- [`cache`](#cache)
Manage the RobotCode analysis cache.
- [`code`](#code)
Performs static code analysis to identify potential issues in the specified *PATHS*.
##### cache
Manage the RobotCode analysis cache.
Provides subcommands to inspect, list, and clear cached data (library docs,
variables, resources, namespaces).
**Usage:**
```text
robotcode analyze cache [OPTIONS] COMMAND [ARGS]...
```
**Options:**
- `--help`
Show this message and exit.
**Commands:**
- [`clear`](#clear)
Clear the analysis cache.
- [`info`](#info)
Show cache statistics.
- [`list`](#list)
List cached entries.
- [`path`](#path)
Print the cache directory path.
- [`prune`](#prune)
Remove the entire cache directory.
###### clear
Clear the analysis cache.
Removes cached entries from the database. By default clears all sections.
Use --section to clear specific sections only.
**Usage:**
```text
robotcode analyze cache clear [OPTIONS] [PATHS]...
```
**Options:**
- `-s, --section SECTION *`
Clear only specific sections (library, variables, resource, namespace). Can be specified multiple times.
- `--help`
Show this message and exit.
###### info
Show cache statistics.
Displays the cache directory, database size, app version, and per-section
entry counts with timestamps.
**Usage:**
```text
robotcode analyze cache info [OPTIONS] [PATHS]...
```
**Options:**
- `--help`
Show this message and exit.
###### list
List cached entries.
Shows all entries in the cache with their timestamps and sizes. Use
--section to filter by specific cache sections. Use --pattern to filter
entries by glob pattern.
**Usage:**
```text
robotcode analyze cache list [OPTIONS] [PATHS]...
```
**Options:**
- `-s, --section SECTION *`
Filter by section (library, variables, resource, namespace). Can be specified multiple times.
- `-p, --pattern PATTERN *`
Filter entries by glob pattern (e.g. 'robot.*', '*BuiltIn*'). Can be specified multiple times.
- `--help`
Show this message and exit.
###### path
Print the cache directory path.
Outputs the resolved cache directory for the current project and
Python/Robot Framework version combination.
**Usage:**
```text
robotcode analyze cache path [OPTIONS] [PATHS]...
```
**Options:**
- `--help`
Show this message and exit.
###### prune
Remove the entire cache directory.
Deletes the .robotcode_cache directory and all its contents, including
caches for all Python and Robot Framework versions.
**Usage:**
```text
robotcode analyze cache prune [OPTIONS] [PATHS]...
```
**Options:**
- `--force`
Force prune even if cache is in use by another process.
- `--help`
Show this message and exit.
##### code
Performs static code analysis to identify potential issues in the specified
*PATHS*. The analysis detects syntax errors, missing keywords or variables,
missing arguments, and other problems.
- **PATHS**: Can be individual files or directories. If no *PATHS* are
provided, the current directory is analyzed by default.
The return code is a bitwise combination of the following values:
- `0`: **SUCCESS** - No issues detected. - `1`: **ERRORS** - Critical issues
found. - `2`: **WARNINGS** - Non-critical issues detected. - `4`:
**INFORMATIONS** - General information messages. - `8`: **HINTS** -
Suggestions or improvements.
*Examples*:
```
robotcode analyze code
robotcode analyze code --filter **/*.robot
robotcode analyze code tests/acceptance/first.robot
robotcode analyze code -mi DuplicateKeyword tests/acceptance/first.robot
robotcode --format json analyze code
```
**Usage:**
```text
robotcode analyze code [OPTIONS] [PATHS]...
```
**Options:**
- `--version`
Show the version and exit.
- `-f, --filter PATTERN *`
Glob pattern to filter files to analyze. Can be specified multiple times.
- `-v, --variable name:value *`
Set variables in the test data. see `robot --variable` option.
- `-V, --variablefile PATH *`
Python or YAML file file to read variables from. see `robot --variablefile` option.
- `-P, --pythonpath PATH *`
Additional locations where to search test libraries and other extensions when they are imported. see `robot --pythonpath` option.
- `-mi, --modifiers-ignore CODE *`
Specifies the diagnostics codes to ignore.
- `-me, --modifiers-error CODE *`
Specifies the diagnostics codes to treat as errors.
- `-mw, --modifiers-warning CODE *`
Specifies the diagnostics codes to treat as warning.
- `-mI, --modifiers-information CODE *`
Specifies the diagnostics codes to treat as information.
- `-mh, --modifiers-hint CODE *`
Specifies the diagnostics codes to treat as hint.
- `-xm, --exit-code-mask [error|warn|info|hint|all] *`
Specifies which diagnostic severities should not affect the exit code. For example, with 'warn' in the mask, warnings won't cause a non-zero exit code.
- `-xe, --extend-exit-code-mask [error|warn|info|hint|all] *`
Extend the exit code mask with the specified values. This appends to the default mask, defined in the config file.
- `--severity [error|warn|warning|info|information|hint] *`
Only report diagnostics of these severities. Repeatable and comma-separated. Filtered-out severities are ignored entirely β they don't appear in the output, the summary or the exit code. When omitted, all severities are reported.
- `--code CODE *`
Only report diagnostics with these codes (e.g. `KeywordNotFound`). Repeatable and comma-separated; matching is case-insensitive. Unlike the modifiers, this filters without changing severity. Combined with --severity, both must match. When omitted, all codes are reported.
- `--load-library-timeout SECONDS`
Timeout (in seconds) for loading libraries and variable files during analysis. Must be > 0. Overrides config file and environment variable when set. [env var: ROBOTCODE_LOAD_LIBRARY_TIMEOUT]
- `--collect-unused / --no-collect-unused`
Enable or disable collection of unused keyword and unused variable diagnostics. Overrides the config file setting when specified.
- `--cache-namespaces / --no-cache-namespaces`
Enable or disable caching of fully analyzed namespace data to disk. Can speed up startup for large projects by skipping re-analysis of unchanged files.
- `--show-tracebacks / --no-show-tracebacks`
Include the full diagnostic message in the text output, including Python tracebacks and PYTHONPATH listings that Robot Framework appends to import errors. Off by default to keep output concise. Has no effect on JSON output, which always carries the full message.
- `--full-paths / --no-full-paths`
Show full paths instead of paths relative to the project root. Applies to both text and JSON output. [default: no-full-paths]
- `--output-format [concise|json|json-indent|sarif|github|gitlab]`
Output format for the analysis result. Overrides the global `--format` for this command. `concise` (default) is the human-readable text output; `sarif` emits a SARIF 2.1.0 log; `github` emits GitHub Actions workflow annotations; `gitlab` emits a GitLab Code Quality report.
- `--output-file FILE`
Write the report to FILE instead of stdout. Useful with `--output-format sarif`/`gitlab` to produce an artifact for CI upload.
- `--help`
Show this message and exit.
#### config
Shows information about the configuration.
**Usage:**
```text
robotcode config [OPTIONS] COMMAND [ARGS]...
```
**Options:**
- `--help`
Show this message and exit.
**Commands:**
- [`files`](#files)
Search for configuration files and list them.
- [`info`](#info)
Shows informations about possible configuration settings.
- [`root`](#root)
Searches for the root folder of the project and prints them.
- [`show`](#show)
Shows the current configuration.
##### files
Search for configuration files and list them.
Takes a list of PATHS or if no PATHS are given, takes the current working
directory, to search for configuration files and prints them.
Examples:
```
robotcode config files
robotcode config files tests/acceptance/first.robot
```
**Usage:**
```text
robotcode config files [OPTIONS] [PATHS]... [USER]
```
**Options:**
- `--help`
Show this message and exit.
##### info
Shows informations about possible configuration settings.
**Usage:**
```text
robotcode config info [OPTIONS] COMMAND [ARGS]...
```
**Options:**
- `--help`
Show this message and exit.
**Commands:**
- [`desc`](#desc)
Shows the description of the specified configuration settings.
- [`list`](#list)
Lists all possible configuration settings.
###### desc
Shows the description of the specified configuration settings.
If no NAME is given shows the description of all possible configuration
settings. Wildcards are supported.
Examples:
```
robotcode config info desc
robotcode config info desc python-path
robotcode config info desc rebot.*
robotcode config info desc *tag*
```
**Usage:**
```text
robotcode config info desc [OPTIONS] [NAME]...
```
**Options:**
- `--help`
Show this message and exit.
###### list
Lists all possible configuration settings.
If NAME is given searches for given name. Wildcards are supported.
Examples:
```
robotcode config info list
robotcode config info list rebot.*
robotcode config info list *tag*
```
**Usage:**
```text
robotcode config info list [OPTIONS] [NAME]...
```
**Options:**
- `--help`
Show this message and exit.
##### root
Searches for the root folder of the project and prints them.
Takes a list of PATHS or if no PATHS are given, takes the current working
directory, to search for the root of the project and prints this.
Examples:
```
robotcode config root
robotcode config root tests/acceptance/first.robot
```
**Usage:**
```text
robotcode config root [OPTIONS] [PATHS]...
```
**Options:**
- `--help`
Show this message and exit.
##### show
Shows the current configuration.
Takes a list of PATHS or if no PATHS are given, takes the current working
directory, to search for configuration files and prints the current
configuration.
Examples:
```
robotcode config show
robotcode config show tests/acceptance/first.robot
robotcode --format json config show
```
**Usage:**
```text
robotcode config show [OPTIONS] [PATHS]...
```
**Options:**
- `-s, --single`
Shows single files, not the combined config.
- `--help`
Show this message and exit.
#### debug
Starts a Robot Framework debug session and waits for incomming connections.
**Usage:**
```text
robotcode debug [OPTIONS] [ROBOT_OPTIONS_AND_ARGS]...
```
**Options:**
- `--debug / --no-debug`
Enable/disable debug mode [default: debug]
- `--stop-on-entry / --no-stop-on-entry`
Breaks into debugger when a robot framework run starts. [default: no-stop-on-entry]
- `--wait-for-client / --no-wait-for-client`
Waits until a debug client is connected. [env var: ROBOTCODE_WAIT_FOR_CLIENT; default: wait-for-client]
- `--wait-for-client-timeout FLOAT`
Timeout in seconds for waiting for a connection with a debug client. [env var: ROBOTCODE_WAIT_FOR_CLIENT_TIMEOUT; default: 15]
- `--configuration-done-timeout FLOAT`
Timeout to wait for a configuration from client. [env var: ROBOTCODE_CONFIGURATION_DONE_TIMEOUT; default: 15]
- `--debugpy / --no-debugpy`
Enable/disable python debugging. [env var: ROBOTCODE_DEBUGPY; default: no-debugpy]
- `--debugpy-wait-for-client / --no-debugpy-wait-for-client`
Waits for a debugpy client to connect. [env var: ROBOTCODE_DEBUGPY_WAIT_FOR_CLIENT]
- `--debugpy-port INTEGER`
The port for the debugpy session. [default: 5678]
- `--output-messages / --no-output-messages`
Send output messages from robot framework to client. [default: no-output-messages]
- `--output-log / --no-output-log`
Send log messages from robotframework to client. [default: output-log]
- `--output-timestamps / --no-output-timestamps`
Include timestamps in log and output messages. [default: no-output-timestamps]
- `--group-output / --no-group-output`
Fold/group messages or log messages. [default: no-group-output]
- `--tcp [:]{{ post.frontmatter.title }}
{{ new Date(post.frontmatter.date).toLocaleDateString('en-US', { year: 'numeric', month: 'long', day: 'numeric' }) }}