Interfaces

Observability CLI reference

Inspect live health, traces, events, resources, and LayerStack state.

On this page+

Before you run
Fleet health
snapshot
Traces and events
trace
events
Resources and layers
cgroup
layerstack
Evidence and error contract

Before you run

sandbox-observability-cli exposes five read-only views. Unlike the runtime client, its sandbox selector belongs to each operation. snapshot may omit it for a fleet view; every other operation requires it.

Shell

sandbox-observability-cli help
sandbox-observability-cli help snapshot
sandbox-observability-cli snapshot

Start with snapshot, then query the evidence relevant to the failure. The architecture guide explains the runtime, LayerStack, and workspace boundaries behind this evidence.

Find an operation

Showing 5 of 5 commands.

Fleet health

Inspect normalized live availability and active runtime state.

`snapshot`

Inspect live sandbox health

Read-only

Show live state for one sandbox, or aggregate every ready manager-known sandbox when sandbox_id is omitted.

When to use it

Omit sandbox_id for fleet aggregation, or supply one id to select a manager record.
Each node normalizes availability and includes an errors array for partial failures.
Use this first for live health, then follow with telemetry or resource operations.

Syntax

Usage

sandbox-observability-cli snapshot [--sandbox-id ID]

Arguments

Argument	Type	Required	Description
`--sandbox-id ID`	`string`	No	Optional target sandbox id. When omitted, the manager queries all ready sandboxes.

Examples

Shell

sandbox-observability-cli snapshot
sandbox-observability-cli snapshot --sandbox-id eos-abc

Representative result

JSON output

{
  "sandboxes": [
    {
      "sandbox_id": "sbox-example",
      "lifecycle_state": "ready",
      "availability": "available",
      "sampled_at_unix_ms": 1784016000000,
      "errors": [],
      "daemon": {
        "daemon_pid": 321,
        "runtime_dir": "/run/ephemeral-os"
      },
      "resources": {
        "latest": null,
        "history": []
      },
      "workspaces": [],
      "stack": {
        "layer_count": 2,
        "layers_bytes": 4096,
        "active_leases": 0
      }
    }
  ]
}

Result contract: A sandbox-scoped call returns the node shape rather than the aggregate sandboxes wrapper.

Related operationstrace cgroup layerstack

Traces and events

Fold telemetry into a span waterfall or filtered fact stream.

`trace`

Render a span waterfall

Read-only

Fold the telemetry log into a nested span waterfall for one exact trace or the most recent root trace.

When to use it

Use last for the most recently started root trace, or provide an exact trace/request id.
spans is a forest containing completed span data, offsets, children, and events.
An unknown trace is not an operation error; it returns an empty spans array.

Syntax

Usage

sandbox-observability-cli trace --sandbox-id ID [--trace-id TRACE|last]

Arguments

Argument	Type	Required	Description
`--sandbox-id ID`	`string`	Yes	Target sandbox id (selects the daemon to query).
`--trace-id TRACE\|last`	`string`	No; default last	Trace id to render, or 'last' for the most recent root trace.

Examples

Shell

sandbox-observability-cli trace --sandbox-id eos-abc --trace-id req-7f3
sandbox-observability-cli trace --sandbox-id eos-abc --trace-id last

Representative result

JSON output

{
  "view": "trace",
  "trace": "request-uuid",
  "spans": [
    {
      "span": {
        "ts": 1784016000250,
        "trace": "request-uuid",
        "span": "d-0",
        "name": "daemon.dispatch",
        "dur_ms": 250,
        "status": "completed",
        "attrs": {
          "op": "exec_command"
        }
      },
      "offset_ms": 0,
      "children": [],
      "events": []
    }
  ]
}

Result contract: Span status is completed, error, cancelled, or timed_out; attrs is operation-specific.

Related operationsevents snapshot

`events`

Filter cross-trace domain events

Read-only

Fold the telemetry log into a newest-first event stream with optional exact-name, timestamp, and count filters.

When to use it

name is an exact dotted label such as lease.acquired.
last_n retains only the newest matches; omit filters for every retained event.
Each event carries trace and parent identifiers that can be followed into trace.

Syntax

Usage

sandbox-observability-cli events --sandbox-id ID [--name NAME] [--since-ms MS] [--last-n N]

Arguments

Argument	Type	Required	Description
`--sandbox-id ID`	`string`	Yes	Target sandbox id (selects the daemon to query).
`--name NAME`	`string`	No	Filter to events with this exact name (e.g. lease.acquired).
`--since-ms MS`	`integer`	No	Only events at or after this unix-ms timestamp.
`--last-n N`	`integer`	No	Keep only the N newest matched events.

Examples

Shell

sandbox-observability-cli events --sandbox-id eos-abc
sandbox-observability-cli events --sandbox-id eos-abc --name lease.acquired --last-n 20

Representative result

JSON output

{
  "view": "events",
  "events": [
    {
      "ts": 1784016000120,
      "trace": "request-uuid",
      "parent": "d-1",
      "name": "lease.acquired",
      "attrs": {
        "layer_id": "layer-7"
      }
    }
  ]
}

Result contract: attrs is an open domain-fact object and varies by event name.

Related operationstrace

Resources and layers

Inspect resource samples and the active layer manifest.

`cgroup`

Read a resource time series

Read-only

Read sandbox CPU, memory, and I/O counters from Docker Engine, or workspace disk samples from daemon telemetry.

When to use it

Use sandbox scope for Docker CPU, memory, and block-I/O counters.
Use a workspace id as scope for daemon-recorded workspace disk samples.
Deltas appear only when a prior counter sample exists in the selected window.

Syntax

Usage

sandbox-observability-cli cgroup --sandbox-id ID [--scope SCOPE] [--window-ms MS]

Arguments

Argument	Type	Required	Description
`--sandbox-id ID`	`string`	Yes	Target sandbox id (selects the daemon to query).
`--scope SCOPE`	`string`	No; default sandbox	Resource scope: 'sandbox' or a workspace id.
`--window-ms MS`	`integer`	No; default 60000	Lookback window in milliseconds (max 600000).

Examples

Shell

sandbox-observability-cli cgroup --sandbox-id eos-abc
sandbox-observability-cli cgroup --sandbox-id eos-abc --scope ws-1 --window-ms 60000

Representative result

JSON output

{
  "view": "cgroup",
  "scope": "sandbox",
  "series": [
    {
      "ts": 1784016000000,
      "sample_delta_ms": null,
      "metrics": {
        "metrics_source": "docker_engine",
        "cpu_usec": 912345,
        "mem_cur": 67108864,
        "mem_max": 1073741824,
        "io_rbytes": 4096,
        "io_wbytes": 8192
      },
      "deltas": {}
    }
  ]
}

Result contract: Sandbox metrics identify metrics_source as docker_engine.

Related operationssnapshot layerstack

`layerstack`

Inspect the active layer manifest

Read-only

Show live layer sizes, leases, base bookings, and optional trend data from the active manifest.

When to use it

Omit workspace_id for the full active manifest and optional trend.
Include workspace_id for that session lower layers and private upperdir bytes.
Layers are returned in manifest order; booked_by and leased_by_workspaces describe live use.

Syntax

Usage

sandbox-observability-cli layerstack --sandbox-id ID [--workspace-id WS] [--window-ms MS]

Arguments

Argument	Type	Required	Description
`--sandbox-id ID`	`string`	Yes	Target sandbox id (selects the daemon to query).
`--workspace-id WS`	`string`	No	Show one workspace's lower layers and private upperdir.
`--window-ms MS`	`integer`	No; default 60000	Lookback window in milliseconds for the stack trend (max 600000).

Examples

Shell

sandbox-observability-cli layerstack --sandbox-id eos-abc
sandbox-observability-cli layerstack --sandbox-id eos-abc --workspace-id ws-7

Representative result

JSON output

{
  "view": "layerstack",
  "manifest_version": 7,
  "root_hash": "sha256:…",
  "active_lease_count": 1,
  "total_bytes": 16384,
  "total_allocated_bytes": 20480,
  "storage_logical_bytes": 24576,
  "storage_allocated_bytes": 28672,
  "staging_entry_count": 0,
  "layers": [
    {
      "layer_id": "layer-7",
      "bytes": 8192,
      "allocated_bytes": 12288,
      "leased_by_workspaces": 1,
      "booked_by": []
    }
  ],
  "trend": []
}

Result contract: With workspace_id, the result changes to { view, workspace, mounts, upper_bytes }.

Related operationssnapshot cgroup

Evidence and error contract

Results are point-in-time evidence, not an event subscription. Timestamps, offsets, and counters can advance between calls. Fleet snapshots preserve partial failures in each node’s errors array so one unavailable sandbox does not erase healthy results.

Exit 0 writes JSON to stdout. Exit 1 writes gateway or operation errors to stderr, and exit 2 reports local syntax, validation, or configuration failure.