API test scenarios: format, execution and reports

Run reusable API scenarios through HTTP, SDK, CLI, SSH or MCP. Configure requests, variables, checks and captures with separate secrets and bounded execution.

Updated 27 October 2026

A portable version 1 scenario

Send a serialized scenario in input to POST /api/tools/mock-apis-api-test-scenarios/run. The browser workbench exports the same format. Requests run in array order against baseUrl, which must be a public HTTP/HTTPS origin on port 80, 443, 8080 or 8443. Each request path starts with one slash; cross-origin destinations are rejected.

The configuration has version, name, baseUrl, variables, secretNames, stopOnFailure and requests. There are one to six requests, named for the behavior they check. Each request needs method, path and expect.status. Headers and body are optional. GET and HEAD cannot carry a body.

api-scenario.json

{
  "version": 1,
  "name": "Read an item",
  "baseUrl": "https://api.example.com",
  "variables": {},
  "secretNames": [
    "accessToken"
  ],
  "stopOnFailure": true,
  "requests": [
    {
      "name": "List items",
      "method": "GET",
      "path": "/items",
      "headers": {
        "x-access-token": "{{accessToken}}"
      },
      "body": "",
      "expect": {
        "status": 200
      },
      "extract": {
        "name": "itemId",
        "path": "items.0.id"
      }
    },
    {
      "name": "Read first item",
      "method": "GET",
      "path": "/items/{{itemId}}",
      "headers": {
        "x-access-token": "{{accessToken}}"
      },
      "body": "",
      "expect": {
        "status": 200,
        "json": {
          "path": "active",
          "equals": true
        }
      }
    }
  ]
}

Variables, checks and captures

Ordinary variables are string values referenced with {{name}}. Define them before use. Up to twenty ordinary variables and ten declared secret names are allowed. Variables captured by earlier requests can be referenced by later requests. A capture is {name, path}, where path is a dotted JSON field path with at most eight segments. Captures accept non-secret strings, numbers or booleans up to 1,024 characters.

Each expect object requires status (100–599), and may include contains (up to 500 characters) and json: {path, equals}. JSON equality is type-sensitive and supports a string, finite number, boolean or null. A missing property fails. Templates in JSON bodies belong inside quoted strings; substitutions are escaped as JSON string content. No scripts or expressions are executed.

Supply secrets outside the configuration

Declare secret names in secretNames. Supply their values as a serialized JSON object in options.secret. Every declared secret needs a value, and unknown secret keys are rejected. Secret placeholders can only appear in request headers. Authorization and other credential headers require a declared-secret placeholder rather than a literal value.

Only Accept, Content-Type, Authorization and non-routing X-* request headers are accepted. Cookies, Host, proxy and hop-by-hop headers are not accepted. All steps stay on the same origin, and redirects are not followed. Known secret values and sensitive response fields are redacted in reports. The server retains no scenario or execution history.

SDK execution

const result = await socket.run(
  "mock-apis-api-test-scenarios",
  JSON.stringify(scenario),
  { secret: JSON.stringify({ accessToken }) }
);
const report = JSON.parse(result.output);
if (!report.passed) console.error("Scenario checks failed");

HTTP, CLI, SSH and MCP

All interfaces use the same validation and network runner. The CLI reads the scenario JSON from stdin. For SSH, use the existing socket-help command gateway and the same run command. MCP uses run_tool with toolId mock-apis-api-test-scenarios, the serialized scenario in input, and optional string options.secret. describe_tool returns the input example and current limits.

A successful HTTP execution returns output as a JSON string containing the report, plus normal hosted usage. HTTP 200 means a report was produced; inspect report.passed to decide whether its checks succeeded. A completed report uses one hosted run even when an assertion or a request fails. Configuration errors return 400 and do not consume a run.

HTTP execution without secret inputs

jq -Rs '{input: .}' api-scenario.json | \
  curl https://socket.help/api/tools/mock-apis-api-test-scenarios/run \
    -H 'Content-Type: application/json' --data-binary @-

# CLI
socket-help run mock-apis-api-test-scenarios < api-scenario.json

Report structure and execution bounds

A report contains name, origin, passed, total, completed, durationMs and results. Each result has its name, method, path, outcome and checks; completed requests also include available status, durationMs, response bytes, truncated, sanitized headers and body. Captured variable names appear in captured; their values are not separately returned.

Outcomes are passed, failed or skipped. stopOnFailure defaults to true. Failed expectations, missing captures, response truncation and request errors fail that step. When stopping, later steps are marked skipped. With stopOnFailure false, independent requests continue. A dependent step with a missing variable fails before it is sent. Cancelling or an overall timeout can stop the request before a complete report is returned; requests already sent cannot be undone.

Scenario execution limits
Limit	Value
Requests	1–6, sequential, one iteration
Scenario JSON	32 KiB
Per-request deadline	3 seconds
Overall deadline	20 seconds
Body template / expanded body	8 KiB / 16 KiB
Response body	64 KiB per request
Headers	12 per request
Secret options JSON	4,096 characters