Exceptions and escalations

A failure is never silent. The run ends in a terminal status, the step that failed carries the error, and the events say what happened before it. A problem found after the route answered, such as an agent that no longer exists in the organization, ends the run as failed with the reason on its record.

{ "seq": 7, "type": "log", "step_key": "tool:query_positions#1", "level": "error", "message": "Snowflake: 503 from warehouse WH_PROD", "data": {} }
{ "seq": 8, "type": "step.failed", "step_key": "tool:query_positions#1", "message": "connection refused after 3 attempts", "data": {} }
{ "seq": 9, "type": "failed", "step_key": null, "message": "tool:query_positions#1 failed", "data": {} }

Two layers can fail: your call, and the run it started. Handle each with the signal NSTACK gives you.

For a webhook you control, make the start idempotent on your side: set the run up with test=true, store the instance_id next to your own record, then start it by id. A crash between the two calls leaves an id you can start again; a duplicate start answers 409.

async function startOnce(payload: object, key: string) {
  let instanceId = await db.get(`nstack:instance:${key}`);
  if (!instanceId) {
    const setup = await post(`${wf}/workflows/${workflowId}?env=${envId}&test=true`, payload);
    instanceId = setup.instance_id;
    await db.set(`nstack:instance:${key}`, instanceId);
  }
  const res = await postRaw(`${wf}/workflows/${workflowId}?env=${envId}&instance_id=${instanceId}`, payload);
  if (res.status === 409) return instanceId;             // already started: fine
  if (res.status === 503) throw new Retry(res);          // back off and call startOnce again
  if (!res.ok) throw new Error(`refused: ${res.status}`);
  return instanceId;
}

def start_once(payload, key):
    instance_id = db.get(f"nstack:instance:{key}")
    if not instance_id:
        setup = post(f"{wf}/workflows/{workflow_id}", params={"env": env_id, "test": "true"}, json=payload)
        instance_id = setup["instance_id"]
        db.set(f"nstack:instance:{key}", instance_id)
    res = post_raw(f"{wf}/workflows/{workflow_id}", params={"env": env_id, "instance_id": instance_id}, json=payload)
    if res.status_code == 409:
        return instance_id            # already started: fine
    if res.status_code == 503:
        raise Retry(res)              # back off and call start_once again
    res.raise_for_status()
    return instance_id

An agent run answers its run_id before the agent starts. Store the id before you do anything else, and check the record before you call again; the same thread_id keeps a repeated task in one conversation.

A step that changes something in Safe mode does not run until a person approves it. The run moves to waiting with reason: "approval", the approver decides in Studio, and the server stamps who decided and when. The run then continues or ends. Your side of this is routing: find the runs that wait, get them in front of the right person, and stop waiting when they decide.

1. Find them. GET …/runs?status=waiting&env_id=…, on a timer. Read title, subject_id, actor, and queued_at.
2. Route them. Post the run to your pager, chat, or case system with the run id. Studio's Runs page finds a run by id, and the approver acts there.
3. Age them. A run that waits past your own limit is an escalation. Re-post it to the next person; the record keeps waiting until someone decides.
4. Close the loop. When the status leaves waiting, read the outcome and update your case.

const WAIT_LIMIT_MS = 30 * 60 * 1000;

async function sweep() {
  const [waiting, failed] = await Promise.all([
    list({ status: "waiting", env_id: prodEnvId }),
    list({ status: "failed", from: new Date(Date.now() - 60_000).toISOString() }),
  ]);
  for (const r of waiting.items) {
    const age = Date.now() - Date.parse(r.queued_at);
    const to = age > WAIT_LIMIT_MS ? ops.escalationChannel : ops.approverFor(r.subject_id);
    await ops.post(to, `${r.title} is waiting for approval (${Math.round(age / 60000)} min). Run ${r.run_id}.`);
  }
  for (const r of failed.items) await ops.post(ops.oncall, `${r.title} failed. Run ${r.run_id}.`);
}

• In design Approvals through the API. approval.requested and approval.decided events, a decision route for your own approval UI, and SLA timers that escalate on their own.
• In design Idempotency-Key on every route that starts a run, so a retry never starts a second one.
• In design Signals, so your system resumes a waiting run with a callback or a decision.