---
name: ndo-repro
description: Build an NDO microservice locally with Docker, push it to artifactory, deploy it to a dev env, then reproduce or validate the fix by driving the Business-Operation-Manager (BOM) API and reading live pod logs. Use when debugging or verifying a UNM-* ticket without waiting for CI, when the UI flow is hard to reproduce, when driving the replacement/Map-To/target-insert flow without a browser, or when the user says "repro via API", "drive BOM", "ship to <env>", "deploy my build to dev-2", "validate the fix on the cluster", "run ndo-repro in <env>". Covers env discovery across the saas-rnd-oss and ndo-shared clusters.
---

# NDO build → deploy → repro loop

Full loop on one env, no CI wait: build the service locally, push to artifactory, repoint the k8s deployment, then drive BOM's API and read pod logs to prove the ticket's acceptance criteria.

Two scripts, both env-aware via `-e <alias>`:
- `~/.claude/skills/ndo-repro/ndo-ship.sh` — doctor / test / build / push / deploy / status / rollback
- `~/.claude/skills/ndo-repro/ndo-api.sh` — env registry / auth / BOM API / logs

Run `--help` on either for the full command list.

## Envs

Aliases come from a discovered registry (`envs.tsv`, refreshed with `ndo-api.sh env discover` — it scans every kube context for a namespace running `consolidated-inventory-manager-v1` and reads the `public-gateway` ingress host).

```
ndo-api.sh env ls                 # alias → context / namespace / gateway
ndo-api.sh -e oss-01/dev-2 env show
```

Alias shape is `<cluster>/<env>` (`oss-01/dev-2`, `oss-03/dev-1`) plus `shared-244` for `ndo-shared-244/ndo`. A bare `dev-2` is accepted **only** if it is unique across clusters; otherwise the script lists the candidates and stops — never guess which cluster the user meant, ask.

Everything needs the corporate VPN. `ndo-dev-1` is decommissioned; do not use it.

## Step 0 — preflight

```
ndo-ship.sh doctor -e <env>
```
Checks docker/OrbStack, buildx, artifactory login, host arch, and kube access for the env. If it reports "NOT logged in": `ndo-ship.sh login` (interactive artifactory password prompt — the user runs it, prefix with `!` in the CLI).

## Step 1 — build (tests first)

```
ndo-ship.sh build <service> [--ticket 231239] [--skip-tests] [--no-cache]
```
- Runs unit tests first — Maven `mvn -B test` for Java services, the dockerfile's `test` stage or `go test ./...` for Go — and aborts the build if they fail. Do not pass `--skip-tests` when the user asked for "build and unit tests successful".
- Java services: runs `mvn -B -DskipTests package` after the tests so `target/*.jar` exists for the `COPY`.
- Builds `--platform linux/amd64`. **Never drop this** — the Mac is arm64, the nodes are amd64, and the mismatch only surfaces as a crashlooping pod after deploy.
- Uses `Dockerfile_local` if present, else `Dockerfile`, and `--target release` when the dockerfile has stages. See `reference/dockerfile-local.md` before writing one.
- Image ref: `[REDACTED REGISTRY]/<[REDACTED USER]>/<service>_unm_<ticket>:<utc-timestamp>`. Ticket is parsed from the git branch (`bugfix/UNM-231239` → `231239`). The timestamp tag matters: deployments run `imagePullPolicy: IfNotPresent`, so a reused tag silently keeps the old image.

The ref is cached, so `push`/`deploy` need no `--tag`.

## Step 2 — push + deploy

```
ndo-ship.sh push   <service>
ndo-ship.sh deploy <service> -e <env> --yes
# or all of it:
ndo-ship.sh ship   <service> -e <env> --yes
```

`deploy` records the currently deployed image as a rollback point, `kubectl set image`s the deployment, and waits for `rollout status`. On failure it dumps pod state.

**`deploy`/`ship`/`rollback`/`pullsecret` mutate a shared env.** They refuse to run without `--yes`, and `--yes` is only yours to pass after the user has approved *that* deploy to *that* env. Approval for one env or one ticket does not carry over.

Rollback: `ndo-ship.sh rollback <service> -e <env> --yes`.

If pods go `ImagePullBackOff`, the nodes have no credentials for the `:17009` personal repo:
```
ndo-ship.sh pullsecret <service> -e <env> --yes
```
which creates a `docker-registry` secret from the local docker keychain and patches the deployment's `imagePullSecrets`.

## Step 3 — confirm what is actually running

The single most common cause of "the fix didn't work" is the wrong image.

```
ndo-api.sh -e <env> image <service>
ndo-api.sh -e <env> pods  <service>
```
Match the tag to the build you just pushed. Product images look like `…:release_2024.4_<date>`; yours look like `…/<user>/<service>_unm_<ticket>:<timestamp>`.

## Step 4 — drive the BOM API

Auth is automatic and per-env: a keycloak password-grant token (realm `default`, client `frontend`, dev sysadm creds) is minted and refreshed on expiry. Override with `NDO_USER` / `NDO_PASS` / `NDO_REALM` / `NDO_CLIENT`. Tokens live in `~/.cache/ndo-repro/token-<env>.txt`, mode 600 — never echo one into chat or a committed file.

Stateful operation lifecycle (BOM `/business-operation-manager/v1`):
- **initiate**: `POST /operation-request/initiate?key=<opKey>` → returns `operation-request-id` (rid).
- **prepare a sub-operation**: `POST /operation-request/{rid}/prepare?key=<subOpKey>` with `{data, sources, parent-path}` (BOM injects operation-data/inputs from the session).
- **perform a read/action**: `POST /operation-request/{rid}/perform` with `{"method":"GET","url":"/consolidated-inventory-manager/v3/<path>","body":{…}}` — the inner call is wrapped.

Replacement (CIM `/v3/replacement`) endpoints, all via `perform` GET:
- `/report` — impact summary; `resolved-issues` / `unresolved-issues` is the pass/fail metric.
- `/target` — target tree (chassis + slots; does **not** expose ports/interfaces).
- `/target/slots` — slots for a target component.
- `/mapping`, `/mapping/available-target-values` — Map-To candidates (`{impact-type, impacted-entity-mkey, ref-endpoint-mkey, [filter], [only-total]}`); `total:0` = "No available interfaces".
- target insert sub-op key: `nc_op_ci_<as-is|to-be>_hw-component.replacement.target.insert.module`.

Finding ids: `/report` gives source/target mkeys; `/target` gives chassis + slot ids; a DL spec read (`/device-library/v1/restconf/data/hw-component?depth=3&filter=[{op:eq,property:id,value:[<srcId>]}]`) gives `port-interface`/`port-type`.

```
ndo-api.sh -e <env> initiate nc_op_ci_as-is_hw-component.replacement
ndo-api.sh -e <env> report <rid>
ndo-api.sh -e <env> avail  <rid> <impactMkey> <refMkey>
ndo-api.sh -e <env> get    <rid> /v3/replacement/target
```

## Step 5 — read live logs (ground truth)

```
ndo-api.sh -e <env> logs consolidated-inventory-manager 15m '\[UNM-231239\]'
```
Strips `tenant_id`/`thread`/`traceId`/`spanId`/`request_id` noise. Grep the ticket tag for the dev's INFO traces plus `WARN`/`ERROR`; correlate one call end to end by `request_id=` (drop the sed filter when you need it).

Known noise to ignore: `Unknown token audience: netcracker` — a k8s m2m quirk on the dev envs, not your bug unless the user says otherwise.

## Validating acceptance criteria

When asked to "validate the issue is resolved and acceptance criteria fulfilled", the deliverable is evidence, not an opinion:
1. State the deployed image tag and prove it is your build.
2. For each acceptance criterion, name the API call that exercises it and show the response field that decides pass/fail (e.g. `unresolved-issues: 0`, `total > 0`).
3. Show the log lines that confirm the new code path ran.
4. Report any criterion you could **not** exercise, and why — do not infer a pass from an adjacent one.

## Safety

- Read-mostly on the API side. `prepare`/`perform` writes mutate only the draft stateful session — fine for repro. Do not `/complete` a replacement unless asked.
- Deploying replaces a running service other people may be using. Confirm the env with the user first, keep the rollback point, and roll back when done if they asked you to.
- Never push to `:17099`/`:17003` (product repos) — `:17009` personal only.
- Never open MRs, push branches, or change CI without explicit approval.
- If a stateful session is polluted by earlier inserts, initiate a fresh rid rather than fighting old state.

## Pattern that works

fix in source → `ndo-ship.sh build` (tests gate it) → `push` → confirm env with user → `deploy --yes` → verify image tag → initiate/drive the exact sub-op the UI would → read the report metric → if it still fails, read CIM logs for the real reason → new hypothesis → repeat.

## Media (when QA attaches gifs/videos)
- GIF frames: Python+PIL (`Image.open(g); im.seek(i)`); crop the devtools network panel and upscale to read request names/statuses.
- Video: `ffmpeg -i in.mp4 -vf fps=1/5 out%03d.jpg`, then narrow with `-ss <start> -to <end> -vf fps=1`.
