ai-for-dummies/submitted-skills/Anonymous Operational Submission/skills/ndo-repro/SKILL.md

name, description

name

description

ndo-repro

Build an NDO microservice locally with Docker, push it to artifactory, deploy it to a dev env, then reproduce or validate the fix by driving the Business-Operation-Manager (BOM) API and reading live pod logs. Use when debugging or verifying a UNM-* ticket without waiting for CI, when the UI flow is hard to reproduce, when driving the replacement/Map-To/target-insert flow without a browser, or when the user says "repro via API", "drive BOM", "ship to <env>", "deploy my build to dev-2", "validate the fix on the cluster", "run ndo-repro in <env>". Covers env discovery across the saas-rnd-oss and ndo-shared clusters.

NDO build → deploy → repro loop

Full loop on one env, no CI wait: build the service locally, push to artifactory, repoint the k8s deployment, then drive BOM's API and read pod logs to prove the ticket's acceptance criteria.

Two scripts, both env-aware via -e <alias>:

• ~/.claude/skills/ndo-repro/ndo-ship.sh — doctor / test / build / push / deploy / status / rollback
• ~/.claude/skills/ndo-repro/ndo-api.sh — env registry / auth / BOM API / logs

Run --help on either for the full command list.

Envs

Aliases come from a discovered registry (envs.tsv, refreshed with ndo-api.sh env discover — it scans every kube context for a namespace running consolidated-inventory-manager-v1 and reads the public-gateway ingress host).

ndo-api.sh env ls                 # alias → context / namespace / gateway
ndo-api.sh -e oss-01/dev-2 env show

Alias shape is <cluster>/<env> (oss-01/dev-2, oss-03/dev-1) plus shared-244 for ndo-shared-244/ndo. A bare dev-2 is accepted only if it is unique across clusters; otherwise the script lists the candidates and stops — never guess which cluster the user meant, ask.

Everything needs the corporate VPN. ndo-dev-1 is decommissioned; do not use it.

Step 0 — preflight

ndo-ship.sh doctor -e <env>

Checks docker/OrbStack, buildx, artifactory login, host arch, and kube access for the env. If it reports "NOT logged in": ndo-ship.sh login (interactive artifactory password prompt — the user runs it, prefix with ! in the CLI).

Step 1 — build (tests first)

ndo-ship.sh build <service> [--ticket 231239] [--skip-tests] [--no-cache]

• Runs unit tests first — Maven mvn -B test for Java services, the dockerfile's test stage or go test ./... for Go — and aborts the build if they fail. Do not pass --skip-tests when the user asked for "build and unit tests successful".
• Java services: runs mvn -B -DskipTests package after the tests so target/*.jar exists for the COPY.
• Builds --platform linux/amd64. Never drop this — the Mac is arm64, the nodes are amd64, and the mismatch only surfaces as a crashlooping pod after deploy.
• Uses Dockerfile_local if present, else Dockerfile, and --target release when the dockerfile has stages. See reference/dockerfile-local.md before writing one.
• Image ref: [REDACTED REGISTRY]/<[REDACTED USER]>/<service>_unm_<ticket>:<utc-timestamp>. Ticket is parsed from the git branch (bugfix/UNM-231239 → 231239). The timestamp tag matters: deployments run imagePullPolicy: IfNotPresent, so a reused tag silently keeps the old image.

The ref is cached, so push/deploy need no --tag.

Step 2 — push + deploy

ndo-ship.sh push   <service>
ndo-ship.sh deploy <service> -e <env> --yes
# or all of it:
ndo-ship.sh ship   <service> -e <env> --yes

deploy records the currently deployed image as a rollback point, kubectl set images the deployment, and waits for rollout status. On failure it dumps pod state.

deploy/ship/rollback/pullsecret mutate a shared env. They refuse to run without --yes, and --yes is only yours to pass after the user has approved that deploy to that env. Approval for one env or one ticket does not carry over.

Rollback: ndo-ship.sh rollback <service> -e <env> --yes.

If pods go ImagePullBackOff, the nodes have no credentials for the :17009 personal repo:

ndo-ship.sh pullsecret <service> -e <env> --yes

which creates a docker-registry secret from the local docker keychain and patches the deployment's imagePullSecrets.

Step 3 — confirm what is actually running

The single most common cause of "the fix didn't work" is the wrong image.

ndo-api.sh -e <env> image <service>
ndo-api.sh -e <env> pods  <service>

Match the tag to the build you just pushed. Product images look like …:release_2024.4_<date>; yours look like …/<user>/<service>_unm_<ticket>:<timestamp>.

Step 4 — drive the BOM API

Auth is automatic and per-env: a keycloak password-grant token (realm default, client frontend, dev sysadm creds) is minted and refreshed on expiry. Override with NDO_USER / NDO_PASS / NDO_REALM / NDO_CLIENT. Tokens live in ~/.cache/ndo-repro/token-<env>.txt, mode 600 — never echo one into chat or a committed file.

Stateful operation lifecycle (BOM /business-operation-manager/v1):

• initiate: POST /operation-request/initiate?key=<opKey> → returns operation-request-id (rid).
• prepare a sub-operation: POST /operation-request/{rid}/prepare?key=<subOpKey> with {data, sources, parent-path} (BOM injects operation-data/inputs from the session).
• perform a read/action: POST /operation-request/{rid}/perform with {"method":"GET","url":"/consolidated-inventory-manager/v3/<path>","body":{…}} — the inner call is wrapped.

Replacement (CIM /v3/replacement) endpoints, all via perform GET:

• /report — impact summary; resolved-issues / unresolved-issues is the pass/fail metric.
• /target — target tree (chassis + slots; does not expose ports/interfaces).
• /target/slots — slots for a target component.
• /mapping, /mapping/available-target-values — Map-To candidates ({impact-type, impacted-entity-mkey, ref-endpoint-mkey, [filter], [only-total]}); total:0 = "No available interfaces".
• target insert sub-op key: nc_op_ci_<as-is|to-be>_hw-component.replacement.target.insert.module.

Finding ids: /report gives source/target mkeys; /target gives chassis + slot ids; a DL spec read (/device-library/v1/restconf/data/hw-component?depth=3&filter=[{op:eq,property:id,value:[<srcId>]}]) gives port-interface/port-type.

ndo-api.sh -e <env> initiate nc_op_ci_as-is_hw-component.replacement
ndo-api.sh -e <env> report <rid>
ndo-api.sh -e <env> avail  <rid> <impactMkey> <refMkey>
ndo-api.sh -e <env> get    <rid> /v3/replacement/target

Step 5 — read live logs (ground truth)

ndo-api.sh -e <env> logs consolidated-inventory-manager 15m '\[UNM-231239\]'

Strips tenant_id/thread/traceId/spanId/request_id noise. Grep the ticket tag for the dev's INFO traces plus WARN/ERROR; correlate one call end to end by request_id= (drop the sed filter when you need it).

Known noise to ignore: Unknown token audience: netcracker — a k8s m2m quirk on the dev envs, not your bug unless the user says otherwise.

Validating acceptance criteria

When asked to "validate the issue is resolved and acceptance criteria fulfilled", the deliverable is evidence, not an opinion:

1. State the deployed image tag and prove it is your build.
2. For each acceptance criterion, name the API call that exercises it and show the response field that decides pass/fail (e.g. unresolved-issues: 0, total > 0).
3. Show the log lines that confirm the new code path ran.
4. Report any criterion you could not exercise, and why — do not infer a pass from an adjacent one.

Safety

• Read-mostly on the API side. prepare/perform writes mutate only the draft stateful session — fine for repro. Do not /complete a replacement unless asked.
• Deploying replaces a running service other people may be using. Confirm the env with the user first, keep the rollback point, and roll back when done if they asked you to.
• Never push to :17099/:17003 (product repos) — :17009 personal only.
• Never open MRs, push branches, or change CI without explicit approval.
• If a stateful session is polluted by earlier inserts, initiate a fresh rid rather than fighting old state.

Pattern that works

fix in source → ndo-ship.sh build (tests gate it) → push → confirm env with user → deploy --yes → verify image tag → initiate/drive the exact sub-op the UI would → read the report metric → if it still fails, read CIM logs for the real reason → new hypothesis → repeat.

Media (when QA attaches gifs/videos)

• GIF frames: Python+PIL (Image.open(g); im.seek(i)); crop the devtools network panel and upscale to read request names/statuses.
• Video: ffmpeg -i in.mp4 -vf fps=1/5 out%03d.jpg, then narrow with -ss <start> -to <end> -vf fps=1.