Marcos Paulo
137 lines
8.7 KiB
Markdown
137 lines
8.7 KiB
Markdown
---
|
|
name: ndo-repro
|
|
description: Build an NDO microservice locally with Docker, push it to artifactory, deploy it to a dev env, then reproduce or validate the fix by driving the Business-Operation-Manager (BOM) API and reading live pod logs. Use when debugging or verifying a UNM-* ticket without waiting for CI, when the UI flow is hard to reproduce, when driving the replacement/Map-To/target-insert flow without a browser, or when the user says "repro via API", "drive BOM", "ship to <env>", "deploy my build to dev-2", "validate the fix on the cluster", "run ndo-repro in <env>". Covers env discovery across the saas-rnd-oss and ndo-shared clusters.
|
|
---
|
|
|
|
# NDO build → deploy → repro loop
|
|
|
|
Full loop on one env, no CI wait: build the service locally, push to artifactory, repoint the k8s deployment, then drive BOM's API and read pod logs to prove the ticket's acceptance criteria.
|
|
|
|
Two scripts, both env-aware via `-e <alias>`:
|
|
- `~/.claude/skills/ndo-repro/ndo-ship.sh` — doctor / test / build / push / deploy / status / rollback
|
|
- `~/.claude/skills/ndo-repro/ndo-api.sh` — env registry / auth / BOM API / logs
|
|
|
|
Run `--help` on either for the full command list.
|
|
|
|
## Envs
|
|
|
|
Aliases come from a discovered registry (`envs.tsv`, refreshed with `ndo-api.sh env discover` — it scans every kube context for a namespace running `consolidated-inventory-manager-v1` and reads the `public-gateway` ingress host).
|
|
|
|
```
|
|
ndo-api.sh env ls # alias → context / namespace / gateway
|
|
ndo-api.sh -e oss-01/dev-2 env show
|
|
```
|
|
|
|
Alias shape is `<cluster>/<env>` (`oss-01/dev-2`, `oss-03/dev-1`) plus `shared-244` for `ndo-shared-244/ndo`. A bare `dev-2` is accepted **only** if it is unique across clusters; otherwise the script lists the candidates and stops — never guess which cluster the user meant, ask.
|
|
|
|
Everything needs the corporate VPN. `ndo-dev-1` is decommissioned; do not use it.
|
|
|
|
## Step 0 — preflight
|
|
|
|
```
|
|
ndo-ship.sh doctor -e <env>
|
|
```
|
|
Checks docker/OrbStack, buildx, artifactory login, host arch, and kube access for the env. If it reports "NOT logged in": `ndo-ship.sh login` (interactive artifactory password prompt — the user runs it, prefix with `!` in the CLI).
|
|
|
|
## Step 1 — build (tests first)
|
|
|
|
```
|
|
ndo-ship.sh build <service> [--ticket 231239] [--skip-tests] [--no-cache]
|
|
```
|
|
- Runs unit tests first — Maven `mvn -B test` for Java services, the dockerfile's `test` stage or `go test ./...` for Go — and aborts the build if they fail. Do not pass `--skip-tests` when the user asked for "build and unit tests successful".
|
|
- Java services: runs `mvn -B -DskipTests package` after the tests so `target/*.jar` exists for the `COPY`.
|
|
- Builds `--platform linux/amd64`. **Never drop this** — the Mac is arm64, the nodes are amd64, and the mismatch only surfaces as a crashlooping pod after deploy.
|
|
- Uses `Dockerfile_local` if present, else `Dockerfile`, and `--target release` when the dockerfile has stages. See `reference/dockerfile-local.md` before writing one.
|
|
- Image ref: `[REDACTED REGISTRY]/<[REDACTED USER]>/<service>_unm_<ticket>:<utc-timestamp>`. Ticket is parsed from the git branch (`bugfix/UNM-231239` → `231239`). The timestamp tag matters: deployments run `imagePullPolicy: IfNotPresent`, so a reused tag silently keeps the old image.
|
|
|
|
The ref is cached, so `push`/`deploy` need no `--tag`.
|
|
|
|
## Step 2 — push + deploy
|
|
|
|
```
|
|
ndo-ship.sh push <service>
|
|
ndo-ship.sh deploy <service> -e <env> --yes
|
|
# or all of it:
|
|
ndo-ship.sh ship <service> -e <env> --yes
|
|
```
|
|
|
|
`deploy` records the currently deployed image as a rollback point, `kubectl set image`s the deployment, and waits for `rollout status`. On failure it dumps pod state.
|
|
|
|
**`deploy`/`ship`/`rollback`/`pullsecret` mutate a shared env.** They refuse to run without `--yes`, and `--yes` is only yours to pass after the user has approved *that* deploy to *that* env. Approval for one env or one ticket does not carry over.
|
|
|
|
Rollback: `ndo-ship.sh rollback <service> -e <env> --yes`.
|
|
|
|
If pods go `ImagePullBackOff`, the nodes have no credentials for the `:17009` personal repo:
|
|
```
|
|
ndo-ship.sh pullsecret <service> -e <env> --yes
|
|
```
|
|
which creates a `docker-registry` secret from the local docker keychain and patches the deployment's `imagePullSecrets`.
|
|
|
|
## Step 3 — confirm what is actually running
|
|
|
|
The single most common cause of "the fix didn't work" is the wrong image.
|
|
|
|
```
|
|
ndo-api.sh -e <env> image <service>
|
|
ndo-api.sh -e <env> pods <service>
|
|
```
|
|
Match the tag to the build you just pushed. Product images look like `…:release_2024.4_<date>`; yours look like `…/<user>/<service>_unm_<ticket>:<timestamp>`.
|
|
|
|
## Step 4 — drive the BOM API
|
|
|
|
Auth is automatic and per-env: a keycloak password-grant token (realm `default`, client `frontend`, dev sysadm creds) is minted and refreshed on expiry. Override with `NDO_USER` / `NDO_PASS` / `NDO_REALM` / `NDO_CLIENT`. Tokens live in `~/.cache/ndo-repro/token-<env>.txt`, mode 600 — never echo one into chat or a committed file.
|
|
|
|
Stateful operation lifecycle (BOM `/business-operation-manager/v1`):
|
|
- **initiate**: `POST /operation-request/initiate?key=<opKey>` → returns `operation-request-id` (rid).
|
|
- **prepare a sub-operation**: `POST /operation-request/{rid}/prepare?key=<subOpKey>` with `{data, sources, parent-path}` (BOM injects operation-data/inputs from the session).
|
|
- **perform a read/action**: `POST /operation-request/{rid}/perform` with `{"method":"GET","url":"/consolidated-inventory-manager/v3/<path>","body":{…}}` — the inner call is wrapped.
|
|
|
|
Replacement (CIM `/v3/replacement`) endpoints, all via `perform` GET:
|
|
- `/report` — impact summary; `resolved-issues` / `unresolved-issues` is the pass/fail metric.
|
|
- `/target` — target tree (chassis + slots; does **not** expose ports/interfaces).
|
|
- `/target/slots` — slots for a target component.
|
|
- `/mapping`, `/mapping/available-target-values` — Map-To candidates (`{impact-type, impacted-entity-mkey, ref-endpoint-mkey, [filter], [only-total]}`); `total:0` = "No available interfaces".
|
|
- target insert sub-op key: `nc_op_ci_<as-is|to-be>_hw-component.replacement.target.insert.module`.
|
|
|
|
Finding ids: `/report` gives source/target mkeys; `/target` gives chassis + slot ids; a DL spec read (`/device-library/v1/restconf/data/hw-component?depth=3&filter=[{op:eq,property:id,value:[<srcId>]}]`) gives `port-interface`/`port-type`.
|
|
|
|
```
|
|
ndo-api.sh -e <env> initiate nc_op_ci_as-is_hw-component.replacement
|
|
ndo-api.sh -e <env> report <rid>
|
|
ndo-api.sh -e <env> avail <rid> <impactMkey> <refMkey>
|
|
ndo-api.sh -e <env> get <rid> /v3/replacement/target
|
|
```
|
|
|
|
## Step 5 — read live logs (ground truth)
|
|
|
|
```
|
|
ndo-api.sh -e <env> logs consolidated-inventory-manager 15m '\[UNM-231239\]'
|
|
```
|
|
Strips `tenant_id`/`thread`/`traceId`/`spanId`/`request_id` noise. Grep the ticket tag for the dev's INFO traces plus `WARN`/`ERROR`; correlate one call end to end by `request_id=` (drop the sed filter when you need it).
|
|
|
|
Known noise to ignore: `Unknown token audience: netcracker` — a k8s m2m quirk on the dev envs, not your bug unless the user says otherwise.
|
|
|
|
## Validating acceptance criteria
|
|
|
|
When asked to "validate the issue is resolved and acceptance criteria fulfilled", the deliverable is evidence, not an opinion:
|
|
1. State the deployed image tag and prove it is your build.
|
|
2. For each acceptance criterion, name the API call that exercises it and show the response field that decides pass/fail (e.g. `unresolved-issues: 0`, `total > 0`).
|
|
3. Show the log lines that confirm the new code path ran.
|
|
4. Report any criterion you could **not** exercise, and why — do not infer a pass from an adjacent one.
|
|
|
|
## Safety
|
|
|
|
- Read-mostly on the API side. `prepare`/`perform` writes mutate only the draft stateful session — fine for repro. Do not `/complete` a replacement unless asked.
|
|
- Deploying replaces a running service other people may be using. Confirm the env with the user first, keep the rollback point, and roll back when done if they asked you to.
|
|
- Never push to `:17099`/`:17003` (product repos) — `:17009` personal only.
|
|
- Never open MRs, push branches, or change CI without explicit approval.
|
|
- If a stateful session is polluted by earlier inserts, initiate a fresh rid rather than fighting old state.
|
|
|
|
## Pattern that works
|
|
|
|
fix in source → `ndo-ship.sh build` (tests gate it) → `push` → confirm env with user → `deploy --yes` → verify image tag → initiate/drive the exact sub-op the UI would → read the report metric → if it still fails, read CIM logs for the real reason → new hypothesis → repeat.
|
|
|
|
## Media (when QA attaches gifs/videos)
|
|
- GIF frames: Python+PIL (`Image.open(g); im.seek(i)`); crop the devtools network panel and upscale to read request names/statuses.
|
|
- Video: `ffmpeg -i in.mp4 -vf fps=1/5 out%03d.jpg`, then narrow with `-ss <start> -to <end> -vf fps=1`.
|