Skip to content
SageMaker HyperPod Cluster Debugger

SageMaker HyperPod Cluster DebuggerSkill

by AWS Labs
Added to Onei
v0.0.1
Apache-2.0
Repository Docs
markdown Development

Summary

Read-only diagnostics for cluster-wide SageMaker HyperPod failures on EKS or Slurm — CloudFormation errors, EFA health checks, lifecycle scripts, capacity, dangling nodes and autoscaler conflicts.

Features

  • Read-only by policy — mutating commands are handed to a human, never executed
  • Covers both EKS and Slurm HyperPod orchestrators
  • Checks whether a resource is IaC-managed before suggesting any CLI fix
  • Ordered escalation: investigate, then reboot, then replace, with volume-loss warnings
  • Findings route to per-section remediation runbooks rather than generic advice
  • --validate pre-flight mode for capacity and subnet sizing before cluster creation

Install This Skill

Add this skill to your favorite AI agent in a few steps.

Any AI agent

This skill is plain instructions — it works with any assistant that accepts custom instructions or system prompts.

  1. Copy the skill content with the button below.
  2. Paste it into your agent's instruction file or system prompt (for example AGENTS.md, .cursorrules, or a custom instructions field).
  3. Ask the agent to apply the skill whenever the task matches.

Skill Content

Markdown Content

Copy this content and use it with your preferred AI agent

---
name: hyperpod-cluster-debugger
description: Diagnose and remediate cluster-wide HyperPod (EKS or Slurm) problems — creation / deployment failures (CloudFormation, EFA health check, lifecycle scripts, capacity), EKS access, node replacement, CloudFormation nested-stack errors, post-maintenance rollback state, dangling nodes, autoscaler conflicts. Includes `--validate` pre-flight. Read-only.
metadata:
  version: "0.0.1"
---

# HyperPod Cluster Debugger

**Operating policy.** Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a **Suggested command (run this yourself)** block and wait for the customer to run it. Destructive order: **investigate → reboot → replace** (replace destroys root + secondary volumes; not supported on Slurm controller nodes).

**Before any state-changing CLI: ask if it's IaC-managed.** HyperPod clusters, SGs, EKS access entries, and IAM are usually provisioned via CloudFormation / CDK / Terraform. If yes, the fix belongs in IaC — running the CLI will drift and the next deploy reverts it. Use the CLI only when IaC is unavailable (locked out, predates IaC, mid-review).

`scripts/diagnose-cluster.sh` is read-only: it collects state via AWS APIs (and SSM for Slurm controller health) and prints each issue as `[FAIL] ... → references/<file>.md § <section>`.

| Reference                                                                 | Open when                                                           |
| ------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| [cluster-diagnostics-detail.md](references/cluster-diagnostics-detail.md) | Per-finding remediation runbook (§ A–L)                             |
| [cluster-operations.md](references/cluster-operations.md)                 | Operational deep-dives (EFA SG, EKS access, SSM, Slurm, filesystem) |
| [cloudformation-errors.md](references/cloudformation-errors.md)           | § H needs the full per-resource CFN error catalog                   |
| [capacity-planning.md](references/capacity-planning.md)                   | § B or `--validate` flags capacity / subnet sizing                  |
| [lifecycle-scripts.md](references/lifecycle-scripts.md)                   | § C points at a specific lifecycle failure                          |
| [iam-permissions.md](references/iam-permissions.md)                       | Full IAM policy for the diagnostic                                  |

---

## Workflow

1. Collect HyperPod cluster name (not EKS name), region, exact error string.
2. Run `scripts/diagnose-cluster.sh` (or `--validate` for pre-create).
3. For every `[FAIL]` line, `Read` the referenced section.
4. Present finding, root cause, and the Suggested-command block verbatim. Wait for customer approval.
5. Re-run the diagnostic to confirm.

---

## Step 1: Run diagnostics

```bash
# Diagnose an existing cluster:
bash scripts/diagnose-cluster.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION>

# Pre-flight (no cluster needed) — validates SGs, subnets, IAM, VPC endpoints,
# optionally S3 lifecycle scripts and per-AZ capacity:
bash scripts/diagnose-cluster.sh --validate --region <REGION> \
  --sg-ids <sg-1,sg-2> --subnet-ids <sub-1,sub-2> [--iam-role <role-arn>] \
  [--s3-uri s3://<BUCKET>/path/] [--instance-type ml.p5.48xlarge]
```

Pass `--instance-type` when the target instance type is known — enables the per-AZ capacity check (warns if none of the provided subnets are in an AZ that offers that type, which causes insufficient-capacity failures at creation time).

Tags: `[PASS]` · `[FAIL]` (counted, has `→ references/...` pointer) · `[WARN]` · `[INFO]`. Priorities: **P0** blocks operation · **P1** degraded · **P2** informational.

---

## Step 2: Match signal → section

**Error messages / events:**

| Signal                                                                       | Section                                                        |
| ---------------------------------------------------------------------------- | -------------------------------------------------------------- |
| `"EFA health checks did not run successfully"` (public-doc verbatim signal)  | **[A: EFA Health Checks](#a-efa-health-checks)**               |
| Insufficient-capacity or AZ-mismatch failure at creation                     | **[B: Capacity & AZ](#b-capacity--az)**                        |
| Lifecycle-script failure or timeout during provisioning                      | **[C: Lifecycle Scripts](#c-lifecycle-scripts)**               |
| kubectl auth error (server asks for credentials / no API group list)         | **[D: EKS Access](#d-eks-access--kubectl)**                    |
| `InService` but not all instances visible                                    | **[E: Cluster Provisioning](#e-cluster-provisioning)**         |
| `"Target is not connected"` / SSM errors                                     | **[F: SSM Connectivity](#f-ssm-connectivity)**                 |
| Node replacement not happening / `batch-replace` not working                 | **[G: Node Replacement](#g-node-replacement)**                 |
| `"Embedded stack failed"` / any CloudFormation error                         | **[H: CloudFormation Errors](#h-cloudformation-errors)**       |
| `UpdateClusterSoftware` failed or cluster in post-maintenance rollback state | **[J: AMI & Cluster Updates](#j-ami--cluster-updates)**        |
| Dangling / orphaned nodes in EKS vs `list-cluster-nodes`                     | **[K: Dangling Nodes & Cleanup](#k-dangling-nodes--cleanup)**  |
| Cluster Autoscaler breaks after HyperPod attached                            | **[L: Autoscaler Compatibility](#l-autoscaler-compatibility)** |
| Slow I/O, FSx throughput saturated                                           | [cluster-operations.md § 9](references/cluster-operations.md)  |
| Slurm node name → instance ID lookup                                         | **[I: Utilities](#i-utilities)**                               |

---

## A: EFA Health Checks

SG missing self-reference. Add inbound + outbound self-ref to every SG on the cluster, plus least-privilege egress for the AWS APIs the node needs (HTTPS 443 to S3 / ECR / SageMaker / SSM / STS / CloudWatch Logs — via VPC-endpoint prefix-lists when possible). Full procedure: [cluster-diagnostics-detail.md § A](references/cluster-diagnostics-detail.md#a-efa-health-checks).

## B: Capacity & AZ

Instance type unavailable in the requested AZ. Verify with `describe-instance-type-offerings`, then change AZ, use Flexible Training Plans, or request ODCR. Full: [§ B](references/cluster-diagnostics-detail.md#b-capacity--az) · strategy: [capacity-planning.md](references/capacity-planning.md).

## C: Lifecycle Scripts

Script failed or timed out during provisioning. Read CloudWatch under `/aws/sagemaker/Clusters/<name>/<id>` — common causes: missing S3 VPC endpoint, IAM gap, CRLF line endings, instance-group name mismatch. Full: [§ C](references/cluster-diagnostics-detail.md#c-lifecycle-scripts) · layout: [lifecycle-scripts.md](references/lifecycle-scripts.md).

## D: EKS Access / kubectl

IAM identity not in EKS access entries. Verify with `sts get-caller-identity`, create an access entry with admin policy, update kubeconfig. Full: [§ D](references/cluster-diagnostics-detail.md#d-eks-access--kubectl).

## E: Cluster Provisioning

`InService` without all instances is expected under Continuous Provisioning — failures surface as events, not cluster errors. For stuck `Creating`/`Updating`/`Deleting`: check CFN nested stacks (§ H), IAM, capacity, events; if stuck `Deleting` check VPC ENI dependencies. Full: [§ E](references/cluster-diagnostics-detail.md#e-cluster-provisioning).

## F: SSM Connectivity

`Target is not connected`: use `sagemaker-cluster:<CLUSTER_ID>_<GROUP>-<INSTANCE_ID>` format (not raw EC2 ID), install session-manager-plugin, confirm node `Running`. Check IAM + VPC endpoints on timeouts. Full: [§ F](references/cluster-diagnostics-detail.md#f-ssm-connectivity).

## G: Node Replacement

Auto-repair: confirm `NodeRecovery=Automatic`, check Health Monitoring Agent (HMA) logs + node labels / Slurm reason, confirm capacity. Manual: reboot first, replace only if reboot fails. Replace requires the cluster to have been patched via `UpdateClusterSoftware` at least once and cannot target a Slurm controller node. Full: [§ G](references/cluster-diagnostics-detail.md#g-node-replacement).

## H: CloudFormation Errors

`Embedded stack failed` hides the real error. Drill into nested stacks via Events tab (filter Failed) until you reach a non-stack resource. CLI: `describe-stack-events --query 'StackEvents[?ResourceStatus==\`CREATE_FAILED\`]'`. Also covers SLR creation failures and permission-boundary denials. Full: [§ H](references/cluster-diagnostics-detail.md#h-cloudformation-errors) · catalog: [cloudformation-errors.md](references/cloudformation-errors.md).

## I: Utilities

Map Slurm node names (`ip-10-x-y-z`) to HyperPod instance IDs via `list-cluster-nodes` or on-node `/opt/ml/config/resource_config.json`. Full: [§ I](references/cluster-diagnostics-detail.md#i-utilities).

## J: AMI & Cluster Updates

`UpdateClusterSoftware` fails and rolls back, or the cluster stays in a post-maintenance rollback state. Common causes: lifecycle script incompatible with new AMI, HMA version too old, insufficient rolling-update capacity. If the cluster has active nodes, collect diagnostics and escalate rather than delete-and-recreate. Full: [§ J](references/cluster-diagnostics-detail.md#j-ami--cluster-updates).

## K: Dangling Nodes & Cleanup

Nodes in `kubectl get nodes` but not in `list-cluster-nodes` (ghost EKS nodes), or the inverse (HyperPod nodes that never registered kubelet). Script flags both. Full: [§ K](references/cluster-diagnostics-detail.md#k-dangling-nodes--cleanup).

## L: Autoscaler Compatibility

Cluster Autoscaler errors on HyperPod provider IDs and breaks autoscaling for all node groups. No officially endorsed workaround — escalate to AWS Support. Karpenter does not conflict with HyperPod nodes by default. Full: [§ L](references/cluster-diagnostics-detail.md#l-autoscaler-compatibility).

---

## Prerequisites

- `aws` CLI v2.13+ authenticated to the cluster's account
- `jq`, `python3`, `bash` 4.2+
- `kubectl` authenticated to the EKS cluster (EKS checks skipped if absent)
- `session-manager-plugin` (Slurm controller health checks only)

IAM policy: [references/iam-permissions.md](references/iam-permissions.md).

## Defaults

- **Region** — required: pass `--region` or set `$AWS_DEFAULT_REGION`.
- **Mode** — `--cluster <NAME>` (diagnose) or `--validate` (pre-create).
- **Event window** — up to 500 most recent events (5 × 100, paginated).
- **Colors** — auto-disabled on non-TTY; `--no-color` to force off.

## Error handling

| Failure                                             | Script                                                     | Tell the customer                                     |
| --------------------------------------------------- | ---------------------------------------------------------- | ----------------------------------------------------- |
| `aws sts get-caller-identity` fails                 | Exit 1                                                     | "Fix AWS credentials and rerun."                      |
| Cluster not found                                   | Exit 1 after listing region's clusters                     | "Confirm HyperPod cluster name (not EKS) and region." |
| `sagemaker:*` / `ec2:*` / `eks:*` / `logs:*` denied | Warn, add `Missing IAM permission for <API>`, continue     | "Grant the listed IAM action and rerun."              |
| `kubectl` absent or unauthenticated                 | Skip EKS checks (access entries, add-ons, aws-auth, nodes) | "Install/authenticate kubectl."                       |
| `session-manager-plugin` absent (Slurm)             | Skip Slurm controller probe                                | "Install session-manager-plugin."                     |
| SSM throttled / times out (180s)                    | Retry with backoff; warn and continue if still failing     | "Rerun later — script is idempotent."                 |
| CloudWatch log group not found                      | Skip CloudWatch check                                      | "CloudWatch not configured on this cluster."          |

Exit codes: `0` no critical failures · `1` one or more critical failures (cluster not found, fatal prerequisite missing, or any `[FAIL]` in diagnose or `--validate` mode). `[WARN]` lines do not affect the exit code.

## Skill delegation

| Need                            | Use                        |
| ------------------------------- | -------------------------- |
| Shell on nodes                  | `hyperpod-ssm`             |
| Version comparison across nodes | `hyperpod-version-checker` |

## Escalate to AWS Support

Escalate when:

1. EFA health checks fail despite correct SG rules.
2. Capacity errors persist despite a valid Flexible Training Plan / ODCR.
3. Node replacement fails repeatedly without clear events / log signal.
4. Cluster stuck in a non-terminal state (`Creating`, `Updating`, or a post-maintenance rollback state) for an extended period.
5. CloudFormation root-cause is an internal service error.

### Before opening the case

Run these commands and attach the output. Goal: AWS Support has everything at case open.

```bash
# 1. Cluster identity + status (confirms region, ARN, orchestrator, instance groups)
aws sagemaker describe-cluster --cluster-name <CLUSTER> --region <REGION>

# 2. Full cluster-level diagnostic bundle
bash scripts/diagnose-cluster.sh --cluster <CLUSTER> --region <REGION> > diag.txt

# 3. Per-node log/config bundle to S3 (delegates to hyperpod-issue-report skill)
#    See skills/hyperpod-issue-report/SKILL.md for the exact invocation.
```

### Include in the case

- Cluster name + ARN (or `ClusterId` suffix) and AWS region
- `ClusterStatus` + `FailureMessage` from `describe-cluster`
- Timestamp window (UTC start / end) of the failure
- Exact error strings observed (copy verbatim from events / logs / console)
- Affected instance IDs / `NodeLogicalId`s / instance group names
- `diag.txt` from step 2 above
- S3 URI of the `hyperpod-issue-report` bundle from step 3

Usage Instructions

Learn how to use this skill with different AI agents.

Generic Instructions

Install the marketplace with /plugin marketplace add awslabs/agent-plugins, then install the sagemaker-ai plugin. Triggers on HyperPod cluster creation failures, CloudFormation nested-stack errors, EFA health check failures, node replacement and autoscaler conflicts.

Example Usage

My HyperPod EKS cluster has been stuck in Failed since the last CloudFormation deploy — work out what actually broke and what I should run.

Description

A SageMaker HyperPod cluster that will not come up is an expensive problem, and the failure is rarely where the error message points. This official AWS skill gives a coding agent a structured diagnostic pass over cluster-wide HyperPod problems on both EKS and Slurm: creation and deployment failures, CloudFormation nested-stack errors, EFA health check failures, lifecycle script failures, capacity and subnet sizing, EKS access, node replacement, post-maintenance rollback state, dangling nodes and autoscaler conflicts.

Read-only by construction

The skill's operating policy is the interesting part. The agent runs diagnostics itself — scripts/diagnose-cluster.sh collects state through AWS APIs, and SSM for Slurm controller health — but it never runs a command that changes cluster, node or workload state. Anything mutating is presented as a "Suggested command (run this yourself)" block for a human to execute. Escalation is ordered explicitly: investigate, then reboot, then replace, with the warning that replacement destroys root and secondary volumes and is not supported on Slurm controller nodes at all.

The infrastructure-as-code check

Before any state-changing CLI suggestion, the skill asks whether the resource is IaC-managed — HyperPod clusters, security groups, EKS access entries and IAM almost always are. If it is, the fix belongs in CloudFormation, CDK or Terraform, because running the CLI creates drift the next deploy silently reverts. The CLI path is reserved for cases where IaC genuinely is not available: locked out, predating IaC, or mid-review. This is the kind of institutional knowledge that separates a useful diagnostic from one that makes tomorrow worse.

Findings that route somewhere

The diagnostic prints each issue as [FAIL] ... → references/<file>.md § <section>, pointing at per-finding remediation runbooks: a CloudFormation per-resource error catalog, capacity and subnet planning, lifecycle script failures, operational deep-dives on EFA security groups, EKS access, SSM, Slurm and filesystems, plus the full IAM policy the diagnostic itself needs. A --validate pre-flight mode runs the same checks before a cluster is created.

Part of the sagemaker-ai plugin in AWS Labs' official agent plugins repository, Apache-2.0 licensed.

Install

Related Skills

Skill: NVIDIA RAG Blueprint

by NVIDIA

Deploy, configure and troubleshoot the NVIDIA RAG Blueprint from your agent — Docker Compose, Helm or library, with feature toggles for VLM, guardrails and agentic RAG.

Development

Skill: AWS Lambda Durable Functions

by AWS Labs

Build multi-step Lambda workflows that survive interruption and run for up to a year — the replay model, step operations, wait/callback patterns, saga-style compensation and local testing.

Development

Skill: MSBuild Binlog Failure Analysis

by .NET Team at Microsoft

Diagnose an MSBuild build failure from a .binlog file — query the binary log through a dedicated MCP server instead of grepping console output, with a text-replay fallback.

Development
Browse all skills →