Skip to content

Lab 21: Load balance workloads and troubleshoot with Network Watcher

Previous: Lab 20 · Catalog · Next: Lab 22

This self-contained lab uses Azure CLI commands hosted in PowerShell. Complete the guided lane or the automated lane—not both with the same run ID.

Scenario, role, and outcome

Scenario: You are a network operations administrator publishing a resilient http backend. Build and prove this disposable service path: A Standard public frontend and outbound rule connect to two private backend NICs; probe and rule traffic traverse the NSG, Network Watcher agents run on both VMs, and Connection Monitor measures the VM-to-VM TCP path.

Learner role: A network operations administrator publishing a resilient HTTP backend.

Outcome: Use Azure CLI in PowerShell to create two private backends behind a standard load balancer, configure health probing and balanced traffic, and reconcile frontend, backend health, and diagnostic evidence, with recoverable state and deterministic cleanup.

Item Value
Duration 150 minutes
Difficulty applied
Cost class elevated
Command surface Azure CLI (az, az rest, Bicep, AzCopy, or KQL where required) hosted in PowerShell
Live state Not executed during this offline rebuild

Completion criteria:

  • All required checkpoints report pass.
  • The deterministic break/fix is injected, diagnosed, and repaired.
  • cleanup.json reports no active run-owned resources, with any retained item explicitly documented.
LAB-21 five-checkpoint roadmap: Load balance workloads and troubleshoot with Network Watcher
Five learning checkpoints in task order. These arrows show the exercise sequence; see the architecture figure for service connections. Open at full size.

Objectives and checkpoints

Objective Skill Checkpoints
NW-VNET-05 Troubleshoot network connectivity LAB21-CP01, LAB21-CP05
NW-DNSLB-02 Configure an internal or public load balancer LAB21-CP02, LAB21-CP05
NW-DNSLB-03 Troubleshoot load balancing LAB21-CP03, LAB21-CP05
MR-MONITOR-06 Use Azure Network Watcher and Connection monitor LAB21-CP04, LAB21-CP05

The authoritative objective wording comes from the Microsoft AZ-104 study guide, skills measured as of 2026-04-17.

Architecture and service topology

LAB-21: Load balance workloads and troubleshoot with Network Watcher; service topology and connections
Load balance workloads and troubleshoot with Network Watcher — topology for this lab. Open at full size.

The editable source is architecture.mmd. A Standard public frontend and outbound rule connect to two private backend NICs; probe and rule traffic traverse the NSG, Network Watcher agents run on both VMs, and Connection Monitor measures the VM-to-VM TCP path.

Concept primer and design decisions

A load balancer is only functional when backend NICs join the pool, the health probe succeeds, and NSGs permit the probe and data paths. Network Watcher adds point-in-time diagnostics and longitudinal connection monitoring.

Design decisions:

  • Give backends no individual public IPs.
  • Install distinct responses to prove distribution.
  • Validate probe health before the frontend.

Required and optional inputs

Input Required Source Safe example Gate behavior
run-id — Unique lowercase run ownership identifier Yes parameter az104l21-01 block
subscription-id — Expected disposable subscription ID Yes parameter-or-environment (AZ104_SUBSCRIPTION_ID) 00000000-0000-0000-0000-000000000000 block
location — Approved primary Azure region Yes parameter-or-environment (AZ104_LOCATION) westeurope block

Secrets stay in temporary environment variables and are never written to run.json, validation evidence, or Git. A skip-checkpoint gate produces a visible partial result; it never becomes a pass.

Read-only preflight

Readiness, preview, explicit execution, and result validation
Follow the read-only checks and execution gates defined by this lab. A failed required gate blocks progress. Open at full size.

Sign in deliberately, inspect the active context, then run the lab preflight. It never signs in, changes context, installs an extension, registers a provider, or creates a resource.

az login
az account show --query '{cloud:environmentName,subscription:id,tenant:tenantId,user:user.name}' --output json
./scripts/cli/Preflight.ps1 -SubscriptionId $env:AZ104_SUBSCRIPTION_ID -RunId 'az104l21-01' -Location 'westeurope'

Representative redacted output:

context                 True  cloud=AzureCloud; tenant=<redacted>; subscription=<redacted>
azure-cli               True  2.88.x
provider:<namespace>    True  Registered
region                  True  westeurope
Preflight passed. It performed no Azure mutation.

If a required row fails, stop. Correct the local tool, context, provider, quota, SKU, region, permission, or input outside the lab, then rerun preflight.

Task 1 — Verify load-balancer and Network Watcher diagnostics support

Checkpoint: LAB21-CP01

Purpose and operational relevance: Read the provider, region, SKU, feature, quota, and companion-tool signals needed for verify load-balancer and network watcher diagnostics support, without changing Azure state. The change is unsafe when network Watcher connectivity tests require a supported source VM, agent, and outbound service path. This checkpoint therefore proves: The Standard load-balancer SKU and Network Watcher diagnostic commands are available in the selected region.

Run these commands from PowerShell. If you already used the automated lane, choose a new run ID before trying this guided lane.

az provider show --namespace Microsoft.Network --query registrationState --output tsv
az provider show --namespace Microsoft.Compute --query registrationState --output tsv
az network watcher list --query "[?location=='$Location'].{name:name,resourceGroup:resourceGroup}" --output table

Expected state: The Standard load-balancer SKU and Network Watcher diagnostic commands are available in the selected region.

Representative redacted output:

The Standard load-balancer SKU and Network Watcher diagnostic commands are available in the selected region.

Positive validation:

$network = az provider show --namespace Microsoft.Network --query registrationState --output tsv
if ($network -ne 'Registered') { throw 'Microsoft.Network is not registered.' }

Expected positive result: The Standard load-balancer SKU and Network Watcher diagnostic commands are available in the selected region.

Negative validation:

$watchers = az network watcher list --query "[?location=='$Location'] | length(@)" --output tsv
if ([int]$watchers -lt 1) { throw 'Network Watcher is not available in the selected region.' }

Expected negative result: LAB21-CP01 negative boundary: The undesired state is absent; the negative assertion must not report: Network Watcher is not available in the selected region.

Evidence to retain:

  • LAB21-CP01 UTC result for verify load-balancer and network watcher diagnostics support.
  • Exact run-owned resource/object ID returned by the commands in LAB21-CP01.
  • Positive assertion proving: The Standard load-balancer SKU and Network Watcher diagnostic commands are available in the selected region.
  • Negative assertion proving absence of: Network Watcher is not available in the selected region.

Common failure and safe retry: Network Watcher connectivity tests require a supported source VM, agent, and outbound service path. Retry LAB21-CP01 for verify load-balancer and network watcher diagnostics support only after its exact recorded ID and current property are read. If the ID exists, reissue only the idempotent update; if it is absent, repeat only the owning create command, then rerun both assertions.

Cleanup dependency: After LAB21-CP02, LAB21-CP03, LAB21-CP04, LAB21-CP05, remove the exact manifest IDs created or configured by LAB21-CP01 for verify load-balancer and network watcher diagnostics support; its residual probe must then return zero active IDs.

Task 2 — Create two private backends behind a Standard load balancer

Checkpoint: LAB21-CP02

Purpose and operational relevance: Provision and immediately inventory the exact run-owned resources needed to create two private backends behind a standard load balancer, so partial completion remains recoverable. The change is unsafe when a backend NIC in the wrong VNet or region cannot join the intended pool. This checkpoint therefore proves: Both backend NICs join the pool and the frontend, health probe, and rule reference one consistent load-balancer resource.

Run these commands from PowerShell. If you already used the automated lane, choose a new run ID before trying this guided lane.

$vnet = "vnet-$suffix"; $publicIp = "pip-$suffix"; $loadBalancer = "lb-$suffix"; $nsg = "nsg-$suffix"; $vm1 = "vm-1-$suffix"; $vm2 = "vm-2-$suffix"
$keyPath = Join-Path $StateDir 'id_ed25519'
ssh-keygen -t ed25519 -N '' -f $keyPath | Out-Null
az network vnet create --resource-group $ResourceGroupName --name $vnet --address-prefixes 10.21.0.0/16 --subnet-name backend --subnet-prefixes 10.21.1.0/24 --output none
az network public-ip create --resource-group $ResourceGroupName --name $publicIp --location $Location --sku Standard --allocation-method Static --output none
az network lb create --resource-group $ResourceGroupName --name $loadBalancer --sku Standard --public-ip-address $publicIp --frontend-ip-name frontend --backend-pool-name backend --output none
az network lb probe create --resource-group $ResourceGroupName --lb-name $loadBalancer --name ssh --protocol Tcp --port 22 --interval 15 --threshold 2 --output none
az network lb rule create --resource-group $ResourceGroupName --lb-name $loadBalancer --name ssh --protocol Tcp --frontend-port 2222 --backend-port 22 --frontend-ip-name frontend --backend-pool-name backend --probe-name ssh --idle-timeout 4 --disable-outbound-snat true --output none
az network lb outbound-rule create --resource-group $ResourceGroupName --lb-name $loadBalancer --name outbound --frontend-ip-configs frontend --address-pool backend --protocol All --outbound-ports 1024 --idle-timeout 4 --output none

Expected state: Both backend NICs join the pool and the frontend, health probe, and rule reference one consistent load-balancer resource.

Representative redacted output:

Both backend NICs join the pool and the frontend, health probe, and rule reference one consistent load-balancer resource.

Positive validation:

$probePort = az network lb probe show --resource-group $ResourceGroupName --lb-name $loadBalancer --name ssh --query port --output tsv
if ([int]$probePort -ne 22) { throw "Probe port is $probePort." }

Expected positive result: Both backend NICs join the pool and the frontend, health probe, and rule reference one consistent load-balancer resource.

Negative validation:

$frontendPort = az network lb rule show --resource-group $ResourceGroupName --lb-name $loadBalancer --name ssh --query frontendPort --output tsv
if ([int]$frontendPort -ne 2222) { throw "Frontend port is $frontendPort." }

Expected negative result: LAB21-CP02 negative boundary: The undesired state is absent; the negative assertion must not report: Frontend port is reported frontendPort.

Evidence to retain:

  • LAB21-CP02 UTC result for create two private backends behind a standard load balancer.
  • Exact run-owned resource/object ID returned by the commands in LAB21-CP02.
  • Positive assertion proving: Both backend NICs join the pool and the frontend, health probe, and rule reference one consistent load-balancer resource.
  • Negative assertion proving absence of: Frontend port is reported frontendPort.

Common failure and safe retry: A backend NIC in the wrong VNet or region cannot join the intended pool. Retry LAB21-CP02 for create two private backends behind a standard load balancer only after its exact recorded ID and current property are read. If the ID exists, reissue only the idempotent update; if it is absent, repeat only the owning create command, then rerun both assertions.

Cleanup dependency: After LAB21-CP03, LAB21-CP04, LAB21-CP05, remove the exact manifest IDs created or configured by LAB21-CP02 for create two private backends behind a standard load balancer; its residual probe must then return zero active IDs.

Task 3 — Configure health probing and balanced traffic

Checkpoint: LAB21-CP03

Purpose and operational relevance: Configure and independently inspect the control-plane and data-path properties needed to configure health probing and balanced traffic. The change is unsafe when a probe port with no listening workload leaves every backend unhealthy even though the rule exists. This checkpoint therefore proves: The TCP probe and load-balancing rule share the intended ports, backend pool, and idle-timeout behavior.

Run these commands from PowerShell. If you already used the automated lane, choose a new run ID before trying this guided lane.

az network nsg create --resource-group $ResourceGroupName --name $nsg --output none
az network nsg rule create --resource-group $ResourceGroupName --nsg-name $nsg --name AllowLoadBalancerProbe --priority 200 --direction Inbound --access Allow --protocol Tcp --source-address-prefixes AzureLoadBalancer --destination-port-ranges 22 --output none
az network nsg rule create --resource-group $ResourceGroupName --nsg-name $nsg --name AllowFrontendSsh --priority 210 --direction Inbound --access Allow --protocol Tcp --source-address-prefixes Internet --destination-port-ranges 22 --output none
az network nic create --resource-group $ResourceGroupName --name "nic-1-$suffix" --vnet-name $vnet --subnet backend --network-security-group $nsg --lb-name $loadBalancer --lb-address-pools backend --output none
az network nic create --resource-group $ResourceGroupName --name "nic-2-$suffix" --vnet-name $vnet --subnet backend --network-security-group $nsg --lb-name $loadBalancer --lb-address-pools backend --output none
az vm create --resource-group $ResourceGroupName --name $vm1 --nics "nic-1-$suffix" --image Ubuntu2204 --size Standard_B1s --admin-username azureadmin --ssh-key-values "$keyPath.pub" --output none
az vm create --resource-group $ResourceGroupName --name $vm2 --nics "nic-2-$suffix" --image Ubuntu2204 --size Standard_B1s --admin-username azureadmin --ssh-key-values "$keyPath.pub" --output none

Expected state: The TCP probe and load-balancing rule share the intended ports, backend pool, and idle-timeout behavior.

Representative redacted output:

The TCP probe and load-balancing rule share the intended ports, backend pool, and idle-timeout behavior.

Positive validation:

$backends = az network lb address-pool show --resource-group $ResourceGroupName --lb-name $loadBalancer --name backend --query "backendIPConfigurations | length(@)" --output tsv
if ([int]$backends -ne 2) { throw "Expected two LB backends; found $backends." }

Expected positive result: The TCP probe and load-balancing rule share the intended ports, backend pool, and idle-timeout behavior.

Negative validation:

$publicIps = az vm list-ip-addresses --resource-group $ResourceGroupName --query "[?virtualMachine.name=='$vm1' || virtualMachine.name=='$vm2'].virtualMachine.network.publicIpAddresses[] | length(@)" --output tsv
if ($publicIps -and [int]$publicIps -ne 0) { throw 'A backend VM has a direct public IP.' }

Expected negative result: LAB21-CP03 negative boundary: The undesired state is absent; the negative assertion must not report: A backend VM has a direct public IP.

Evidence to retain:

  • LAB21-CP03 UTC result for configure health probing and balanced traffic.
  • Exact run-owned resource/object ID returned by the commands in LAB21-CP03.
  • Positive assertion proving: The TCP probe and load-balancing rule share the intended ports, backend pool, and idle-timeout behavior.
  • Negative assertion proving absence of: A backend VM has a direct public IP.

Common failure and safe retry: A probe port with no listening workload leaves every backend unhealthy even though the rule exists. Retry LAB21-CP03 for configure health probing and balanced traffic only after its exact recorded ID and current property are read. If the ID exists, reissue only the idempotent update; if it is absent, repeat only the owning create command, then rerun both assertions.

Cleanup dependency: After LAB21-CP04, LAB21-CP05, remove the exact manifest IDs created or configured by LAB21-CP03 for configure health probing and balanced traffic; its residual probe must then return zero active IDs.

Task 4 — Run topology-compatible Network Watcher diagnostics

Checkpoint: LAB21-CP04

Purpose and operational relevance: Exercise the deterministic operational or denied path needed to run topology-compatible network watcher diagnostics, then retain comparable before-and-after results. The change is unsafe when test-connectivity fails when the VM agent extension or required outbound connectivity was never provisioned. This checkpoint therefore proves: IP-flow, next-hop, or connection-troubleshoot evidence identifies the effective network decision without relying on an unavailable public path.

Run these commands from PowerShell. If you already used the automated lane, choose a new run ID before trying this guided lane.

$vm1Id = az vm show --resource-group $ResourceGroupName --name $vm1 --query id --output tsv
$vm2Id = az vm show --resource-group $ResourceGroupName --name $vm2 --query id --output tsv
foreach ($vmName in @($vm1, $vm2)) {
    az vm extension set --resource-group $ResourceGroupName --vm-name $vmName --publisher Microsoft.Azure.NetworkWatcher --name NetworkWatcherAgentLinux --enable-auto-upgrade true --output none
    $extensionState = az vm extension show --resource-group $ResourceGroupName --vm-name $vmName --name NetworkWatcherAgentLinux --query provisioningState --output tsv
    if ($extensionState -ne 'Succeeded') { throw "Network Watcher agent on $vmName is $extensionState." }
}
$watcher = az network watcher list --query "[?location=='$Location'] | [0]" --output json | ConvertFrom-Json
if (-not $watcher) { throw 'No Network Watcher exists in the selected region.' }
$external.networkWatcherId = [string]$watcher.id
$external.networkWatcherName = [string]$watcher.name
$external.networkWatcherResourceGroup = [string]$watcher.resourceGroup
$external.connectionMonitorName = "cm-$suffix"
$external.connectionMonitorId = az network watcher connection-monitor create --location $Location --name $external.connectionMonitorName --endpoint-source-name source-vm --endpoint-source-resource-id $vm1Id --endpoint-source-type AzureVM --endpoint-dest-name destination-vm --endpoint-dest-resource-id $vm2Id --endpoint-dest-type AzureVM --test-config-name ssh-tcp --protocol Tcp --tcp-port 22 --frequency 60 --test-group-name backend-ssh --tags purpose=az104-lab labId=21 runId=$RunId --query id --output tsv
$vm2Address = az vm list-ip-addresses --resource-group $ResourceGroupName --name $vm2 --query "[0].virtualMachine.network.privateIpAddresses[0]" --output tsv
az network watcher test-connectivity --resource-group $ResourceGroupName --source-resource $vm1 --dest-address $vm2Address --dest-port 22 --protocol Tcp --output json

Expected state: IP-flow, next-hop, or connection-troubleshoot evidence identifies the effective network decision without relying on an unavailable public path.

Representative redacted output:

IP-flow, next-hop, or connection-troubleshoot evidence identifies the effective network decision without relying on an unavailable public path.

Positive validation:

$monitorState = az network watcher connection-monitor show --location $Location --name $state.inputs['external-connection-monitor-name'] --query provisioningState --output tsv
if ($monitorState -ne 'Succeeded') { throw "Connection Monitor provisioning state is $monitorState." }

Expected positive result: IP-flow, next-hop, or connection-troubleshoot evidence identifies the effective network decision without relying on an unavailable public path.

Negative validation:

$agentCount = 0
foreach ($vmName in @($vm1, $vm2)) {
    $stateValue = az vm extension show --resource-group $ResourceGroupName --vm-name $vmName --name NetworkWatcherAgentLinux --query provisioningState --output tsv
    if ($stateValue -eq 'Succeeded') { $agentCount++ }
}
if ($agentCount -ne 2) { throw "Expected two healthy Network Watcher agents; found $agentCount." }

Expected negative result: LAB21-CP04 negative boundary: The undesired state is absent; the negative assertion must not report: Expected two healthy Network Watcher agents; found reported agentCount.

Evidence to retain:

  • LAB21-CP04 UTC result for run topology-compatible network watcher diagnostics.
  • Exact run-owned resource/object ID returned by the commands in LAB21-CP04.
  • Positive assertion proving: IP-flow, next-hop, or connection-troubleshoot evidence identifies the effective network decision without relying on an unavailable public path.
  • Negative assertion proving absence of: Expected two healthy Network Watcher agents; found reported agentCount.

Common failure and safe retry: test-connectivity fails when the VM agent extension or required outbound connectivity was never provisioned. Retry LAB21-CP04 for run topology-compatible network watcher diagnostics only after its exact recorded ID and current property are read. If the ID exists, reissue only the idempotent update; if it is absent, repeat only the owning create command, then rerun both assertions.

Cleanup dependency: After LAB21-CP05, remove the exact manifest IDs created or configured by LAB21-CP04 for run topology-compatible network watcher diagnostics; its residual probe must then return zero active IDs.

Task 5 — Reconcile frontend, backend health, and diagnostic evidence

Checkpoint: LAB21-CP05

Purpose and operational relevance: Reconcile live service properties, returned IDs, and dependency-aware removal needed to reconcile frontend, backend health, and diagnostic evidence before handoff. The change is unsafe when a provisioned load balancer can still have zero healthy backends and serve no traffic. This checkpoint therefore proves: Frontend configuration, backend membership, probe/rule bindings, and diagnostic results all refer to the same run-owned traffic path.

Run these commands from PowerShell. If you already used the automated lane, choose a new run ID before trying this guided lane.

az network lb show --resource-group $ResourceGroupName --name $loadBalancer --query "{sku:sku.name,frontend:frontendIPConfigurations[0].name,backendPool:backendAddressPools[0].name,probe:probes[0].port,rule:loadBalancingRules[0].frontendPort}" --output json
$sourceIp = az vm list-ip-addresses --resource-group $ResourceGroupName --name $vm1 --query "[0].virtualMachine.network.privateIpAddresses[0]" --output tsv
$destinationIp = az vm list-ip-addresses --resource-group $ResourceGroupName --name $vm2 --query "[0].virtualMachine.network.privateIpAddresses[0]" --output tsv
az network watcher show-next-hop --resource-group $ResourceGroupName --vm $vm1 --source-ip $sourceIp --dest-ip $destinationIp --output json
az network watcher connection-monitor show --location $Location --name $external.connectionMonitorName --query "{state:provisioningState,source:endpoints[0].resourceId,destination:endpoints[1].resourceId}" --output json

Expected state: Frontend configuration, backend membership, probe/rule bindings, and diagnostic results all refer to the same run-owned traffic path.

Representative redacted output:

Frontend configuration, backend membership, probe/rule bindings, and diagnostic results all refer to the same run-owned traffic path.

Positive validation:

$sku = az network lb show --resource-group $ResourceGroupName --name $loadBalancer --query sku.name --output tsv
if ($sku -ne 'Standard') { throw "Load Balancer SKU is $sku." }

Expected positive result: Frontend configuration, backend membership, probe/rule bindings, and diagnostic results all refer to the same run-owned traffic path.

Negative validation:

$faultRule = az network nsg rule list --resource-group $ResourceGroupName --nsg-name $nsg --query "[?name=='DenyInjectedSsh'] | length(@)" --output tsv
if ([int]$faultRule -ne 0) { throw 'The injected deny rule still exists.' }

Expected negative result: LAB21-CP05 negative boundary: The undesired state is absent; the negative assertion must not report: The injected deny rule still exists.

Evidence to retain:

  • LAB21-CP05 UTC result for reconcile frontend, backend health, and diagnostic evidence.
  • Exact run-owned resource/object ID returned by the commands in LAB21-CP05.
  • Positive assertion proving: Frontend configuration, backend membership, probe/rule bindings, and diagnostic results all refer to the same run-owned traffic path.
  • Negative assertion proving absence of: The injected deny rule still exists.

Common failure and safe retry: A provisioned load balancer can still have zero healthy backends and serve no traffic. Retry LAB21-CP05 for reconcile frontend, backend health, and diagnostic evidence only after its exact recorded ID and current property are read. If the ID exists, reissue only the idempotent update; if it is absent, repeat only the owning create command, then rerun both assertions.

Cleanup dependency: After all service validation and evidence capture, remove the exact manifest IDs created or configured by LAB21-CP05 for reconcile frontend, backend health, and diagnostic evidence; its residual probe must then return zero active IDs.

Final validation and result interpretation

Run independent deployment validation after all required checkpoints:

./scripts/cli/Validate.ps1 -RunId 'az104l21-01' -Mode Deployment
az resource list --resource-group 'rg-az104-l21-az104l21-01' --query '[].{name:name,type:type}' --output table
  • pass: every required positive and negative check passed.
  • partial: every required check passed, but at least one optional gate was deliberately skipped.
  • fail: a required checkpoint failed; do not record the lab as complete.

Keep the redacted validation.json; do not retain credentials, tokens, access keys, certificate material, email addresses, tenant IDs, or unredacted command output.

Deterministic break/fix exercise

Injection: Add a higher-priority NSG deny for the probe and backend port.

Inject the bounded fault:

az network nsg rule create --resource-group $ResourceGroupName --nsg-name $nsg --name DenyInjectedSsh --priority 100 --direction Inbound --access Deny --protocol Tcp --source-address-prefixes "*" --destination-port-ranges 22 --output none

Capture the failed state with a read-only query:

az network nsg rule list --resource-group $ResourceGroupName --nsg-name $nsg --query "[].{name:name,priority:priority,access:access,port:destinationPortRange}" --output table
az network lb probe show --resource-group $ResourceGroupName --lb-name $loadBalancer --name ssh --output json

Repair only the injected setting, then repeat the same query:

az network nsg rule delete --resource-group $ResourceGroupName --nsg-name $nsg --name DenyInjectedSsh
az network nsg rule list --resource-group $ResourceGroupName --nsg-name $nsg --query "[?destinationPortRange=='22'].{name:name,priority:priority,access:access}" --output table

Expected symptom: Backends become unhealthy and the frontend stops distributing traffic.

Diagnose with a read-only service query:

az network nsg rule list --resource-group $ResourceGroupName --nsg-name $nsg --query "[].{name:name,priority:priority,access:access,port:destinationPortRange}" --output table
az network lb probe show --resource-group $ResourceGroupName --lb-name $loadBalancer --name ssh --output json

Diagnosis: Inspect probe status, NSG effective rules, and connection troubleshoot output.

Repair: Remove the injected rule and wait for healthy probes.

Before/after evidence must show the failed negative or positive check before repair and the same check passing afterward. Do not inject a second fault until the first is removed.

Optional job-style challenge

Add an operational check that distinguishes DNS, frontend, probe, NSG, and backend failures.

Deliver a short change record containing assumptions, exact commands, redacted evidence, cost and risk notes, rollback, and residual results. The challenge is optional and never changes required checkpoint status.

Troubleshooting

Evidence-led troubleshooting from observation to revalidation
Use read-only evidence to identify the cause, make a targeted repair, and rerun the relevant validation. Open at full size.
Symptom Likely cause Safe next step
Context mismatch Active CLI context differs from run.json Stop, inspect az account show, and select the intended disposable context yourself.
Provider or feature unavailable Registration, region, feature, or SKU gate is unmet Read the preflight result; do not register or enable tenant features implicitly.
Name already exists The run ID is reused or a globally unique name collided Keep existing state intact and choose a new run ID.
Expected state is delayed The service is converging asynchronously Repeat only the read-only query with bounded retries; do not duplicate creation.
Permission denied Role, Graph scope, or data-plane authorization is insufficient Confirm the declared least-privilege boundary; do not broaden access automatically.
Cleanup refuses ownership ID, context, or tags differ from the manifest Investigate the mismatch; never bypass ownership verification.

Cleanup and residual verification

Run-owned cleanup boundary and residual-resource audit
Only verified run-owned objects are eligible for cleanup. Report deliberately retained or soft-deleted items; do not assume they were purged. Open at full size.

Preview exact targets first, execute only after ownership review, then validate post-cleanup:

./scripts/cli/Cleanup.ps1 -RunId 'az104l21-01'
./scripts/cli/Cleanup.ps1 -RunId 'az104l21-01' -Execute
./scripts/cli/Validate.ps1 -RunId 'az104l21-01' -Mode PostCleanup
$faultRule = az network nsg rule list --resource-group $ResourceGroupName --nsg-name $nsg --query "[?name=='DenyInjectedSsh'] | length(@)" --output tsv
if ([int]$faultRule -ne 0) { throw 'The injected deny rule still exists.' }

cleanup.json passes only when no active manifest-managed object remains. Soft-deleted or intentionally retained items must be listed with their reason and expected disposition. The lifecycle never performs irreversible purge automatically.

Exam debrief and assessment

Explain why the expected state, negative check, and cleanup boundary matter—not only which command was used. Map any missed concept back to its task anchor before reviewing the answer key.

Complete QUESTIONS.md, then use ANSWERS.md for option-by-option remediation. Scores of 85–100% indicate mastery, 70–84% indicate targeted review, and below 70% means repeat the mapped tasks.

Microsoft Learn sources

Lifecycle script appendix

The guided task blocks above and these executable scripts are generated from the same checkpoint data. The scripts are an optional automated lane; use a different run ID if you already completed the guided lane.

Preflight.ps1

# BEGIN GENERATED AZ104 V2
#requires -Version 7.4
[CmdletBinding()]
[Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSAvoidUsingWriteHost', '', Justification = 'Interactive lab progress is intentionally written to the host.')]
[Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSReviewUnusedParameter', '', Justification = 'All lifecycle scripts expose a stable cross-lab interface.')]
[Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSUseDeclaredVarsMoreThanAssignments', '', Justification = 'Generated task variables are intentionally shared across checkpoint blocks.')]
param(
    [string]$SubscriptionId = $env:AZ104_SUBSCRIPTION_ID,
    [Parameter(Mandatory)][ValidatePattern('^[a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?$')][string]$RunId,
    [Parameter(Mandatory)][ValidatePattern('^[a-z0-9]+$')][string]$Location,
    [string]$SecondaryLocation = $env:AZ104_SECONDARY_LOCATION
)

Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
$PSNativeCommandUseErrorActionPreference = $true

if (-not (Get-Command az -ErrorAction SilentlyContinue)) {
    throw 'Azure CLI is required. Run the repository readiness initializer, then retry.'
}

$account = az account show --output json | ConvertFrom-Json
if (-not $account) { throw 'No active Azure CLI context. Run az login deliberately before this lab.' }
if ($SubscriptionId -and [string]$account.id -ne $SubscriptionId) {
    throw "Context mismatch: active subscription is $($account.id), expected $SubscriptionId. Preflight will not switch it."
}
if (-not $SubscriptionId) { $SubscriptionId = [string]$account.id }
# The account read above is the mandatory context gate. All remaining Azure
# probes are collected as results instead of allowing one unavailable SKU or
# provider query to abort the readiness report before it can explain the gap.
$PSNativeCommandUseErrorActionPreference = $false
$suffix = (($RunId -replace '[^a-z0-9]', '') + '000000000000').Substring(0, 12)
$ResourceGroupName = $(if ('21' -in @('00', '01', '02')) { $null } else { "rg-az104-l21-$RunId" })

$checks = [System.Collections.Generic.List[object]]::new()
function Add-PreflightResult {
    param([string]$Id, [bool]$Required, [bool]$Passed, [string]$Actual)
    $checks.Add([pscustomobject]@{ id = $Id; required = $Required; passed = $Passed; actual = $Actual })
}

Add-PreflightResult -Id 'context' -Required $true -Passed $true -Actual "cloud=$($account.environmentName); tenant=<redacted>; subscription=<redacted>"
$azVersion = (az version --query '"azure-cli"' --output tsv)
$bicepVersion = (az bicep version 2>&1 | Out-String).Trim()
Add-PreflightResult -Id 'azure-cli' -Required $true -Passed ([bool]$azVersion) -Actual $azVersion
Add-PreflightResult -Id 'powershell' -Required $true -Passed ($PSVersionTable.PSVersion -ge [version]'7.4') -Actual $PSVersionTable.PSVersion.ToString()
Add-PreflightResult -Id 'bicep' -Required $true -Passed ([bool]$bicepVersion) -Actual $bicepVersion

$requiredEnvironmentVariables = @()
foreach ($variableName in $requiredEnvironmentVariables) {
    $present = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($variableName))
    Add-PreflightResult -Id "input:$variableName" -Required $true -Passed $present -Actual $(if ($present) { 'present (value redacted)' } else { 'missing' })
}

$providers = @(
    'Microsoft.Compute'
    'Microsoft.Network'
)
foreach ($provider in $providers) {
    $registrationState = az provider show --namespace $provider --query registrationState --output tsv 2>$null
    Add-PreflightResult -Id "provider:$provider" -Required $true -Passed ($registrationState -eq 'Registered') -Actual $(if ($registrationState) { $registrationState } else { 'Unavailable' })
}

$knownLocation = az account list-locations --query "[?name=='$Location'].name | [0]" --output tsv
Add-PreflightResult -Id 'region' -Required $true -Passed ($knownLocation -eq $Location) -Actual $(if ($knownLocation) { $knownLocation } else { 'not available' })

$probePassed = $true
$probeActual = 'assertion passed (output not persisted)'
try {
    $LASTEXITCODE = 0
    & {
$network = az provider show --namespace Microsoft.Network --query registrationState --output tsv
if ($network -ne 'Registered') { throw 'Microsoft.Network is not registered.' }
$watchers = az network watcher list --query "[?location=='$Location'] | length(@)" --output tsv
if ([int]$watchers -lt 1) { throw 'Network Watcher is not available in the selected region.' }
    } | Out-Null
    if ($LASTEXITCODE -ne 0) { throw "native exit code $LASTEXITCODE" }
} catch {
    $probePassed = $false
    $probeActual = "assertion failed: $($_.Exception.GetType().Name)"
}
Add-PreflightResult -Id 'preflight.authored.service' -Required $true -Passed $probePassed -Actual $probeActual

$checks | Format-Table -AutoSize
$failedRequired = @($checks | Where-Object { $_.required -and -not $_.passed })
if ($failedRequired.Count -gt 0) {
    throw "Preflight blocked: $($failedRequired.id -join ', ')"
}
Write-Host 'Preflight passed. It performed no sign-in, context switch, provider registration, or Azure mutation.'
# END GENERATED AZ104 V2

Setup.ps1

# BEGIN GENERATED AZ104 V2
#requires -Version 7.4
[CmdletBinding()]
[Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSAvoidUsingWriteHost', '', Justification = 'Interactive lab progress is intentionally written to the host.')]
[Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSReviewUnusedParameter', '', Justification = 'All lifecycle scripts expose a stable cross-lab interface.')]
[Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSUseDeclaredVarsMoreThanAssignments', '', Justification = 'Generated task variables are intentionally shared across checkpoint blocks.')]
param(
    [string]$SubscriptionId = $env:AZ104_SUBSCRIPTION_ID,
    [Parameter(Mandatory)][ValidatePattern('^[a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?$')][string]$RunId,
    [Parameter(Mandatory)][ValidatePattern('^[a-z0-9]+$')][string]$Location,
    [string]$SecondaryLocation = $env:AZ104_SECONDARY_LOCATION,
    [switch]$AcknowledgeCost,
    [switch]$AcknowledgeTenantChange,
    [switch]$Execute
)

Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
$PSNativeCommandUseErrorActionPreference = $true

if (-not (Get-Command az -ErrorAction SilentlyContinue)) {
    throw 'Azure CLI is required. Run the repository readiness initializer, then retry.'
}

$LabRoot = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path
$StateDir = Join-Path $LabRoot ".state/$RunId"
$Manifest = Join-Path $StateDir 'run.json'
$ResourceGroupName = $(if ('21' -in @('00', '01', '02')) { $null } else { "rg-az104-l21-$RunId" })
$suffix = (($RunId -replace '[^a-z0-9]', '') + '000000000000').Substring(0, 12)

Write-Host 'LAB-21 execution plan'
Write-Host "  subscription: $SubscriptionId"
Write-Host "  location: $Location"
Write-Host '  command surface: Azure CLI hosted in PowerShell'
Write-Host '  state: run.json, validation.json, cleanup.json'
if (-not $Execute) {
    Write-Host 'Preview only. Review context, inputs, cost, tenant scope, and cleanup before using -Execute.'
    return
}
if ($true -and -not $AcknowledgeCost) { throw 'This lab requires -AcknowledgeCost before execution.' }
if ($false -and -not $AcknowledgeTenantChange) { throw 'This lab requires -AcknowledgeTenantChange before execution.' }

& (Join-Path $PSScriptRoot 'Preflight.ps1') -SubscriptionId $SubscriptionId -RunId $RunId -Location $Location -SecondaryLocation $SecondaryLocation
if (Test-Path -LiteralPath $Manifest) { throw "State already exists at $Manifest. Choose a new run ID." }
New-Item -ItemType Directory -Path $StateDir -Force | Out-Null
$account = az account show --output json | ConvertFrom-Json
if (-not $SubscriptionId) { $SubscriptionId = [string]$account.id }
$now = (Get-Date).ToUniversalTime().ToString('o')
$state = @{
    schemaVersion = '1.0.0'
    labId = 'LAB-21'
    runId = $RunId
    createdAt = $now
    updatedAt = $now
    status = 'initialized'
    context = @{ cloud = [string]$account.environmentName; tenantId = [string]$account.tenantId; subscriptionId = [string]$SubscriptionId }
    acknowledgements = @{ cost = [bool]$AcknowledgeCost; tenantChange = [bool]$AcknowledgeTenantChange }
    inputs = @{ 'location' = $Location; 'secondary-location' = $SecondaryLocation }
    checkpointStates = @(
        @{ checkpointId = 'LAB21-CP01'; required = $true; status = 'pending'; updatedAt = $now; message = 'Not started.' }
        @{ checkpointId = 'LAB21-CP02'; required = $true; status = 'pending'; updatedAt = $now; message = 'Not started.' }
        @{ checkpointId = 'LAB21-CP03'; required = $true; status = 'pending'; updatedAt = $now; message = 'Not started.' }
        @{ checkpointId = 'LAB21-CP04'; required = $true; status = 'pending'; updatedAt = $now; message = 'Not started.' }
        @{ checkpointId = 'LAB21-CP05'; required = $true; status = 'pending'; updatedAt = $now; message = 'Not started.' }
    )
    managedObjects = @()
    originalSettings = @()
}
$external = @{}

function Save-RunState {
    $state.updatedAt = (Get-Date).ToUniversalTime().ToString('o')
    $state | ConvertTo-Json -Depth 30 | Set-Content -LiteralPath $Manifest -Encoding utf8
}

function Save-ExternalState {
    foreach ($key in @($external.Keys)) {
        $value = $external[$key]
        if ($null -eq $value -or $value -is [string] -or $value -is [ValueType]) {
            $inputKey = 'external-' + (([string]$key -creplace '([a-z0-9])([A-Z])', '$1-$2') -replace '[^a-zA-Z0-9-]', '-').ToLowerInvariant()
            $state.inputs[$inputKey] = $value
        }
    }
    Save-RunState
}

function Write-CheckpointState {
    param([string]$CheckpointId, [string]$Status, [string]$Message)
    $entry = @($state.checkpointStates | Where-Object { $_.checkpointId -eq $CheckpointId })[0]
    $entry.status = $Status
    $entry.updatedAt = (Get-Date).ToUniversalTime().ToString('o')
    $entry.message = $Message
}

function Add-ManagedObject {
    param([string]$CheckpointId, [string]$Kind, [string]$Id, [string]$Name, [string]$Type, [string]$Scope, [string]$OwnershipMethod)
    if ([string]::IsNullOrWhiteSpace($Id)) { return }
    $existing = @($state.managedObjects | Where-Object { $_.id -eq $Id })
    if ($existing.Count -gt 0) { return }
    $expectedTags = @{}
    if ($OwnershipMethod -eq 'manifest-id-and-tags') {
        $expectedTags = @{ purpose = 'az104-lab'; labId = '21'; runId = $RunId }
    }
    $state.managedObjects += @{
        checkpointId = $CheckpointId
        kind = $Kind
        id = $Id
        name = $Name
        type = $Type
        scope = $Scope
        ownership = @{ method = $OwnershipMethod; expectedTags = $expectedTags }
        recordedAt = (Get-Date).ToUniversalTime().ToString('o')
        lifecycleStatus = 'active'
    }
    Save-RunState
}

function Sync-ManagedResource {
    param([string]$CheckpointId)
    # The entire recovery inventory is best effort so a state-helper or Azure
    # query failure cannot mask the original mutation error. Every object that
    # can be recovered is still persisted immediately as it is discovered.
    $nativePreference = $PSNativeCommandUseErrorActionPreference
    $PSNativeCommandUseErrorActionPreference = $false
    try {
        Save-ExternalState
        $resourceGroupNames = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase)
        if ($ResourceGroupName) { $null = $resourceGroupNames.Add([string]$ResourceGroupName) }
        foreach ($key in @($external.Keys | Where-Object { [string]$_ -match '(?i)ResourceGroupName$' })) {
            if ($external[$key]) { $null = $resourceGroupNames.Add([string]$external[$key]) }
        }
        foreach ($groupName in $resourceGroupNames) {
            $groupJson = az group show --subscription $SubscriptionId --name $groupName --output json 2>$null
            $group = $(if ($LASTEXITCODE -eq 0 -and $groupJson) { $groupJson | ConvertFrom-Json } else { $null })
            if ($group) {
                $groupCheckpoint = $(if ([string]$group.name -eq [string]$ResourceGroupName) { 'LAB21-CP01' } else { $CheckpointId })
                Add-ManagedObject -CheckpointId $groupCheckpoint -Kind 'azure-resource' -Id ([string]$group.id) -Name ([string]$group.name) -Type 'Microsoft.Resources/resourceGroups' -Scope "/subscriptions/$SubscriptionId" -OwnershipMethod 'manifest-id-and-tags'
                $resourcesJson = az resource list --subscription $SubscriptionId --resource-group $groupName --output json 2>$null
                $resources = @($(if ($LASTEXITCODE -eq 0 -and $resourcesJson) { $resourcesJson | ConvertFrom-Json } else { @() }))
                foreach ($resource in $resources) {
                    Add-ManagedObject -CheckpointId $CheckpointId -Kind 'azure-resource' -Id ([string]$resource.id) -Name ([string]$resource.name) -Type ([string]$resource.type) -Scope ([string]$group.id) -OwnershipMethod 'manifest-id-and-tags'
                }
            }
        }
        if ($external.ContainsKey('groupId') -and $external.groupId) { Add-ManagedObject -CheckpointId $CheckpointId -Kind 'entra-object' -Id ([string]$external.groupId) -Name 'lab-group' -Type 'Microsoft.Graph/group' -Scope $state.context.tenantId -OwnershipMethod 'manifest-id' }
        if ($external.ContainsKey('guestUserId') -and $external.guestUserId) { Add-ManagedObject -CheckpointId $CheckpointId -Kind 'entra-object' -Id ([string]$external.guestUserId) -Name 'guest-user' -Type 'Microsoft.Graph/user' -Scope $state.context.tenantId -OwnershipMethod 'manifest-id' }
        if ($external.ContainsKey('userIds')) {
            foreach ($userId in @($external.userIds)) { Add-ManagedObject -CheckpointId $CheckpointId -Kind 'entra-object' -Id ([string]$userId) -Name 'lab-user' -Type 'Microsoft.Graph/user' -Scope $state.context.tenantId -OwnershipMethod 'manifest-id' }
        }
        if ($external.ContainsKey('delegationRecordId') -and $external.delegationRecordId) {
            Add-ManagedObject -CheckpointId $CheckpointId -Kind 'azure-resource' -Id ([string]$external.delegationRecordId) -Name ([string]$external.delegationRecordName) -Type 'Microsoft.Network/dnsZones/NS' -Scope ([string]$external.parentZoneId) -OwnershipMethod 'manifest-id'
        }
        if ($external.ContainsKey('connectionMonitorId') -and $external.connectionMonitorId) {
            Add-ManagedObject -CheckpointId $CheckpointId -Kind 'azure-resource' -Id ([string]$external.connectionMonitorId) -Name ([string]$external.connectionMonitorName) -Type 'Microsoft.Network/networkWatchers/connectionMonitors' -Scope ([string]$external.networkWatcherId) -OwnershipMethod 'manifest-id'
        }
    } catch {
        Write-Warning "Recovery inventory for $CheckpointId was incomplete: $($_.Exception.Message)"
    } finally {
        $PSNativeCommandUseErrorActionPreference = $nativePreference
    }
}

# The manifest exists before the first Azure mutation.
Save-RunState
$state.status = 'setup-in-progress'
Save-RunState

$expiresOn = (Get-Date).ToUniversalTime().AddDays(1).ToString('yyyy-MM-dd')
try {
    az group create --subscription $SubscriptionId --name $ResourceGroupName --location $Location --tags purpose=az104-lab labId=21 runId=$RunId expiresOn=$expiresOn --output none
} catch {
    Write-CheckpointState -CheckpointId 'LAB21-CP01' -Status 'fail' -Message "Resource-group boundary creation failed: $($_.Exception.GetType().Name)"
    $state.status = 'failed'
    Save-RunState
    throw
} finally {
    Sync-ManagedResource -CheckpointId 'LAB21-CP01'
}

# CHECKPOINT LAB21-CP01 BEGIN
try {
    Write-CheckpointState -CheckpointId 'LAB21-CP01' -Status 'in-progress' -Message 'Checkpoint execution started.'
    $activeCheckpointId = 'LAB21-CP01'
    Save-RunState

    az provider show --namespace Microsoft.Network --query registrationState --output tsv
    az provider show --namespace Microsoft.Compute --query registrationState --output tsv
    az network watcher list --query "[?location=='$Location'].{name:name,resourceGroup:resourceGroup}" --output table

    Sync-ManagedResource -CheckpointId 'LAB21-CP01'
    Write-CheckpointState -CheckpointId 'LAB21-CP01' -Status 'pass' -Message 'The Standard load-balancer SKU and Network Watcher diagnostic commands are available in the selected region.'
    Save-RunState
} catch {
    Sync-ManagedResource -CheckpointId 'LAB21-CP01'
    Write-CheckpointState -CheckpointId 'LAB21-CP01' -Status 'fail' -Message "Checkpoint failed: $($_.Exception.GetType().Name)"
    $state.status = 'failed'
    Save-RunState
    throw
}
# CHECKPOINT LAB21-CP01 END

# CHECKPOINT LAB21-CP02 BEGIN
try {
    Write-CheckpointState -CheckpointId 'LAB21-CP02' -Status 'in-progress' -Message 'Checkpoint execution started.'
    $activeCheckpointId = 'LAB21-CP02'
    Save-RunState

    $vnet = "vnet-$suffix"; $publicIp = "pip-$suffix"; $loadBalancer = "lb-$suffix"; $nsg = "nsg-$suffix"; $vm1 = "vm-1-$suffix"; $vm2 = "vm-2-$suffix"
    $keyPath = Join-Path $StateDir 'id_ed25519'
    ssh-keygen -t ed25519 -N '' -f $keyPath | Out-Null
    try {
        az network vnet create --resource-group $ResourceGroupName --name $vnet --address-prefixes 10.21.0.0/16 --subnet-name backend --subnet-prefixes 10.21.1.0/24 --output none
    } finally {
        Sync-ManagedResource -CheckpointId 'LAB21-CP02'
    }
    try {
        az network public-ip create --resource-group $ResourceGroupName --name $publicIp --location $Location --sku Standard --allocation-method Static --output none
    } finally {
        Sync-ManagedResource -CheckpointId 'LAB21-CP02'
    }
    try {
        az network lb create --resource-group $ResourceGroupName --name $loadBalancer --sku Standard --public-ip-address $publicIp --frontend-ip-name frontend --backend-pool-name backend --output none
    } finally {
        Sync-ManagedResource -CheckpointId 'LAB21-CP02'
    }
    try {
        az network lb probe create --resource-group $ResourceGroupName --lb-name $loadBalancer --name ssh --protocol Tcp --port 22 --interval 15 --threshold 2 --output none
    } finally {
        Sync-ManagedResource -CheckpointId 'LAB21-CP02'
    }
    try {
        az network lb rule create --resource-group $ResourceGroupName --lb-name $loadBalancer --name ssh --protocol Tcp --frontend-port 2222 --backend-port 22 --frontend-ip-name frontend --backend-pool-name backend --probe-name ssh --idle-timeout 4 --disable-outbound-snat true --output none
    } finally {
        Sync-ManagedResource -CheckpointId 'LAB21-CP02'
    }
    try {
        az network lb outbound-rule create --resource-group $ResourceGroupName --lb-name $loadBalancer --name outbound --frontend-ip-configs frontend --address-pool backend --protocol All --outbound-ports 1024 --idle-timeout 4 --output none
    } finally {
        Sync-ManagedResource -CheckpointId 'LAB21-CP02'
    }

    Sync-ManagedResource -CheckpointId 'LAB21-CP02'
    Write-CheckpointState -CheckpointId 'LAB21-CP02' -Status 'pass' -Message 'Both backend NICs join the pool and the frontend, health probe, and rule reference one consistent load-balancer resource.'
    Save-RunState
} catch {
    Sync-ManagedResource -CheckpointId 'LAB21-CP02'
    Write-CheckpointState -CheckpointId 'LAB21-CP02' -Status 'fail' -Message "Checkpoint failed: $($_.Exception.GetType().Name)"
    $state.status = 'failed'
    Save-RunState
    throw
}
# CHECKPOINT LAB21-CP02 END

# CHECKPOINT LAB21-CP03 BEGIN
try {
    Write-CheckpointState -CheckpointId 'LAB21-CP03' -Status 'in-progress' -Message 'Checkpoint execution started.'
    $activeCheckpointId = 'LAB21-CP03'
    Save-RunState

    try {
        az network nsg create --resource-group $ResourceGroupName --name $nsg --output none
    } finally {
        Sync-ManagedResource -CheckpointId 'LAB21-CP03'
    }
    try {
        az network nsg rule create --resource-group $ResourceGroupName --nsg-name $nsg --name AllowLoadBalancerProbe --priority 200 --direction Inbound --access Allow --protocol Tcp --source-address-prefixes AzureLoadBalancer --destination-port-ranges 22 --output none
    } finally {
        Sync-ManagedResource -CheckpointId 'LAB21-CP03'
    }
    try {
        az network nsg rule create --resource-group $ResourceGroupName --nsg-name $nsg --name AllowFrontendSsh --priority 210 --direction Inbound --access Allow --protocol Tcp --source-address-prefixes Internet --destination-port-ranges 22 --output none
    } finally {
        Sync-ManagedResource -CheckpointId 'LAB21-CP03'
    }
    try {
        az network nic create --resource-group $ResourceGroupName --name "nic-1-$suffix" --vnet-name $vnet --subnet backend --network-security-group $nsg --lb-name $loadBalancer --lb-address-pools backend --output none
    } finally {
        Sync-ManagedResource -CheckpointId 'LAB21-CP03'
    }
    try {
        az network nic create --resource-group $ResourceGroupName --name "nic-2-$suffix" --vnet-name $vnet --subnet backend --network-security-group $nsg --lb-name $loadBalancer --lb-address-pools backend --output none
    } finally {
        Sync-ManagedResource -CheckpointId 'LAB21-CP03'
    }
    try {
        az vm create --resource-group $ResourceGroupName --name $vm1 --nics "nic-1-$suffix" --image Ubuntu2204 --size Standard_B1s --admin-username azureadmin --ssh-key-values "$keyPath.pub" --output none
    } finally {
        Sync-ManagedResource -CheckpointId 'LAB21-CP03'
    }
    try {
        az vm create --resource-group $ResourceGroupName --name $vm2 --nics "nic-2-$suffix" --image Ubuntu2204 --size Standard_B1s --admin-username azureadmin --ssh-key-values "$keyPath.pub" --output none
    } finally {
        Sync-ManagedResource -CheckpointId 'LAB21-CP03'
    }

    Sync-ManagedResource -CheckpointId 'LAB21-CP03'
    Write-CheckpointState -CheckpointId 'LAB21-CP03' -Status 'pass' -Message 'The TCP probe and load-balancing rule share the intended ports, backend pool, and idle-timeout behavior.'
    Save-RunState
} catch {
    Sync-ManagedResource -CheckpointId 'LAB21-CP03'
    Write-CheckpointState -CheckpointId 'LAB21-CP03' -Status 'fail' -Message "Checkpoint failed: $($_.Exception.GetType().Name)"
    $state.status = 'failed'
    Save-RunState
    throw
}
# CHECKPOINT LAB21-CP03 END

# CHECKPOINT LAB21-CP04 BEGIN
try {
    Write-CheckpointState -CheckpointId 'LAB21-CP04' -Status 'in-progress' -Message 'Checkpoint execution started.'
    $activeCheckpointId = 'LAB21-CP04'
    Save-RunState

    $vm1Id = az vm show --resource-group $ResourceGroupName --name $vm1 --query id --output tsv
    $vm2Id = az vm show --resource-group $ResourceGroupName --name $vm2 --query id --output tsv
    foreach ($vmName in @($vm1, $vm2)) {
        az vm extension set --resource-group $ResourceGroupName --vm-name $vmName --publisher Microsoft.Azure.NetworkWatcher --name NetworkWatcherAgentLinux --enable-auto-upgrade true --output none
        $extensionState = az vm extension show --resource-group $ResourceGroupName --vm-name $vmName --name NetworkWatcherAgentLinux --query provisioningState --output tsv
        if ($extensionState -ne 'Succeeded') { throw "Network Watcher agent on $vmName is $extensionState." }
    }
    $watcher = az network watcher list --query "[?location=='$Location'] | [0]" --output json | ConvertFrom-Json
    if (-not $watcher) { throw 'No Network Watcher exists in the selected region.' }
    $external.networkWatcherId = [string]$watcher.id
    $external.networkWatcherName = [string]$watcher.name
    $external.networkWatcherResourceGroup = [string]$watcher.resourceGroup
    $external.connectionMonitorName = "cm-$suffix"
    try {
        $external.connectionMonitorId = az network watcher connection-monitor create --location $Location --name $external.connectionMonitorName --endpoint-source-name source-vm --endpoint-source-resource-id $vm1Id --endpoint-source-type AzureVM --endpoint-dest-name destination-vm --endpoint-dest-resource-id $vm2Id --endpoint-dest-type AzureVM --test-config-name ssh-tcp --protocol Tcp --tcp-port 22 --frequency 60 --test-group-name backend-ssh --tags purpose=az104-lab labId=21 runId=$RunId --query id --output tsv
    } finally {
        Sync-ManagedResource -CheckpointId 'LAB21-CP04'
    }
    $vm2Address = az vm list-ip-addresses --resource-group $ResourceGroupName --name $vm2 --query "[0].virtualMachine.network.privateIpAddresses[0]" --output tsv
    az network watcher test-connectivity --resource-group $ResourceGroupName --source-resource $vm1 --dest-address $vm2Address --dest-port 22 --protocol Tcp --output json

    Sync-ManagedResource -CheckpointId 'LAB21-CP04'
    Write-CheckpointState -CheckpointId 'LAB21-CP04' -Status 'pass' -Message 'IP-flow, next-hop, or connection-troubleshoot evidence identifies the effective network decision without relying on an unavailable public path.'
    Save-RunState
} catch {
    Sync-ManagedResource -CheckpointId 'LAB21-CP04'
    Write-CheckpointState -CheckpointId 'LAB21-CP04' -Status 'fail' -Message "Checkpoint failed: $($_.Exception.GetType().Name)"
    $state.status = 'failed'
    Save-RunState
    throw
}
# CHECKPOINT LAB21-CP04 END

# CHECKPOINT LAB21-CP05 BEGIN
try {
    Write-CheckpointState -CheckpointId 'LAB21-CP05' -Status 'in-progress' -Message 'Checkpoint execution started.'
    $activeCheckpointId = 'LAB21-CP05'
    Save-RunState

    az network lb show --resource-group $ResourceGroupName --name $loadBalancer --query "{sku:sku.name,frontend:frontendIPConfigurations[0].name,backendPool:backendAddressPools[0].name,probe:probes[0].port,rule:loadBalancingRules[0].frontendPort}" --output json
    $sourceIp = az vm list-ip-addresses --resource-group $ResourceGroupName --name $vm1 --query "[0].virtualMachine.network.privateIpAddresses[0]" --output tsv
    $destinationIp = az vm list-ip-addresses --resource-group $ResourceGroupName --name $vm2 --query "[0].virtualMachine.network.privateIpAddresses[0]" --output tsv
    az network watcher show-next-hop --resource-group $ResourceGroupName --vm $vm1 --source-ip $sourceIp --dest-ip $destinationIp --output json
    az network watcher connection-monitor show --location $Location --name $external.connectionMonitorName --query "{state:provisioningState,source:endpoints[0].resourceId,destination:endpoints[1].resourceId}" --output json

    Sync-ManagedResource -CheckpointId 'LAB21-CP05'
    Write-CheckpointState -CheckpointId 'LAB21-CP05' -Status 'pass' -Message 'Frontend configuration, backend membership, probe/rule bindings, and diagnostic results all refer to the same run-owned traffic path.'
    Save-RunState
} catch {
    Sync-ManagedResource -CheckpointId 'LAB21-CP05'
    Write-CheckpointState -CheckpointId 'LAB21-CP05' -Status 'fail' -Message "Checkpoint failed: $($_.Exception.GetType().Name)"
    $state.status = 'failed'
    Save-RunState
    throw
}
# CHECKPOINT LAB21-CP05 END

$skippedRequired = @($state.checkpointStates | Where-Object { $_.required -and $_.status -eq 'skipped' })
if ($skippedRequired.Count -gt 0) {
    $state.status = 'failed'
    Save-RunState
    throw "Required checkpoints were skipped: $($skippedRequired.checkpointId -join ', ')"
}
$skippedOptional = @($state.checkpointStates | Where-Object { -not $_.required -and $_.status -eq 'skipped' })
$state.status = $(if ($skippedOptional.Count -gt 0) { 'partial' } else { 'setup-complete' })
Save-RunState
Write-Host "Setup result: $($state.status). State: $Manifest"
Write-Host "Next: ./Validate.ps1 -RunId $RunId -Mode Deployment"
# END GENERATED AZ104 V2

Validate.ps1

# BEGIN GENERATED AZ104 V2
#requires -Version 7.4
[CmdletBinding()]
[Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSAvoidUsingWriteHost', '', Justification = 'Interactive lab progress is intentionally written to the host.')]
[Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSReviewUnusedParameter', '', Justification = 'All lifecycle scripts expose a stable cross-lab interface.')]
[Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSUseDeclaredVarsMoreThanAssignments', '', Justification = 'Generated task variables are intentionally shared across checkpoint blocks.')]
param(
    [string]$SubscriptionId = $env:AZ104_SUBSCRIPTION_ID,
    [Parameter(Mandatory)][ValidatePattern('^[a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?$')][string]$RunId,
    [ValidateSet('Deployment', 'PostCleanup')][string]$Mode = 'Deployment'
)

Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
$PSNativeCommandUseErrorActionPreference = $true

if (-not (Get-Command az -ErrorAction SilentlyContinue)) {
    throw 'Azure CLI is required. Run the repository readiness initializer, then retry.'
}

$LabRoot = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path
$StateDir = Join-Path $LabRoot ".state/$RunId"
$Manifest = Join-Path $StateDir 'run.json'
$ValidationPath = Join-Path $StateDir 'validation.json'
if (-not (Test-Path -LiteralPath $Manifest)) { throw "Run manifest not found: $Manifest" }
$state = Get-Content -LiteralPath $Manifest -Raw | ConvertFrom-Json -AsHashtable
if ($state.labId -ne 'LAB-21' -or $state.runId -ne $RunId) { throw 'Manifest ownership does not match this lab and run ID.' }
$PSNativeCommandUseErrorActionPreference = $false
$accountJson = az account show --output json 2>$null
$account = $(if ($LASTEXITCODE -eq 0 -and $accountJson) { $accountJson | ConvertFrom-Json } else { $null })
if (-not $account -or [string]$account.tenantId -ne [string]$state.context.tenantId -or [string]$account.id -ne [string]$state.context.subscriptionId) {
    $capturedAt = (Get-Date).ToUniversalTime().ToString('o')
    $actualContext = $(if ($account) { "tenant=$($account.tenantId); subscription=$($account.id)" } else { 'active context unavailable' })
    $contextFailure = @{
        schemaVersion = '1.0.0'; labId = 'LAB-21'; runId = $RunId; mode = $Mode
        generatedAt = $capturedAt; result = 'fail'
        summary = @{ required = 1; passed = 0; failed = 1; skipped = 0 }
        checks = @(@{ id = 'context.active'; checkpointId = 'LAB21-CP01'; kind = 'context'; required = $true; status = 'fail'; message = 'Active Azure context does not match the run manifest.'; evidence = @{ command = 'az account show --output json'; expected = 'tenant and subscription exactly match run.json'; actual = $actualContext; capturedAt = $capturedAt; redacted = $true } })
    }
    $contextFailure | ConvertTo-Json -Depth 30 | Set-Content -LiteralPath $ValidationPath -Encoding utf8
    Write-Host "Validation $Mode result: fail"
    Write-Host "Artifact: $ValidationPath"
    exit 1
}
$SubscriptionId = [string]$state.context.subscriptionId
$Location = [string]$state.inputs['location']
$SecondaryLocation = [string]$state.inputs['secondary-location']
$ResourceGroupName = $(if ('21' -in @('00', '01', '02')) { $null } else { "rg-az104-l21-$RunId" })
$suffix = (($RunId -replace '[^a-z0-9]', '') + '000000000000').Substring(0, 12)
$vnet = "vnet-$suffix"; $publicIp = "pip-$suffix"; $loadBalancer = "lb-$suffix"; $nsg = "nsg-$suffix"; $vm1 = "vm-1-$suffix"; $vm2 = "vm-2-$suffix"

$checks = [System.Collections.Generic.List[object]]::new()
function Add-ValidationCheck {
    param([string]$Id, [string]$CheckpointId, [string]$Kind, [bool]$Required, [bool]$Passed, [string]$Command, [string]$Expected, [string]$Actual, [bool]$Skipped = $false)
    $capturedAt = (Get-Date).ToUniversalTime().ToString('o')
    $checks.Add(@{
        id = $Id
        checkpointId = $CheckpointId
        kind = $Kind
        required = $Required
        status = $(if ($Skipped) { 'skipped' } elseif ($Passed) { 'pass' } else { 'fail' })
        message = $(if ($Skipped) { 'Optional gate was deliberately skipped.' } elseif ($Passed) { 'Expected state observed.' } else { 'Expected state was not observed.' })
        evidence = @{ command = $Command; expected = $Expected; actual = $Actual; capturedAt = $capturedAt; redacted = $true }
    })
}

# CHECKPOINT LAB21-CP01 BEGIN
$checkpointState = @($state.checkpointStates | Where-Object { $_.checkpointId -eq 'LAB21-CP01' })[0]
$checkpointObjects = @($state.managedObjects | Where-Object { $_.checkpointId -eq 'LAB21-CP01' })

if ($Mode -eq 'Deployment') {
    if ($checkpointState.status -eq 'skipped') {
        Add-ValidationCheck -Id 'lab21-cp01.positive' -CheckpointId 'LAB21-CP01' -Kind 'positive' -Required ([bool]$checkpointState.required) -Passed $true -Skipped $true -Command 'optional gate evaluation' -Expected 'The optional checkpoint is explicitly skipped when its documented input is absent.' -Actual 'optional gate absent'
        Add-ValidationCheck -Id 'lab21-cp01.negative' -CheckpointId 'LAB21-CP01' -Kind 'negative' -Required ([bool]$checkpointState.required) -Passed $true -Skipped $true -Command 'optional gate evaluation' -Expected 'No mutation occurs for a skipped optional checkpoint.' -Actual 'optional gate absent'
    } else {
    $positivePassed = $checkpointState.status -eq 'pass'
    $positiveActual = "checkpoint=$($checkpointState.status); managedObjects=$($checkpointObjects.Count)"
    foreach ($managedObject in $checkpointObjects) {
        if ($managedObject.type -eq 'Microsoft.Management/managementGroups') {
            $null = az account management-group show --name $managedObject.name --output none 2>$null
            if ($LASTEXITCODE -ne 0) { $positivePassed = $false; $positiveActual += "; missing=$($managedObject.id)" }
        } elseif ($managedObject.kind -eq 'azure-resource') {
            $null = az resource show --ids $managedObject.id --output none 2>$null
            if ($LASTEXITCODE -ne 0) { $positivePassed = $false; $positiveActual += "; missing=$($managedObject.id)" }
        } elseif ($managedObject.kind -eq 'entra-object') {
            $null = az rest --method get --url "https://graph.microsoft.com/v1.0/directoryObjects/$($managedObject.id)" --output none 2>$null
            if ($LASTEXITCODE -ne 0) { $positivePassed = $false; $positiveActual += "; missing=$($managedObject.id)" }
        }
    }

    # AUTHORED SERVICE ASSERTION: positive
    try {
        $LASTEXITCODE = 0
        & {
$network = az provider show --namespace Microsoft.Network --query registrationState --output tsv
if ($network -ne 'Registered') { throw 'Microsoft.Network is not registered.' }
        } | Out-Null
        if ($LASTEXITCODE -ne 0) {
            $positivePassed = $false
            $positiveActual += "; serviceProbeExit=$LASTEXITCODE"
        } else {
            $positiveActual += '; serviceProbe=pass (output not persisted)'
        }
    } catch {
        $positivePassed = $false
        $positiveActual += "; serviceProbeError=$($_.Exception.GetType().Name)"
    }
    Add-ValidationCheck -Id 'lab21-cp01.positive' -CheckpointId 'LAB21-CP01' -Kind 'positive' -Required ([bool]$checkpointState.required) -Passed $positivePassed -Command 'query each manifest-recorded object by exact ID and run the authored service probe' -Expected 'Checkpoint state is pass, recorded objects exist, and service state matches.' -Actual $positiveActual

    $unsafe = @($checkpointObjects | Where-Object { $_.ownership.method -eq 'manifest-id-and-tags' -and ($_.ownership.expectedTags.runId -ne $RunId) })

    # AUTHORED SERVICE ASSERTION: negative
    $negativeProbePassed = $true
    $negativeProbeActual = 'serviceProbe=pass (output not persisted)'
    try {
        $LASTEXITCODE = 0
        & {
$watchers = az network watcher list --query "[?location=='$Location'] | length(@)" --output tsv
if ([int]$watchers -lt 1) { throw 'Network Watcher is not available in the selected region.' }
        } | Out-Null
        if ($LASTEXITCODE -ne 0) {
            $negativeProbePassed = $false
            $negativeProbeActual = "serviceProbeExit=$LASTEXITCODE"
        }
    } catch {
        $negativeProbePassed = $false
        $negativeProbeActual = "serviceProbeError=$($_.Exception.GetType().Name)"
    }
    $negativePassed = $unsafe.Count -eq 0 -and $negativeProbePassed
    $negativeActual = "unsafeObjects=$($unsafe.Count)" + "; $negativeProbeActual"
    Add-ValidationCheck -Id 'lab21-cp01.negative' -CheckpointId 'LAB21-CP01' -Kind 'negative' -Required ([bool]$checkpointState.required) -Passed $negativePassed -Command 'run the authored denied-state probe and compare manifest ownership' -Expected 'The service boundary and ownership checks both pass.' -Actual $negativeActual
    }
} else {
    $active = [System.Collections.Generic.List[string]]::new()
    foreach ($managedObject in $checkpointObjects) {
        if ($managedObject.type -eq 'Microsoft.Management/managementGroups') {
            $null = az account management-group show --name $managedObject.name --output none 2>$null
            if ($LASTEXITCODE -eq 0) { $active.Add([string]$managedObject.id) }
        } elseif ($managedObject.kind -eq 'azure-resource') {
            $null = az resource show --ids $managedObject.id --output none 2>$null
            if ($LASTEXITCODE -eq 0) { $active.Add([string]$managedObject.id) }
        } elseif ($managedObject.kind -eq 'entra-object') {
            $null = az rest --method get --url "https://graph.microsoft.com/v1.0/directoryObjects/$($managedObject.id)" --output none 2>$null
            if ($LASTEXITCODE -eq 0) { $active.Add([string]$managedObject.id) }
        }
    }
    Add-ValidationCheck -Id 'lab21-cp01.residual' -CheckpointId 'LAB21-CP01' -Kind 'residual' -Required ([bool]$checkpointState.required) -Passed ($active.Count -eq 0) -Command 'query every recorded object by exact ID after cleanup' -Expected 'No active manifest-managed object remains.' -Actual "active=$($active -join ',')"
}
# CHECKPOINT LAB21-CP01 END

# CHECKPOINT LAB21-CP02 BEGIN
$checkpointState = @($state.checkpointStates | Where-Object { $_.checkpointId -eq 'LAB21-CP02' })[0]
$checkpointObjects = @($state.managedObjects | Where-Object { $_.checkpointId -eq 'LAB21-CP02' })

if ($Mode -eq 'Deployment') {
    if ($checkpointState.status -eq 'skipped') {
        Add-ValidationCheck -Id 'lab21-cp02.positive' -CheckpointId 'LAB21-CP02' -Kind 'positive' -Required ([bool]$checkpointState.required) -Passed $true -Skipped $true -Command 'optional gate evaluation' -Expected 'The optional checkpoint is explicitly skipped when its documented input is absent.' -Actual 'optional gate absent'
        Add-ValidationCheck -Id 'lab21-cp02.negative' -CheckpointId 'LAB21-CP02' -Kind 'negative' -Required ([bool]$checkpointState.required) -Passed $true -Skipped $true -Command 'optional gate evaluation' -Expected 'No mutation occurs for a skipped optional checkpoint.' -Actual 'optional gate absent'
    } else {
    $positivePassed = $checkpointState.status -eq 'pass'
    $positiveActual = "checkpoint=$($checkpointState.status); managedObjects=$($checkpointObjects.Count)"
    foreach ($managedObject in $checkpointObjects) {
        if ($managedObject.type -eq 'Microsoft.Management/managementGroups') {
            $null = az account management-group show --name $managedObject.name --output none 2>$null
            if ($LASTEXITCODE -ne 0) { $positivePassed = $false; $positiveActual += "; missing=$($managedObject.id)" }
        } elseif ($managedObject.kind -eq 'azure-resource') {
            $null = az resource show --ids $managedObject.id --output none 2>$null
            if ($LASTEXITCODE -ne 0) { $positivePassed = $false; $positiveActual += "; missing=$($managedObject.id)" }
        } elseif ($managedObject.kind -eq 'entra-object') {
            $null = az rest --method get --url "https://graph.microsoft.com/v1.0/directoryObjects/$($managedObject.id)" --output none 2>$null
            if ($LASTEXITCODE -ne 0) { $positivePassed = $false; $positiveActual += "; missing=$($managedObject.id)" }
        }
    }

    # AUTHORED SERVICE ASSERTION: positive
    try {
        $LASTEXITCODE = 0
        & {
$probePort = az network lb probe show --resource-group $ResourceGroupName --lb-name $loadBalancer --name ssh --query port --output tsv
if ([int]$probePort -ne 22) { throw "Probe port is $probePort." }
        } | Out-Null
        if ($LASTEXITCODE -ne 0) {
            $positivePassed = $false
            $positiveActual += "; serviceProbeExit=$LASTEXITCODE"
        } else {
            $positiveActual += '; serviceProbe=pass (output not persisted)'
        }
    } catch {
        $positivePassed = $false
        $positiveActual += "; serviceProbeError=$($_.Exception.GetType().Name)"
    }
    Add-ValidationCheck -Id 'lab21-cp02.positive' -CheckpointId 'LAB21-CP02' -Kind 'positive' -Required ([bool]$checkpointState.required) -Passed $positivePassed -Command 'query each manifest-recorded object by exact ID and run the authored service probe' -Expected 'Checkpoint state is pass, recorded objects exist, and service state matches.' -Actual $positiveActual

    $unsafe = @($checkpointObjects | Where-Object { $_.ownership.method -eq 'manifest-id-and-tags' -and ($_.ownership.expectedTags.runId -ne $RunId) })

    # AUTHORED SERVICE ASSERTION: negative
    $negativeProbePassed = $true
    $negativeProbeActual = 'serviceProbe=pass (output not persisted)'
    try {
        $LASTEXITCODE = 0
        & {
$frontendPort = az network lb rule show --resource-group $ResourceGroupName --lb-name $loadBalancer --name ssh --query frontendPort --output tsv
if ([int]$frontendPort -ne 2222) { throw "Frontend port is $frontendPort." }
        } | Out-Null
        if ($LASTEXITCODE -ne 0) {
            $negativeProbePassed = $false
            $negativeProbeActual = "serviceProbeExit=$LASTEXITCODE"
        }
    } catch {
        $negativeProbePassed = $false
        $negativeProbeActual = "serviceProbeError=$($_.Exception.GetType().Name)"
    }
    $negativePassed = $unsafe.Count -eq 0 -and $negativeProbePassed
    $negativeActual = "unsafeObjects=$($unsafe.Count)" + "; $negativeProbeActual"
    Add-ValidationCheck -Id 'lab21-cp02.negative' -CheckpointId 'LAB21-CP02' -Kind 'negative' -Required ([bool]$checkpointState.required) -Passed $negativePassed -Command 'run the authored denied-state probe and compare manifest ownership' -Expected 'The service boundary and ownership checks both pass.' -Actual $negativeActual
    }
} else {
    $active = [System.Collections.Generic.List[string]]::new()
    foreach ($managedObject in $checkpointObjects) {
        if ($managedObject.type -eq 'Microsoft.Management/managementGroups') {
            $null = az account management-group show --name $managedObject.name --output none 2>$null
            if ($LASTEXITCODE -eq 0) { $active.Add([string]$managedObject.id) }
        } elseif ($managedObject.kind -eq 'azure-resource') {
            $null = az resource show --ids $managedObject.id --output none 2>$null
            if ($LASTEXITCODE -eq 0) { $active.Add([string]$managedObject.id) }
        } elseif ($managedObject.kind -eq 'entra-object') {
            $null = az rest --method get --url "https://graph.microsoft.com/v1.0/directoryObjects/$($managedObject.id)" --output none 2>$null
            if ($LASTEXITCODE -eq 0) { $active.Add([string]$managedObject.id) }
        }
    }
    Add-ValidationCheck -Id 'lab21-cp02.residual' -CheckpointId 'LAB21-CP02' -Kind 'residual' -Required ([bool]$checkpointState.required) -Passed ($active.Count -eq 0) -Command 'query every recorded object by exact ID after cleanup' -Expected 'No active manifest-managed object remains.' -Actual "active=$($active -join ',')"
}
# CHECKPOINT LAB21-CP02 END

# CHECKPOINT LAB21-CP03 BEGIN
$checkpointState = @($state.checkpointStates | Where-Object { $_.checkpointId -eq 'LAB21-CP03' })[0]
$checkpointObjects = @($state.managedObjects | Where-Object { $_.checkpointId -eq 'LAB21-CP03' })

if ($Mode -eq 'Deployment') {
    if ($checkpointState.status -eq 'skipped') {
        Add-ValidationCheck -Id 'lab21-cp03.positive' -CheckpointId 'LAB21-CP03' -Kind 'positive' -Required ([bool]$checkpointState.required) -Passed $true -Skipped $true -Command 'optional gate evaluation' -Expected 'The optional checkpoint is explicitly skipped when its documented input is absent.' -Actual 'optional gate absent'
        Add-ValidationCheck -Id 'lab21-cp03.negative' -CheckpointId 'LAB21-CP03' -Kind 'negative' -Required ([bool]$checkpointState.required) -Passed $true -Skipped $true -Command 'optional gate evaluation' -Expected 'No mutation occurs for a skipped optional checkpoint.' -Actual 'optional gate absent'
    } else {
    $positivePassed = $checkpointState.status -eq 'pass'
    $positiveActual = "checkpoint=$($checkpointState.status); managedObjects=$($checkpointObjects.Count)"
    foreach ($managedObject in $checkpointObjects) {
        if ($managedObject.type -eq 'Microsoft.Management/managementGroups') {
            $null = az account management-group show --name $managedObject.name --output none 2>$null
            if ($LASTEXITCODE -ne 0) { $positivePassed = $false; $positiveActual += "; missing=$($managedObject.id)" }
        } elseif ($managedObject.kind -eq 'azure-resource') {
            $null = az resource show --ids $managedObject.id --output none 2>$null
            if ($LASTEXITCODE -ne 0) { $positivePassed = $false; $positiveActual += "; missing=$($managedObject.id)" }
        } elseif ($managedObject.kind -eq 'entra-object') {
            $null = az rest --method get --url "https://graph.microsoft.com/v1.0/directoryObjects/$($managedObject.id)" --output none 2>$null
            if ($LASTEXITCODE -ne 0) { $positivePassed = $false; $positiveActual += "; missing=$($managedObject.id)" }
        }
    }

    # AUTHORED SERVICE ASSERTION: positive
    try {
        $LASTEXITCODE = 0
        & {
$backends = az network lb address-pool show --resource-group $ResourceGroupName --lb-name $loadBalancer --name backend --query "backendIPConfigurations | length(@)" --output tsv
if ([int]$backends -ne 2) { throw "Expected two LB backends; found $backends." }
        } | Out-Null
        if ($LASTEXITCODE -ne 0) {
            $positivePassed = $false
            $positiveActual += "; serviceProbeExit=$LASTEXITCODE"
        } else {
            $positiveActual += '; serviceProbe=pass (output not persisted)'
        }
    } catch {
        $positivePassed = $false
        $positiveActual += "; serviceProbeError=$($_.Exception.GetType().Name)"
    }
    Add-ValidationCheck -Id 'lab21-cp03.positive' -CheckpointId 'LAB21-CP03' -Kind 'positive' -Required ([bool]$checkpointState.required) -Passed $positivePassed -Command 'query each manifest-recorded object by exact ID and run the authored service probe' -Expected 'Checkpoint state is pass, recorded objects exist, and service state matches.' -Actual $positiveActual

    $unsafe = @($checkpointObjects | Where-Object { $_.ownership.method -eq 'manifest-id-and-tags' -and ($_.ownership.expectedTags.runId -ne $RunId) })

    # AUTHORED SERVICE ASSERTION: negative
    $negativeProbePassed = $true
    $negativeProbeActual = 'serviceProbe=pass (output not persisted)'
    try {
        $LASTEXITCODE = 0
        & {
$publicIps = az vm list-ip-addresses --resource-group $ResourceGroupName --query "[?virtualMachine.name=='$vm1' || virtualMachine.name=='$vm2'].virtualMachine.network.publicIpAddresses[] | length(@)" --output tsv
if ($publicIps -and [int]$publicIps -ne 0) { throw 'A backend VM has a direct public IP.' }
        } | Out-Null
        if ($LASTEXITCODE -ne 0) {
            $negativeProbePassed = $false
            $negativeProbeActual = "serviceProbeExit=$LASTEXITCODE"
        }
    } catch {
        $negativeProbePassed = $false
        $negativeProbeActual = "serviceProbeError=$($_.Exception.GetType().Name)"
    }
    $negativePassed = $unsafe.Count -eq 0 -and $negativeProbePassed
    $negativeActual = "unsafeObjects=$($unsafe.Count)" + "; $negativeProbeActual"
    Add-ValidationCheck -Id 'lab21-cp03.negative' -CheckpointId 'LAB21-CP03' -Kind 'negative' -Required ([bool]$checkpointState.required) -Passed $negativePassed -Command 'run the authored denied-state probe and compare manifest ownership' -Expected 'The service boundary and ownership checks both pass.' -Actual $negativeActual
    }
} else {
    $active = [System.Collections.Generic.List[string]]::new()
    foreach ($managedObject in $checkpointObjects) {
        if ($managedObject.type -eq 'Microsoft.Management/managementGroups') {
            $null = az account management-group show --name $managedObject.name --output none 2>$null
            if ($LASTEXITCODE -eq 0) { $active.Add([string]$managedObject.id) }
        } elseif ($managedObject.kind -eq 'azure-resource') {
            $null = az resource show --ids $managedObject.id --output none 2>$null
            if ($LASTEXITCODE -eq 0) { $active.Add([string]$managedObject.id) }
        } elseif ($managedObject.kind -eq 'entra-object') {
            $null = az rest --method get --url "https://graph.microsoft.com/v1.0/directoryObjects/$($managedObject.id)" --output none 2>$null
            if ($LASTEXITCODE -eq 0) { $active.Add([string]$managedObject.id) }
        }
    }
    Add-ValidationCheck -Id 'lab21-cp03.residual' -CheckpointId 'LAB21-CP03' -Kind 'residual' -Required ([bool]$checkpointState.required) -Passed ($active.Count -eq 0) -Command 'query every recorded object by exact ID after cleanup' -Expected 'No active manifest-managed object remains.' -Actual "active=$($active -join ',')"
}
# CHECKPOINT LAB21-CP03 END

# CHECKPOINT LAB21-CP04 BEGIN
$checkpointState = @($state.checkpointStates | Where-Object { $_.checkpointId -eq 'LAB21-CP04' })[0]
$checkpointObjects = @($state.managedObjects | Where-Object { $_.checkpointId -eq 'LAB21-CP04' })

if ($Mode -eq 'Deployment') {
    if ($checkpointState.status -eq 'skipped') {
        Add-ValidationCheck -Id 'lab21-cp04.positive' -CheckpointId 'LAB21-CP04' -Kind 'positive' -Required ([bool]$checkpointState.required) -Passed $true -Skipped $true -Command 'optional gate evaluation' -Expected 'The optional checkpoint is explicitly skipped when its documented input is absent.' -Actual 'optional gate absent'
        Add-ValidationCheck -Id 'lab21-cp04.negative' -CheckpointId 'LAB21-CP04' -Kind 'negative' -Required ([bool]$checkpointState.required) -Passed $true -Skipped $true -Command 'optional gate evaluation' -Expected 'No mutation occurs for a skipped optional checkpoint.' -Actual 'optional gate absent'
    } else {
    $positivePassed = $checkpointState.status -eq 'pass'
    $positiveActual = "checkpoint=$($checkpointState.status); managedObjects=$($checkpointObjects.Count)"
    foreach ($managedObject in $checkpointObjects) {
        if ($managedObject.type -eq 'Microsoft.Management/managementGroups') {
            $null = az account management-group show --name $managedObject.name --output none 2>$null
            if ($LASTEXITCODE -ne 0) { $positivePassed = $false; $positiveActual += "; missing=$($managedObject.id)" }
        } elseif ($managedObject.kind -eq 'azure-resource') {
            $null = az resource show --ids $managedObject.id --output none 2>$null
            if ($LASTEXITCODE -ne 0) { $positivePassed = $false; $positiveActual += "; missing=$($managedObject.id)" }
        } elseif ($managedObject.kind -eq 'entra-object') {
            $null = az rest --method get --url "https://graph.microsoft.com/v1.0/directoryObjects/$($managedObject.id)" --output none 2>$null
            if ($LASTEXITCODE -ne 0) { $positivePassed = $false; $positiveActual += "; missing=$($managedObject.id)" }
        }
    }

    # AUTHORED SERVICE ASSERTION: positive
    try {
        $LASTEXITCODE = 0
        & {
$monitorState = az network watcher connection-monitor show --location $Location --name $state.inputs['external-connection-monitor-name'] --query provisioningState --output tsv
if ($monitorState -ne 'Succeeded') { throw "Connection Monitor provisioning state is $monitorState." }
        } | Out-Null
        if ($LASTEXITCODE -ne 0) {
            $positivePassed = $false
            $positiveActual += "; serviceProbeExit=$LASTEXITCODE"
        } else {
            $positiveActual += '; serviceProbe=pass (output not persisted)'
        }
    } catch {
        $positivePassed = $false
        $positiveActual += "; serviceProbeError=$($_.Exception.GetType().Name)"
    }
    Add-ValidationCheck -Id 'lab21-cp04.positive' -CheckpointId 'LAB21-CP04' -Kind 'positive' -Required ([bool]$checkpointState.required) -Passed $positivePassed -Command 'query each manifest-recorded object by exact ID and run the authored service probe' -Expected 'Checkpoint state is pass, recorded objects exist, and service state matches.' -Actual $positiveActual

    $unsafe = @($checkpointObjects | Where-Object { $_.ownership.method -eq 'manifest-id-and-tags' -and ($_.ownership.expectedTags.runId -ne $RunId) })

    # AUTHORED SERVICE ASSERTION: negative
    $negativeProbePassed = $true
    $negativeProbeActual = 'serviceProbe=pass (output not persisted)'
    try {
        $LASTEXITCODE = 0
        & {
$agentCount = 0
foreach ($vmName in @($vm1, $vm2)) {
    $stateValue = az vm extension show --resource-group $ResourceGroupName --vm-name $vmName --name NetworkWatcherAgentLinux --query provisioningState --output tsv
    if ($stateValue -eq 'Succeeded') { $agentCount++ }
}
if ($agentCount -ne 2) { throw "Expected two healthy Network Watcher agents; found $agentCount." }
        } | Out-Null
        if ($LASTEXITCODE -ne 0) {
            $negativeProbePassed = $false
            $negativeProbeActual = "serviceProbeExit=$LASTEXITCODE"
        }
    } catch {
        $negativeProbePassed = $false
        $negativeProbeActual = "serviceProbeError=$($_.Exception.GetType().Name)"
    }
    $negativePassed = $unsafe.Count -eq 0 -and $negativeProbePassed
    $negativeActual = "unsafeObjects=$($unsafe.Count)" + "; $negativeProbeActual"
    Add-ValidationCheck -Id 'lab21-cp04.negative' -CheckpointId 'LAB21-CP04' -Kind 'negative' -Required ([bool]$checkpointState.required) -Passed $negativePassed -Command 'run the authored denied-state probe and compare manifest ownership' -Expected 'The service boundary and ownership checks both pass.' -Actual $negativeActual
    }
} else {
    $active = [System.Collections.Generic.List[string]]::new()
    foreach ($managedObject in $checkpointObjects) {
        if ($managedObject.type -eq 'Microsoft.Management/managementGroups') {
            $null = az account management-group show --name $managedObject.name --output none 2>$null
            if ($LASTEXITCODE -eq 0) { $active.Add([string]$managedObject.id) }
        } elseif ($managedObject.kind -eq 'azure-resource') {
            $null = az resource show --ids $managedObject.id --output none 2>$null
            if ($LASTEXITCODE -eq 0) { $active.Add([string]$managedObject.id) }
        } elseif ($managedObject.kind -eq 'entra-object') {
            $null = az rest --method get --url "https://graph.microsoft.com/v1.0/directoryObjects/$($managedObject.id)" --output none 2>$null
            if ($LASTEXITCODE -eq 0) { $active.Add([string]$managedObject.id) }
        }
    }
    Add-ValidationCheck -Id 'lab21-cp04.residual' -CheckpointId 'LAB21-CP04' -Kind 'residual' -Required ([bool]$checkpointState.required) -Passed ($active.Count -eq 0) -Command 'query every recorded object by exact ID after cleanup' -Expected 'No active manifest-managed object remains.' -Actual "active=$($active -join ',')"
}
# CHECKPOINT LAB21-CP04 END

# CHECKPOINT LAB21-CP05 BEGIN
$checkpointState = @($state.checkpointStates | Where-Object { $_.checkpointId -eq 'LAB21-CP05' })[0]
$checkpointObjects = @($state.managedObjects | Where-Object { $_.checkpointId -eq 'LAB21-CP05' })

if ($Mode -eq 'Deployment') {
    if ($checkpointState.status -eq 'skipped') {
        Add-ValidationCheck -Id 'lab21-cp05.positive' -CheckpointId 'LAB21-CP05' -Kind 'positive' -Required ([bool]$checkpointState.required) -Passed $true -Skipped $true -Command 'optional gate evaluation' -Expected 'The optional checkpoint is explicitly skipped when its documented input is absent.' -Actual 'optional gate absent'
        Add-ValidationCheck -Id 'lab21-cp05.negative' -CheckpointId 'LAB21-CP05' -Kind 'negative' -Required ([bool]$checkpointState.required) -Passed $true -Skipped $true -Command 'optional gate evaluation' -Expected 'No mutation occurs for a skipped optional checkpoint.' -Actual 'optional gate absent'
    } else {
    $positivePassed = $checkpointState.status -eq 'pass'
    $positiveActual = "checkpoint=$($checkpointState.status); managedObjects=$($checkpointObjects.Count)"
    foreach ($managedObject in $checkpointObjects) {
        if ($managedObject.type -eq 'Microsoft.Management/managementGroups') {
            $null = az account management-group show --name $managedObject.name --output none 2>$null
            if ($LASTEXITCODE -ne 0) { $positivePassed = $false; $positiveActual += "; missing=$($managedObject.id)" }
        } elseif ($managedObject.kind -eq 'azure-resource') {
            $null = az resource show --ids $managedObject.id --output none 2>$null
            if ($LASTEXITCODE -ne 0) { $positivePassed = $false; $positiveActual += "; missing=$($managedObject.id)" }
        } elseif ($managedObject.kind -eq 'entra-object') {
            $null = az rest --method get --url "https://graph.microsoft.com/v1.0/directoryObjects/$($managedObject.id)" --output none 2>$null
            if ($LASTEXITCODE -ne 0) { $positivePassed = $false; $positiveActual += "; missing=$($managedObject.id)" }
        }
    }

    # AUTHORED SERVICE ASSERTION: positive
    try {
        $LASTEXITCODE = 0
        & {
$sku = az network lb show --resource-group $ResourceGroupName --name $loadBalancer --query sku.name --output tsv
if ($sku -ne 'Standard') { throw "Load Balancer SKU is $sku." }
        } | Out-Null
        if ($LASTEXITCODE -ne 0) {
            $positivePassed = $false
            $positiveActual += "; serviceProbeExit=$LASTEXITCODE"
        } else {
            $positiveActual += '; serviceProbe=pass (output not persisted)'
        }
    } catch {
        $positivePassed = $false
        $positiveActual += "; serviceProbeError=$($_.Exception.GetType().Name)"
    }
    Add-ValidationCheck -Id 'lab21-cp05.positive' -CheckpointId 'LAB21-CP05' -Kind 'positive' -Required ([bool]$checkpointState.required) -Passed $positivePassed -Command 'query each manifest-recorded object by exact ID and run the authored service probe' -Expected 'Checkpoint state is pass, recorded objects exist, and service state matches.' -Actual $positiveActual

    $unsafe = @($checkpointObjects | Where-Object { $_.ownership.method -eq 'manifest-id-and-tags' -and ($_.ownership.expectedTags.runId -ne $RunId) })

    # AUTHORED SERVICE ASSERTION: negative
    $negativeProbePassed = $true
    $negativeProbeActual = 'serviceProbe=pass (output not persisted)'
    try {
        $LASTEXITCODE = 0
        & {
$faultRule = az network nsg rule list --resource-group $ResourceGroupName --nsg-name $nsg --query "[?name=='DenyInjectedSsh'] | length(@)" --output tsv
if ([int]$faultRule -ne 0) { throw 'The injected deny rule still exists.' }
        } | Out-Null
        if ($LASTEXITCODE -ne 0) {
            $negativeProbePassed = $false
            $negativeProbeActual = "serviceProbeExit=$LASTEXITCODE"
        }
    } catch {
        $negativeProbePassed = $false
        $negativeProbeActual = "serviceProbeError=$($_.Exception.GetType().Name)"
    }
    $negativePassed = $unsafe.Count -eq 0 -and $negativeProbePassed
    $negativeActual = "unsafeObjects=$($unsafe.Count)" + "; $negativeProbeActual"
    Add-ValidationCheck -Id 'lab21-cp05.negative' -CheckpointId 'LAB21-CP05' -Kind 'negative' -Required ([bool]$checkpointState.required) -Passed $negativePassed -Command 'run the authored denied-state probe and compare manifest ownership' -Expected 'The service boundary and ownership checks both pass.' -Actual $negativeActual
    }
} else {
    $active = [System.Collections.Generic.List[string]]::new()
    foreach ($managedObject in $checkpointObjects) {
        if ($managedObject.type -eq 'Microsoft.Management/managementGroups') {
            $null = az account management-group show --name $managedObject.name --output none 2>$null
            if ($LASTEXITCODE -eq 0) { $active.Add([string]$managedObject.id) }
        } elseif ($managedObject.kind -eq 'azure-resource') {
            $null = az resource show --ids $managedObject.id --output none 2>$null
            if ($LASTEXITCODE -eq 0) { $active.Add([string]$managedObject.id) }
        } elseif ($managedObject.kind -eq 'entra-object') {
            $null = az rest --method get --url "https://graph.microsoft.com/v1.0/directoryObjects/$($managedObject.id)" --output none 2>$null
            if ($LASTEXITCODE -eq 0) { $active.Add([string]$managedObject.id) }
        }
    }
    Add-ValidationCheck -Id 'lab21-cp05.residual' -CheckpointId 'LAB21-CP05' -Kind 'residual' -Required ([bool]$checkpointState.required) -Passed ($active.Count -eq 0) -Command 'query every recorded object by exact ID after cleanup' -Expected 'No active manifest-managed object remains.' -Actual "active=$($active -join ',')"
}
# CHECKPOINT LAB21-CP05 END

$requiredChecks = @($checks | Where-Object { $_.required })
$failedChecks = @($checks | Where-Object { $_.status -eq 'fail' })
$requiredSkippedChecks = @($checks | Where-Object { $_.required -and $_.status -eq 'skipped' })
$skippedChecks = @($checks | Where-Object { $_.status -eq 'skipped' })
$result = if ($failedChecks.Count -gt 0 -or $requiredSkippedChecks.Count -gt 0) { 'fail' } elseif ($skippedChecks.Count -gt 0) { 'partial' } else { 'pass' }
$document = @{
    schemaVersion = '1.0.0'
    labId = 'LAB-21'
    runId = $RunId
    mode = $Mode
    generatedAt = (Get-Date).ToUniversalTime().ToString('o')
    result = $result
    summary = @{
        required = $requiredChecks.Count
        passed = @($checks | Where-Object { $_.status -eq 'pass' }).Count
        failed = $failedChecks.Count
        skipped = $skippedChecks.Count
    }
    checks = @($checks)
}
$document | ConvertTo-Json -Depth 30 | Set-Content -LiteralPath $ValidationPath -Encoding utf8
Write-Host "Validation $Mode result: $result"
Write-Host "Artifact: $ValidationPath"
if ($result -eq 'fail') { exit 1 }
# END GENERATED AZ104 V2

Cleanup.ps1

# BEGIN GENERATED AZ104 V2
#requires -Version 7.4
[CmdletBinding()]
[Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSAvoidUsingWriteHost', '', Justification = 'Interactive lab progress is intentionally written to the host.')]
[Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSReviewUnusedParameter', '', Justification = 'All lifecycle scripts expose a stable cross-lab interface.')]
[Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSUseDeclaredVarsMoreThanAssignments', '', Justification = 'Generated task variables are intentionally shared across checkpoint blocks.')]
param(
    [string]$SubscriptionId = $env:AZ104_SUBSCRIPTION_ID,
    [Parameter(Mandatory)][ValidatePattern('^[a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?$')][string]$RunId,
    [switch]$Execute
)

Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
$PSNativeCommandUseErrorActionPreference = $true

if (-not (Get-Command az -ErrorAction SilentlyContinue)) {
    throw 'Azure CLI is required. Run the repository readiness initializer, then retry.'
}

$LabRoot = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path
$StateDir = Join-Path $LabRoot ".state/$RunId"
$Manifest = Join-Path $StateDir 'run.json'
$CleanupPath = Join-Path $StateDir 'cleanup.json'
if (-not (Test-Path -LiteralPath $Manifest)) { throw "Run manifest not found: $Manifest" }
$state = Get-Content -LiteralPath $Manifest -Raw | ConvertFrom-Json -AsHashtable
$active = @($state.managedObjects | Where-Object { $_.lifecycleStatus -eq 'active' })
function Write-CleanupRefusal {
    param([string]$Message)
    $refusalActions = @($active | ForEach-Object {
        @{ checkpointId = [string]$_.checkpointId; targetId = [string]$_.id; targetType = [string]$_.type; ownership = @{ method = [string]$_.ownership.method; verified = $false }; status = 'failed'; message = $Message }
    })
    $refusal = @{
        schemaVersion = '1.0.0'; labId = 'LAB-21'; runId = $RunId
        generatedAt = (Get-Date).ToUniversalTime().ToString('o')
        executionMode = $(if ($Execute) { 'execute' } else { 'preview' })
        result = $(if ($Execute) { 'fail' } else { 'preview' })
        ownershipVerified = $false; actions = $refusalActions
        residualChecks = @(@{ id = 'ownership.refusal'; command = 'compare manifest, active context, exact IDs, and ownership proof'; expected = 'all ownership checks pass before mutation'; actual = $Message; status = $(if ($Execute) { 'fail' } else { 'skipped' }) })
        activeManagedObjects = @($active | ForEach-Object id); retainedItems = @()
    }
    $refusal | ConvertTo-Json -Depth 30 | Set-Content -LiteralPath $CleanupPath -Encoding utf8
    throw $Message
}
if ($state.labId -ne 'LAB-21' -or $state.runId -ne $RunId) {
    Write-CleanupRefusal -Message 'Cleanup ownership refusal: manifest lab ID or run ID does not match.'
}
if ($Execute -and $active.Count -eq 0 -and $state.status -eq 'cleaned') {
    $priorRetained = @()
    if (Test-Path -LiteralPath $CleanupPath) {
        try { $priorRetained = @((Get-Content -LiteralPath $CleanupPath -Raw | ConvertFrom-Json).retainedItems) } catch { $priorRetained = @() }
    }
    $idempotent = @{
        schemaVersion = '1.0.0'; labId = 'LAB-21'; runId = $RunId
        generatedAt = (Get-Date).ToUniversalTime().ToString('o'); executionMode = 'execute'; result = 'pass'
        ownershipVerified = $true; actions = @(); activeManagedObjects = @(); retainedItems = $priorRetained
        residualChecks = @(@{ id = 'cleanup.idempotent'; command = 'read run.json active managed-object inventory'; expected = 'zero active manifest-managed objects'; actual = 'active=0; prior cleanup already completed'; status = 'pass' })
    }
    $idempotent | ConvertTo-Json -Depth 30 | Set-Content -LiteralPath $CleanupPath -Encoding utf8
    Write-Host 'Cleanup result: pass (already cleaned)'
    Write-Host "Artifact: $CleanupPath"
    return
}
try {
    $account = az account show --output json | ConvertFrom-Json
} catch {
    Write-CleanupRefusal -Message 'Cleanup ownership refusal: active Azure context could not be read.'
}
if ([string]$account.tenantId -ne [string]$state.context.tenantId -or [string]$account.id -ne [string]$state.context.subscriptionId) {
    Write-CleanupRefusal -Message 'Cleanup ownership refusal: active context does not match the manifest.'
}
$SubscriptionId = [string]$state.context.subscriptionId
$Location = [string]$state.inputs['location']
$SecondaryLocation = [string]$state.inputs['secondary-location']
$ResourceGroupName = $(if ('21' -in @('00', '01', '02')) { $null } else { "rg-az104-l21-$RunId" })
$suffix = (($RunId -replace '[^a-z0-9]', '') + '000000000000').Substring(0, 12)
$connectionMonitorId = $state.inputs['external-connection-monitor-id']
$connectionMonitorName = $state.inputs['external-connection-monitor-name']

$ownershipVerified = $true
$verifiedOwnershipById = @{}
$absentIds = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase)
$verifiedResourceGroupIds = [System.Collections.Generic.List[string]]::new()
$resourceGroupObjects = @($active | Where-Object { $_.type -eq 'Microsoft.Resources/resourceGroups' })
$resourceGroupObject = $resourceGroupObjects | Select-Object -First 1
$nativeProbePreference = $PSNativeCommandUseErrorActionPreference
$PSNativeCommandUseErrorActionPreference = $false

# A manifest entry is necessary but is not, by itself, proof of ownership. Verify
# each manifest-id-and-tags boundary against the live Azure tags before mutation.
foreach ($groupObject in $resourceGroupObjects) {
    $groupVerified = $false
    try {
        $tagJson = az group show --ids $groupObject.id --query tags --output json 2>$null
        if ($LASTEXITCODE -eq 0 -and $tagJson) {
            $tags = $tagJson | ConvertFrom-Json -AsHashtable
            $expectedTags = $groupObject.ownership.expectedTags
            $groupVerified =
                $groupObject.ownership.method -eq 'manifest-id-and-tags' -and
                [string]$tags['purpose'] -eq [string]$expectedTags['purpose'] -and
                [string]$tags['labId'] -eq [string]$expectedTags['labId'] -and
                [string]$tags['runId'] -eq [string]$expectedTags['runId'] -and
                [string]$tags['purpose'] -eq 'az104-lab' -and
                [string]$tags['labId'] -eq '21' -and
                [string]$tags['runId'] -eq $RunId
        } else {
            $groupExists = az group exists --subscription $state.context.subscriptionId --name $groupObject.name --output tsv 2>$null
            if ($LASTEXITCODE -eq 0 -and $groupExists -eq 'false') {
                $null = $absentIds.Add([string]$groupObject.id)
                $groupVerified = $true
            }
        }
    } catch {
        $groupVerified = $false
    }
    $verifiedOwnershipById[[string]$groupObject.id] = $groupVerified
    if ($groupVerified) { $verifiedResourceGroupIds.Add([string]$groupObject.id) }
    if (-not $groupVerified) { $ownershipVerified = $false }
}

foreach ($object in $active) {
    $objectId = [string]$object.id
    if ($verifiedOwnershipById.ContainsKey($objectId)) { continue }
    $method = [string]$object.ownership.method
    $objectVerified = $false

    $absentGroupBoundary = @($resourceGroupObjects | Where-Object {
        $absentIds.Contains([string]$_.id) -and $objectId.StartsWith("$($_.id)/", [System.StringComparison]::OrdinalIgnoreCase)
    }).Count -gt 0
    if ($absentGroupBoundary) {
        $null = $absentIds.Add($objectId)
        $verifiedOwnershipById[$objectId] = $true
        continue
    }

    if ($method -eq 'manifest-id-and-tags' -and $object.kind -eq 'azure-resource') {
        $expectedTags = $object.ownership.expectedTags
        $manifestTagsMatch =
            [string]$expectedTags['purpose'] -eq 'az104-lab' -and
            [string]$expectedTags['labId'] -eq '21' -and
            [string]$expectedTags['runId'] -eq $RunId
        $verifiedByGroupBoundary = @($verifiedResourceGroupIds | Where-Object {
            $objectId.StartsWith("$_/", [System.StringComparison]::OrdinalIgnoreCase)
        }).Count -gt 0
        $verifiedByOwnTags = $false
        try {
            $tagJson = az resource show --ids $objectId --query tags --output json 2>$null
            if ($LASTEXITCODE -eq 0 -and $tagJson) {
                $tags = $tagJson | ConvertFrom-Json -AsHashtable
                $verifiedByOwnTags =
                    [string]$tags['purpose'] -eq [string]$expectedTags['purpose'] -and
                    [string]$tags['labId'] -eq [string]$expectedTags['labId'] -and
                    [string]$tags['runId'] -eq [string]$expectedTags['runId']
            } elseif ($verifiedByGroupBoundary) {
                $parentGroupObject = $resourceGroupObjects | Where-Object { $objectId.StartsWith("$($_.id)/", [System.StringComparison]::OrdinalIgnoreCase) } | Select-Object -First 1
                $exactCount = az resource list --resource-group $parentGroupObject.name --query "[?id=='$objectId'] | length(@)" --output tsv 2>$null
                if ($LASTEXITCODE -eq 0 -and [int]$exactCount -eq 0) {
                    $null = $absentIds.Add($objectId)
                    $verifiedByOwnTags = $true
                }
            }
        } catch {
            $verifiedByOwnTags = $false
        }
        $objectVerified = $manifestTagsMatch -and ($verifiedByOwnTags -or $verifiedByGroupBoundary)
    } elseif ($method -eq 'manifest-id' -and $object.kind -eq 'azure-resource' -and $object.type -eq 'Microsoft.Management/managementGroups') {
        try {
            $managementGroupJson = az account management-group show --name $object.name --expand --recurse --output json 2>&1
            if ($LASTEXITCODE -eq 0) {
                $managementGroup = $managementGroupJson | ConvertFrom-Json
                $objectVerified = [string]$managementGroup.id -eq $objectId -and @($managementGroup.children).Count -eq 0
            } elseif ([string]$managementGroupJson -match '(?i)404|not.?found') {
                $null = $absentIds.Add($objectId)
                $objectVerified = $true
            }
        } catch {
            $objectVerified = $false
        }
    } elseif ($method -eq 'manifest-id' -and $object.kind -eq 'entra-object') {
        try {
            $graphResult = az rest --method get --url "https://graph.microsoft.com/v1.0/directoryObjects/$objectId" --output none 2>&1
            if ($LASTEXITCODE -eq 0) {
                $objectVerified = $true
            } elseif ([string]$graphResult -match '(?i)404|Request_ResourceNotFound') {
                $null = $absentIds.Add($objectId)
                $objectVerified = $true
            }
        } catch {
            $objectVerified = $false
        }
    } elseif ($method -eq 'manifest-id' -and $object.kind -eq 'azure-resource') {
        try {
            $resourceResult = az resource show --ids $objectId --output none 2>&1
            if ($LASTEXITCODE -eq 0) {
                $objectVerified = $true
            } elseif ([string]$resourceResult -match '(?i)404|ResourceNotFound|could not be found') {
                $null = $absentIds.Add($objectId)
                $objectVerified = $true
            }
        } catch {
            $objectVerified = $false
        }
    } elseif ($method -eq 'local-path') {
        try {
            $candidate = if ([System.IO.Path]::IsPathRooted($objectId)) { $objectId } else { Join-Path $LabRoot $objectId }
            $fullPath = [System.IO.Path]::GetFullPath($candidate)
            $statePrefix = [System.IO.Path]::GetFullPath($StateDir).TrimEnd([System.IO.Path]::DirectorySeparatorChar) + [System.IO.Path]::DirectorySeparatorChar
            $inStateBoundary = $fullPath.StartsWith($statePrefix, [System.StringComparison]::OrdinalIgnoreCase)
            $objectVerified = $inStateBoundary
            if ($inStateBoundary -and -not (Test-Path -LiteralPath $fullPath)) { $null = $absentIds.Add($objectId) }
        } catch {
            $objectVerified = $false
        }
    }

    $verifiedOwnershipById[$objectId] = $objectVerified
    if (-not $objectVerified) { $ownershipVerified = $false }
}
$PSNativeCommandUseErrorActionPreference = $nativeProbePreference

if (-not $ownershipVerified) {
    $failedOwnershipIds = @($active | Where-Object { -not $verifiedOwnershipById[[string]$_.id] } | ForEach-Object id)
    Write-CleanupRefusal -Message "Cleanup ownership refusal: live ID and ownership proof failed for $($failedOwnershipIds -join ', ')."
}

$actions = [System.Collections.Generic.List[object]]::new()
$residualChecks = [System.Collections.Generic.List[object]]::new()
if (-not $Execute) {
    Write-Host 'Preview only. The exact manifest-owned targets below will not be changed.'
}

# CHECKPOINT LAB21-CP05 BEGIN
$targets = @($state.managedObjects | Where-Object { $_.checkpointId -eq 'LAB21-CP05' -and $_.lifecycleStatus -eq 'active' })
foreach ($target in $targets) {
    $alreadyAbsent = $absentIds.Contains([string]$target.id)
    $actions.Add(@{ checkpointId = 'LAB21-CP05'; targetId = [string]$target.id; targetType = [string]$target.type; ownership = @{ method = [string]$target.ownership.method; verified = [bool]$verifiedOwnershipById[[string]$target.id] }; status = $(if ($alreadyAbsent) { 'skipped' } else { 'preview' }); message = $(if ($alreadyAbsent) { 'The exact manifest ID was already absent; idempotent cleanup recorded the safe end state.' } elseif ($Execute) { 'Pending removal through the dependency-safe ownership boundary.' } else { 'Would remove this exact manifest-recorded target.' }) })
}
$state.updatedAt = (Get-Date).ToUniversalTime().ToString('o')
$state | ConvertTo-Json -Depth 30 | Set-Content -LiteralPath $Manifest -Encoding utf8
# CHECKPOINT LAB21-CP05 END

# CHECKPOINT LAB21-CP04 BEGIN
$targets = @($state.managedObjects | Where-Object { $_.checkpointId -eq 'LAB21-CP04' -and $_.lifecycleStatus -eq 'active' })
foreach ($target in $targets) {
    $alreadyAbsent = $absentIds.Contains([string]$target.id)
    $actions.Add(@{ checkpointId = 'LAB21-CP04'; targetId = [string]$target.id; targetType = [string]$target.type; ownership = @{ method = [string]$target.ownership.method; verified = [bool]$verifiedOwnershipById[[string]$target.id] }; status = $(if ($alreadyAbsent) { 'skipped' } else { 'preview' }); message = $(if ($alreadyAbsent) { 'The exact manifest ID was already absent; idempotent cleanup recorded the safe end state.' } elseif ($Execute) { 'Pending removal through the dependency-safe ownership boundary.' } else { 'Would remove this exact manifest-recorded target.' }) })
}
$state.updatedAt = (Get-Date).ToUniversalTime().ToString('o')
$state | ConvertTo-Json -Depth 30 | Set-Content -LiteralPath $Manifest -Encoding utf8
# CHECKPOINT LAB21-CP04 END

# CHECKPOINT LAB21-CP03 BEGIN
$targets = @($state.managedObjects | Where-Object { $_.checkpointId -eq 'LAB21-CP03' -and $_.lifecycleStatus -eq 'active' })
foreach ($target in $targets) {
    $alreadyAbsent = $absentIds.Contains([string]$target.id)
    $actions.Add(@{ checkpointId = 'LAB21-CP03'; targetId = [string]$target.id; targetType = [string]$target.type; ownership = @{ method = [string]$target.ownership.method; verified = [bool]$verifiedOwnershipById[[string]$target.id] }; status = $(if ($alreadyAbsent) { 'skipped' } else { 'preview' }); message = $(if ($alreadyAbsent) { 'The exact manifest ID was already absent; idempotent cleanup recorded the safe end state.' } elseif ($Execute) { 'Pending removal through the dependency-safe ownership boundary.' } else { 'Would remove this exact manifest-recorded target.' }) })
}
$state.updatedAt = (Get-Date).ToUniversalTime().ToString('o')
$state | ConvertTo-Json -Depth 30 | Set-Content -LiteralPath $Manifest -Encoding utf8
# CHECKPOINT LAB21-CP03 END

# CHECKPOINT LAB21-CP02 BEGIN
$targets = @($state.managedObjects | Where-Object { $_.checkpointId -eq 'LAB21-CP02' -and $_.lifecycleStatus -eq 'active' })
foreach ($target in $targets) {
    $alreadyAbsent = $absentIds.Contains([string]$target.id)
    $actions.Add(@{ checkpointId = 'LAB21-CP02'; targetId = [string]$target.id; targetType = [string]$target.type; ownership = @{ method = [string]$target.ownership.method; verified = [bool]$verifiedOwnershipById[[string]$target.id] }; status = $(if ($alreadyAbsent) { 'skipped' } else { 'preview' }); message = $(if ($alreadyAbsent) { 'The exact manifest ID was already absent; idempotent cleanup recorded the safe end state.' } elseif ($Execute) { 'Pending removal through the dependency-safe ownership boundary.' } else { 'Would remove this exact manifest-recorded target.' }) })
}
$state.updatedAt = (Get-Date).ToUniversalTime().ToString('o')
$state | ConvertTo-Json -Depth 30 | Set-Content -LiteralPath $Manifest -Encoding utf8
# CHECKPOINT LAB21-CP02 END

# CHECKPOINT LAB21-CP01 BEGIN
$targets = @($state.managedObjects | Where-Object { $_.checkpointId -eq 'LAB21-CP01' -and $_.lifecycleStatus -eq 'active' })
foreach ($target in $targets) {
    $alreadyAbsent = $absentIds.Contains([string]$target.id)
    $actions.Add(@{ checkpointId = 'LAB21-CP01'; targetId = [string]$target.id; targetType = [string]$target.type; ownership = @{ method = [string]$target.ownership.method; verified = [bool]$verifiedOwnershipById[[string]$target.id] }; status = $(if ($alreadyAbsent) { 'skipped' } else { 'preview' }); message = $(if ($alreadyAbsent) { 'The exact manifest ID was already absent; idempotent cleanup recorded the safe end state.' } elseif ($Execute) { 'Pending removal through the dependency-safe ownership boundary.' } else { 'Would remove this exact manifest-recorded target.' }) })
}
$state.updatedAt = (Get-Date).ToUniversalTime().ToString('o')
$state | ConvertTo-Json -Depth 30 | Set-Content -LiteralPath $Manifest -Encoding utf8
# CHECKPOINT LAB21-CP01 END

$cleanupFailure = $null
if ($Execute) {
    $PSNativeCommandUseErrorActionPreference = $true
    $state.status = 'cleanup-in-progress'
    $state.updatedAt = (Get-Date).ToUniversalTime().ToString('o')
    $state | ConvertTo-Json -Depth 30 | Set-Content -LiteralPath $Manifest -Encoding utf8

    try {
        # Restore shared or tenant-wide settings before removing the disposable
        # objects that were used to exercise them.
        foreach ($setting in @($state.originalSettings)) {
            if ($setting.targetId -eq 'authorizationPolicy' -and $setting.property -eq 'allowedToUseSSPR') {
                $restoreBody = @{ allowedToUseSSPR = [bool]$setting.value } | ConvertTo-Json -Compress
                az rest --method patch --url 'https://graph.microsoft.com/v1.0/policies/authorizationPolicy' --headers 'Content-Type=application/json' --body $restoreBody --output none
                $restored = az rest --method get --url 'https://graph.microsoft.com/v1.0/policies/authorizationPolicy?$select=allowedToUseSSPR' --query allowedToUseSSPR --output tsv
                if ([bool]::Parse([string]$restored) -ne [bool]$setting.value) {
                    throw 'Cleanup stopped because authorizationPolicy.allowedToUseSSPR was not restored.'
                }
            }
        }
        foreach ($object in @($active | Where-Object { $_.type -eq 'Microsoft.Management/managementGroups' -and -not $absentIds.Contains([string]$_.id) })) {
            az account management-group delete --name $object.name --output none
        }
        $hasLiveManagedTarget = @($active | Where-Object { -not $absentIds.Contains([string]$_.id) }).Count -gt 0
        if ($hasLiveManagedTarget) {
if ($connectionMonitorId -and $connectionMonitorName) {
    $probePreference = $PSNativeCommandUseErrorActionPreference
    $PSNativeCommandUseErrorActionPreference = $false
    try {
        $liveMonitorId = az network watcher connection-monitor show --location $Location --name $connectionMonitorName --query id --output tsv 2>$null
        if ($LASTEXITCODE -eq 0) {
            if (-not [string]::Equals($liveMonitorId, $connectionMonitorId, [StringComparison]::OrdinalIgnoreCase)) { throw 'Connection Monitor cleanup refused because its exact ID changed.' }
            $PSNativeCommandUseErrorActionPreference = $true
            az network watcher connection-monitor delete --location $Location --name $connectionMonitorName --output none
        }
    } finally {
        $PSNativeCommandUseErrorActionPreference = $probePreference
    }
}
        }
        foreach ($groupToDelete in @($resourceGroupObjects | Where-Object { -not $absentIds.Contains([string]$_.id) })) {
            az group delete --ids $groupToDelete.id --yes
        }
        foreach ($object in @($active | Where-Object { $_.kind -eq 'entra-object' -and -not $absentIds.Contains([string]$_.id) })) {
            $collection = switch ([string]$object.type) {
                'Microsoft.Graph/user' { 'users' }
                'Microsoft.Graph/group' { 'groups' }
                default { 'directoryObjects' }
            }
            az rest --method delete --url "https://graph.microsoft.com/v1.0/$collection/$($object.id)" --output none 2>$null
        }
        foreach ($object in $state.managedObjects) {
            $object.lifecycleStatus = 'deleted'
        }
        foreach ($action in $actions) {
            if ($action.status -eq 'skipped') { continue }
            $action.status = 'deleted'
            $action.message = 'The exact manifest-recorded target was removed through its ownership boundary.'
        }
    } catch {
        $cleanupFailure = "Cleanup mutation failed: $($_.Exception.GetType().Name)"
        foreach ($action in $actions) {
            if ($action.status -eq 'preview') {
                $action.status = 'failed'
                $action.message = $cleanupFailure
            }
        }
    }
}

$PSNativeCommandUseErrorActionPreference = $false
$remaining = [System.Collections.Generic.List[string]]::new()
foreach ($object in $state.managedObjects) {
    if (-not $Execute -and $object.lifecycleStatus -eq 'active') { $remaining.Add([string]$object.id); continue }
    if ($Execute -and $object.type -eq 'Microsoft.Management/managementGroups') {
        $null = az account management-group show --name $object.name --output none 2>$null
        if ($LASTEXITCODE -eq 0) { $remaining.Add([string]$object.id) }
    } elseif ($Execute -and $object.type -eq 'Microsoft.Resources/resourceGroups') {
        $groupExists = az group exists --subscription $SubscriptionId --name $object.name --output tsv 2>$null
        if ($LASTEXITCODE -ne 0 -or $groupExists -ne 'false') { $remaining.Add([string]$object.id) }
    } elseif ($Execute -and $object.kind -eq 'azure-resource') {
        $null = az resource show --ids $object.id --output none 2>$null
        if ($LASTEXITCODE -eq 0) {
            $remaining.Add([string]$object.id)
        } elseif ($object.type -eq 'Microsoft.RecoveryServices/vaults' -and $false) {
            $deletedVault = az backup deleted-vault get --location $Location --name $object.name --output json 2>$null
            if ($LASTEXITCODE -eq 0 -and $deletedVault) {
                $object.lifecycleStatus = 'soft-deleted'
                foreach ($action in @($actions | Where-Object { [string]$_.targetId -eq [string]$object.id -and $_.status -ne 'skipped' })) {
                    $action.status = 'soft-deleted'
                    $action.message = 'The active ARM vault is absent and the exact recoverable deleted-vault record was confirmed; no purge was performed.'
                }
            } else {
                $remaining.Add([string]$object.id)
            }
        }
    } elseif ($Execute -and $object.kind -eq 'entra-object') {
        $null = az rest --method get --url "https://graph.microsoft.com/v1.0/directoryObjects/$($object.id)" --output none 2>$null
        if ($LASTEXITCODE -eq 0) { $remaining.Add([string]$object.id) }
    }
}
$residualChecks.Clear()
$checkpointRemaining = @($remaining | Where-Object { $_ -in @($state.managedObjects | Where-Object { $_.checkpointId -eq 'LAB21-CP01' } | ForEach-Object id) }); $residualChecks.Add(@{ id = 'cp01.residual'; command = 'query manifest-recorded IDs for LAB21-CP01'; expected = 'zero active objects for LAB21-CP01'; actual = "active=$($checkpointRemaining.Count)"; status = $(if ($Execute -and $checkpointRemaining.Count -eq 0) { 'pass' } elseif (-not $Execute) { 'skipped' } else { 'fail' }) })
$checkpointRemaining = @($remaining | Where-Object { $_ -in @($state.managedObjects | Where-Object { $_.checkpointId -eq 'LAB21-CP02' } | ForEach-Object id) }); $residualChecks.Add(@{ id = 'cp02.residual'; command = 'query manifest-recorded IDs for LAB21-CP02'; expected = 'zero active objects for LAB21-CP02'; actual = "active=$($checkpointRemaining.Count)"; status = $(if ($Execute -and $checkpointRemaining.Count -eq 0) { 'pass' } elseif (-not $Execute) { 'skipped' } else { 'fail' }) })
$checkpointRemaining = @($remaining | Where-Object { $_ -in @($state.managedObjects | Where-Object { $_.checkpointId -eq 'LAB21-CP03' } | ForEach-Object id) }); $residualChecks.Add(@{ id = 'cp03.residual'; command = 'query manifest-recorded IDs for LAB21-CP03'; expected = 'zero active objects for LAB21-CP03'; actual = "active=$($checkpointRemaining.Count)"; status = $(if ($Execute -and $checkpointRemaining.Count -eq 0) { 'pass' } elseif (-not $Execute) { 'skipped' } else { 'fail' }) })
$checkpointRemaining = @($remaining | Where-Object { $_ -in @($state.managedObjects | Where-Object { $_.checkpointId -eq 'LAB21-CP04' } | ForEach-Object id) }); $residualChecks.Add(@{ id = 'cp04.residual'; command = 'query manifest-recorded IDs for LAB21-CP04'; expected = 'zero active objects for LAB21-CP04'; actual = "active=$($checkpointRemaining.Count)"; status = $(if ($Execute -and $checkpointRemaining.Count -eq 0) { 'pass' } elseif (-not $Execute) { 'skipped' } else { 'fail' }) })
$checkpointRemaining = @($remaining | Where-Object { $_ -in @($state.managedObjects | Where-Object { $_.checkpointId -eq 'LAB21-CP05' } | ForEach-Object id) }); $residualChecks.Add(@{ id = 'cp05.residual'; command = 'query manifest-recorded IDs for LAB21-CP05'; expected = 'zero active objects for LAB21-CP05'; actual = "active=$($checkpointRemaining.Count)"; status = $(if ($Execute -and $checkpointRemaining.Count -eq 0) { 'pass' } elseif (-not $Execute) { 'skipped' } else { 'fail' }) })
if ($Execute -and $remaining.Count -gt 0) {
    foreach ($object in $state.managedObjects) {
        if ([string]$object.id -in @($remaining)) { $object.lifecycleStatus = 'active' }
    }
    foreach ($action in $actions) {
        if ([string]$action.targetId -in @($remaining) -and $action.status -ne 'skipped') {
            $action.status = 'failed'
            $action.message = 'Residual query found the target still active after cleanup.'
        }
    }
}
$result = if (-not $Execute) { 'preview' } elseif ($cleanupFailure -or $remaining.Count -gt 0) { 'fail' } else { 'pass' }
$cleanup = @{
    schemaVersion = '1.0.0'
    labId = 'LAB-21'
    runId = $RunId
    generatedAt = (Get-Date).ToUniversalTime().ToString('o')
    executionMode = $(if ($Execute) { 'execute' } else { 'preview' })
    result = $result
    ownershipVerified = $ownershipVerified
    actions = @($actions)
    residualChecks = @($residualChecks)
    activeManagedObjects = @($remaining)
    retainedItems = @(

    )
}
$cleanup | ConvertTo-Json -Depth 30 | Set-Content -LiteralPath $CleanupPath -Encoding utf8
if ($Execute) {
    if ($result -eq 'pass') { $state.status = 'cleaned' }
    $state.updatedAt = (Get-Date).ToUniversalTime().ToString('o')
    $state | ConvertTo-Json -Depth 30 | Set-Content -LiteralPath $Manifest -Encoding utf8
}
Write-Host "Cleanup result: $result"
Write-Host "Artifact: $CleanupPath"
Write-Host 'Irreversible purge is never automated; retained or soft-deleted items must be documented explicitly.'
if ($result -eq 'fail') { exit 1 }
# END GENERATED AZ104 V2

Previous: Lab 20 · Catalog · Next: Lab 22