Prompts to help you get started

I need to change the VM SKU of one of our AKS clusters (for example from Standard_D4_V3 to a larger SKU). Which repositories and files do I edit, and what is the safe procedure given most node pool settings are immutable once deployed? Cover: updating `aksVmSku` and `aksNodePoolName` in the environment config under Deployment.Pipelines/Environments/NICE-Services/AKS (the per-environment JSON such as CPE/SKCPE.json, plus _Defaults/AKS.json), why renaming the node pool is required to change the SKU, running the `Deploy-SystemNodePool-Only` deployment action from the AKS.yml pipeline, migrating workloads, and manually deleting the old node pool afterwards.

A pod on one of our AKS clusters is stuck Pending. Diagnose it against our actual configuration: the system node pool carries the `CriticalAddonsOnly=true:NoSchedule` taint, so an application pod without a matching toleration (or without a user node pool to land on) will never schedule there. Also consider the cluster-autoscaler bounds (`aksNodePoolMin` 3 / `aksNodePoolMax` 5), the 110-pods-per-node ceiling, resource requests/limits, and the `ENV_ALLOWED_VM_SKUS` constraint. Give me a prioritized, config-specific troubleshooting plan with the kubectl checks to run.

Explain the deployment and runbook actions exposed by the AKS pipeline (Deployment.Pipelines/Pipelines/NICE-Services/AKS.yml, which drives Platform.Service.AKS/.azure/aks-deploy-v1.yml). For each — Full-Deployment, Deploy-Infrastructure-Only, Deploy-CoreServices-Only, Deploy-SystemNodePool-Only, and the runbook actions Update-AKS-Cluster-Version, Update-Monitoring-And-Logging, Rotate-SSH-Keys — tell me what it changes, when I should pick it, and the blast radius. Which one applies for a Helm-only change versus a Bicep infra change versus a node pool SKU change?

How do I change the number of nodes on one of our AKS clusters? Explain the `aksNodePoolMin` (default 3) and `aksNodePoolMax` (default 5) parameters in the environment config under Deployment.Pipelines/Environments/NICE-Services/AKS, how the cluster-autoscaler uses them, the 110-pods-per-node ceiling on the agent pool, and which deployment action applies the change. Note where a single-node R&D cluster is allowed versus production.

Explain how ingress is configured on our AKS platform using the AzureAppRoutingIngress Helm chart, and how TLS certificates from cert-manager attach to it. Cover the staging and production Let's Encrypt ClusterIssuers, the DNS-01 challenge with the azureDNS solver (backed by the cert-manager managed identity / certManagerIdentityClientId), and what I must configure when exposing a new service and requesting its certificate.

How is persistent storage handled on our AKS platform? Explain the EncryptedStorage Helm chart and the `encrypted-azure-csi` storage class it defines (used by ECK and any workload needing volumes), how a PVC binds to it, and the fact that the resulting Azure managed disks are LRS and zone-pinned — so a pod rescheduled to another zone cannot reattach its disk. What should I know before requesting encrypted storage for a new stateful workload?

Walk me through how Elasticsearch is deployed and managed via ECK on our AKS platform. Cover the Platform.Service.ElasticSearch repo: the Bicep infrastructure (main.bicep + storageAccount.bicep), the k8s/Helm/EsCluster chart whose ElasticsearchCluster.yaml defines the `nodeSets`, `storageClassName: encrypted-azure-csi`, and `volumeClaimDeletePolicy`, the ECK operator, and how release/manifests/manifest.json resolves the Elasticsearch/Kibana image and chart versions fetched into ACR.

An Elasticsearch node in our ECK deployment is unhealthy or a pod is stuck. Diagnose it against our design: PersistentVolumes are LRS and zone-pinned (ADR-003), so a pod rescheduled to a different availability zone cannot reattach its disk and will stay Pending — check the PVC/PV zone against the pod's node. Also check cluster/shard health (green/yellow/red), the quorum of master-eligible nodes, ECK operator events, and the `encrypted-azure-csi` PVCs. Give me the kubectl and Elasticsearch health checks in order and the standard recovery approach.

How do I safely scale our ECK Elasticsearch cluster? Explain the scaling strategy in ADR-004: why we do NOT use pod autoscaling for Elasticsearch (it's stateful — uncontrolled scaling triggers shard rebalancing storms and split-brain risk), the minimum of 3 nodes, and the controlled procedure of changing `nodeSets` node counts plus updating replica/shard allocation settings, then validating cluster health and shard allocation. What are the risks and pre-checks before scaling in production?

Our Elasticsearch pods need an elevated `vm.max_map_count`. Explain why this must be configured on the AKS Linux OS config from the very start — the Linux OS config cannot be changed after an AKS cluster is created — and what that means if an existing cluster wasn't provisioned for Elasticsearch. Which AKS parameter/config sets it, and what is the remediation if it was missed (rebuild versus new node pool)?

How does a change get deployed to an AKS environment? Explain the flow: the top-level pipeline Deployment.Pipelines/Pipelines/NICE-Services/AKS.yml takes an `environment` (e.g. CPE/SKCPE, Shared-CPE-Supported/DEVAKS-02) and a `deploymentaction`, pulls the matching config from Environments/NICE-Services/AKS, references Platform.Service.AKS via resources.repositories, and invokes .azure/aks-deploy-v1.yml. Where are the main validation points (PreDeployment/PostDeployment tests)?

I want to test an edit to the AKS deployment pipeline before merging. Explain the CPE workflow: create a branch in Deployment.Pipelines, create a branch in the source repo (e.g. Platform.Service.AKS), and point the `ref` in the resources.repositories block at my source branch so my change is exercised end to end. What do I run (which environment/deploymentaction, PreDeployment tests) and verify before opening a PR?

Deployment.Pipelines is a mirror of what runs in the production Azure DevOps organizations. Explain how a change is promoted from R&D to Production, why pipeline/environment edits must be manually reproduced in the production DevOps org, and the checklist an engineer should follow so an R&D-only change (e.g. a new environment JSON or pipeline parameter) doesn't silently miss production.

A pipeline that fetches third-party artifacts into ACR (as driven by ECK's release/manifests/manifest.json) has failed. Explain how the FetchThirdPartyRepo flow works, the common failure points (auth to the source registry, ACR import, version/manifest mismatch), which stages and logs to inspect, and the standard recovery approach.

How do I add a new Azure resource to the AKS platform using Bicep? Explain the module structure in Platform.Service.AKS/Bicep (main.bicep entry point plus the DNS, Network, Logging, Monitoring and SystemNodePool modules), how parameters are supplied from the environment JSON under Deployment.Pipelines/Environments, where my new resource/module should live, and which deployment action (Deploy-Infrastructure-Only) applies it.

Which parts of our AKS configuration are immutable once deployed, and how do I change each one safely? Cover the Linux OS config (e.g. vm.max_map_count for Elasticsearch, which must be set at creation), node pool settings that force a new pool (VM SKU via aksVmSku + aksNodePoolName), and the values hard-coded in the Bicep (system-assigned identity, Azure CNI Overlay, standard load balancer SKU, AzureLinux3 OS). For each, tell me the correct change path.

How is observability configured on our AKS platform? Explain how the AmaMonitoring (Azure Monitor Agent) Helm chart, the Bicep Logging module (Log Analytics workspace) and Bicep/Monitoring/dcrRuleassociation.bicep (Data Collection Rule associations) fit together to collect container logs and metrics. Note the historical issue where the Logging DCR failed to send container logs to Log Analytics, and how to verify the DCR association is working.

I need to update only the monitoring, logging and alert configuration on an existing AKS cluster. Explain the `Update-Monitoring-And-Logging` runbook action in the AKS pipeline — what it re-applies (AMA, DCRs, Log Analytics, alerts), why it's safer than a Full-Deployment for this change, and how to validate logs and alerts afterwards.

An alert has fired on the CPE platform. Explain how to investigate it: which Log Analytics tables and KQL queries to run for AKS container logs, which dashboards under Platform.Service.AKS/Monitoring/Dashboards to check, and which runbooks apply — based on our monitoring configuration (AMA + DCRs) and operational guidance.

How do workloads on our AKS platform authenticate to Azure resources? Explain the use of user-assigned Managed Identity / Workload Identity — for example the cert-manager identity (certManagerIdentityClientId) passed into its Helm release for the DNS-01 azureDNS solver — how a service is granted access, and what I must configure for a new workload that needs to read from Key Vault or storage.

How are TLS certificates managed on our AKS platform? Explain how cert-manager is configured with both staging and production Let's Encrypt ClusterIssuers, the DNS-01 challenge using the azureDNS solver backed by a managed identity, how certificates are issued and auto-renewed and attach to ingress, and what to check (ClusterIssuer, Certificate, privateKeySecretRef, DNS-01 records) when a certificate fails to issue.

How are secrets delivered to workloads on our AKS platform? Explain how Azure Key Vault is used, how secrets reach pods via managed identity (rather than being hard-coded in Helm values), and the standard pattern I should follow when a new service needs a secret. What are the do's and don'ts?

What are the key architectural decisions behind the CPE platform, and why were they made? Summarize the relevant ADRs in Platform.Deployment.Documentation (ACR, Azure DevOps, containerisation, deployment automation, infrastructure as code, landing zones, release process, observability) and explain the reasoning and trade-offs.

What is the CPE repository naming convention? Explain the standard described in RepoNaming (e.g. Platform.Service.*, Platform.Deployment.*, Deployment.Pipelines), what each segment means, and how I should name a new repository for a new platform service.

I'm new to the CPE platform team. Give me a structured onboarding path: the key repositories (Platform.Service.AKS, Platform.Service.ElasticSearch, Deployment.Pipelines, Platform.Deployment.Documentation) and what each does, the deployment flow, how monitoring works, and the operational knowledge I should understand in my first two weeks.

Give me a map of the CPE repositories: which repo owns AKS, which owns Elasticsearch/ECK, which owns the deployment pipelines, and which holds architecture and platform documentation. For each, summarize its structure and what I'd go there to change.

A deployment to an AKS cluster failed. Walk me through diagnosing it: identify which stage of Deployment.Pipelines/Pipelines/NICE-Services/AKS.yml → .azure/aks-deploy-v1.yml failed, whether it's a Bicep infrastructure error (Deploy-Infrastructure-Only stage) or a Helm/core-services error (Deploy-CoreServices-Only), the Kubernetes events and pod logs to inspect, whether PreDeployment/PostDeployment tests caught it, and the standard rollback/recovery steps.

We have an active incident on the CPE platform. Help me run a structured investigation: correlate signals across AKS (node pools, taints, autoscaler), Elasticsearch/ECK (shard health, zone-pinned PVs), ingress and networking; identify the likely blast radius; point me to the relevant runbooks; and outline what to capture for the RCA. Have we seen a similar issue before?

What is the standard procedure to deploy or upgrade an AKS cluster on our platform? Explain the inputs to Deployment.Pipelines/Pipelines/NICE-Services/AKS.yml — choosing the `environment` (e.g. CPE/SKCPE), the `deploymentaction` (Full-Deployment vs Deploy-Infrastructure-Only vs Deploy-CoreServices-Only), and enabling PreDeployment/PostDeployment tests — plus the AKS-Deployment / AKS-Configuration operational docs and how to validate the cluster is healthy afterwards.

How do I upgrade the Kubernetes version of one of our AKS clusters? Explain the `Update-AKS-Cluster-Version` runbook action — it updates the control plane, the system node pool, and all user node pools including the ones ECK deploys onto — how `kubernetesVersion` is controlled from outside the pipeline, the ordering, and how to validate workloads (especially ECK) after the upgrade.

I need to plan networking for a new AKS environment. Based on our AKS-VnetCreation guidance and the fact that we use the Azure CNI Overlay network plugin, explain how to size the VNet and subnets, the IP constraints I must respect (authorizedIps for the API server management plane versus the data plane), and how this ties into the Bicep/Network module and NSG rules.