Kubernetes: from a pod to an operated service

1. Declare a desired state

A pod is the scheduling unit containing one or more containers. A Deployment manages replicated pods through a ReplicaSet. You declare the desired number of replicas; controllers work to reconcile actual state. A namespace groups names and provides a scope for policy.

kubectl config current-context
kubectl get nodes
kubectl create namespace auwen-lab
kubectl -n auwen-lab create deployment web --image=nginx:stable
kubectl -n auwen-lab rollout status deployment/web
kubectl -n auwen-lab get pods

What to expect

A node is Ready and a web pod becomes Running/Ready. Image pulls and scheduling can take time.

Its controller knows the desired replica count and creates replacement pods. The replacement has a different identity; applications must tolerate that.

Watch for: A namespace is not a complete security boundary. Cluster context mistakes can send valid commands to the wrong environment.

2. Reach a changing set of pods

A Service gives a selected set of pods a stable networking abstraction. Pod addresses can change. Here ClusterIP stays inside the cluster; port-forward provides a temporary local development connection.

kubectl -n auwen-lab expose deployment web --port=80 --target-port=80
kubectl -n auwen-lab get service web
kubectl -n auwen-lab port-forward service/web 8084:80

What to expect

http://localhost:8084 displays nginx while port-forward runs. Ctrl+C ends the forwarding process, not the Deployment.

The Service created by expose selects the Deployment’s app=web label. A selector mismatch leaves the Service with no matching ready backends.

Watch for: Port-forward is a local debugging tool, not a production ingress design.

3. Observe recovery and scaling

Increasing replicas changes desired capacity. Deleting one pod demonstrates reconciliation. Use a pod name you just inspected; do not delete the Deployment, which would remove the controller and its desired state.

kubectl -n auwen-lab scale deployment web --replicas=2
kubectl -n auwen-lab get pods
# Replace POD_NAME with one web pod from the output:
kubectl -n auwen-lab delete pod POD_NAME
kubectl -n auwen-lab get pods -w

What to expect

The controller creates a replacement and returns toward two ready replicas. Ctrl+C stops watching.

Both replicas share one node and its failure domain. Node failure can take down both. Availability also depends on scheduling, dependencies, networking, and capacity.

Watch for: A replica count is not proof of resilience. Test failure scenarios and application readiness.

4. Diagnose before restarting

Status, events, and logs answer different questions. Pending often points to scheduling or storage. ImagePullBackOff points to an image retrieval problem. CrashLoopBackOff means the process keeps exiting and restarting. Look at the preceding evidence before changing anything.

kubectl -n auwen-lab get pods
kubectl -n auwen-lab describe deployment web
kubectl -n auwen-lab get events --sort-by=.metadata.creationTimestamp
kubectl -n auwen-lab logs deployment/web
# Inspect a specific pod with describe pod POD_NAME

What to expect

Record observed state and a cause hypothesis. The healthy lab should show available replicas and request/startup logs.

Describe the affected pod and inspect its events for image pull errors. Cleanup: kubectl delete namespace auwen-lab. This deletes resources within that namespace, so verify the name first.

Watch for: Do not paste secrets from logs into public issue reports. A restart can erase useful evidence without solving the cause.

5. Keep desired state in a file you can review

Before this lesson: Complete Kubernetes objects, Service, reconciliation and diagnosis (lessons 1–4). Use a dedicated local minikube or equivalent learning cluster, not a work cluster. Outcome: explain and apply a Namespace, Deployment and Service from one file and make a repeatable replica change.

Imperative commands are useful for discovery, but remembering every command is a poor configuration record. A manifest states desired object fields in YAML. apiVersion identifies the API schema; kind identifies the object type; metadata gives identity; spec describes desired behavior. The separator --- allows several YAML documents in one file. This file is not a script: document order alone is not a general application dependency mechanism.

The Deployment's selector identifies the pods it manages. Its pod-template labels match that selector. The Service independently selects matching pod labels; it is not attached to a Deployment by name. A named container port http is resolved by the Service's targetPort. containerPort documents a port; it does not make an application listen or publish a host port by itself.

Save notes.yaml below. This creates a new namespace, auwen-next-lab, so the earlier auwen-lab exercises remain separate. nginx:stable is a mutable teaching tag: record the resolved image ID/digest, and use a reviewed immutable reference when reproducibility matters. Do not infer that a currently documented Kubernetes feature exists on your own server; check kubectl version and your cluster's supported APIs.

Applying a file reconciles the fields you declare. Repeating the same apply should not create duplicate objects because kind/namespace/name identify them. Editing the file preserves intent; an unrelated imperative edit may later be overwritten by that file. kubectl diff is a preview with a meaningful nonzero status when differences exist, not automatically a failed release.

apiVersion: v1
kind: Namespace
metadata:
  name: auwen-next-lab
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
  namespace: auwen-next-lab
spec:
  replicas: 2
  selector:
    matchLabels:
      app: auwen-notes
  template:
    metadata:
      labels:
        app: auwen-notes
    spec:
      containers:
        - name: web
          image: nginx:stable
          ports:
            - name: http
              containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: notes
  namespace: auwen-next-lab
spec:
  selector:
    app: auwen-notes
  ports:
    - port: 80
      targetPort: http

kubectl config current-context
kubectl version
# Stop unless this is your dedicated local learning cluster.
kubectl apply -f notes.yaml
kubectl -n auwen-next-lab rollout status deployment/notes --timeout=120s
kubectl -n auwen-next-lab get pods -l app=auwen-notes
kubectl -n auwen-next-lab get service notes
kubectl -n auwen-next-lab get endpointslices -l kubernetes.io/service-name=notes
kubectl -n auwen-next-lab port-forward service/notes 8087:80

Run it

Keep port-forward running and visit http://localhost:8087. Ctrl+C ends forwarding, not the Deployment. Apply the unchanged file again and inspect object counts. Save the manifest alongside a short record of client/server versions and the pulled image ID.

What to expect

Two pods become ready if scheduling and image retrieval succeed; the Deployment reaches its desired replicas. The Service has a cluster-internal address and EndpointSlices reflecting its selected backends. Reapplying the same file does not create a second notes Deployment. Port-forward provides a temporary local route; it is not a production ingress.

kubectl diff -f notes.yaml
kubectl apply -f notes.yaml
kubectl -n auwen-next-lab get pods -l app=auwen-notes
kubectl -n auwen-next-lab get endpointslices -l kubernetes.io/service-name=notes

# Three replicas belong to the Deployment. A wrong Service selector
# does not stop those pods, but leaves that Service without matching
# backends. Restore app: auwen-notes in the Service and reapply.
# Restore replicas: 2 for the following resource example.
# diff status 1 means differences; status greater than 1 signals error.
# Checkpoint: explain three separate roles for the Deployment selector,
# template labels and Service selector.

Watch for: Do not change an existing Deployment selector casually: it is immutable. YAML parsing alone cannot validate an API schema or admission policy. Deleting this manifest includes deleting its namespace and everything inside it, so reserve cleanup for the end of these dedicated labs. Never copy an unrelated namespace into a deletion command.

Lesson references

• Kubernetes declarative object management; checked 14 September 2026 ↗
• Kubernetes Deployments ↗
• Kubernetes Services ↗
• kubectl diff exit status ↗

6. Give startup, readiness and liveness different jobs

Before this lesson: Complete the manifest lesson and keep auwen-next-lab. Save the separate probe-notes.yaml below. Outcome: remove a pod from ready Service backends without restarting it, then distinguish that from liveness-driven restart behavior.

A startup probe protects a slow-starting application by withholding its regular liveness/readiness checks until startup succeeds. A readiness probe asks whether the container should receive ordinary Service traffic. A liveness probe asks whether restarting the container is an appropriate recovery response. Reusing one shallow check for all three can hide failures or cause restart loops.

The teaching Deployment uses nginx for HTTP and a marker file for readiness. Its shell first creates /tmp/ready, then exec replaces the shell with nginx so the server is the main process. Startup and liveness check the HTTP endpoint. Readiness tests only for the marker. This deliberately simple signal lets us make the pod unready without breaking HTTP or triggering liveness. A real readiness check must represent application readiness, not merely copy this marker trick.

The startup settings allow roughly 60 seconds of failed checks (30 at a two-second period) before a failed start is treated as a failure; exact timing includes scheduling and probe execution. Readiness has one allowed failure at a three-second period. Liveness needs three consecutive failures at a ten-second period, each bounded by its timeout. These are different operational decisions, not arbitrary identical defaults.

A not-ready pod can remain Running with an unchanged restart count. Kubernetes updates endpoint readiness so normal Service traffic avoids it. Probe effects are asynchronous; inspect pod state, events and EndpointSlice conditions rather than relying solely on the browser. Port-forward can reach a selected pod directly and is not a reliable test of Service load-balancing readiness.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: probe-notes
  namespace: auwen-next-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: auwen-probe-notes
  template:
    metadata:
      labels:
        app: auwen-probe-notes
    spec:
      containers:
        - name: web
          image: nginx:stable
          command: ["/bin/sh", "-c"]
          args:
            - "touch /tmp/ready; exec nginx -g 'daemon off;'"
          ports:
            - name: http
              containerPort: 80
          startupProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 2
            failureThreshold: 30
          readinessProbe:
            exec:
              command: ["/bin/sh", "-c", "test -f /tmp/ready"]
            periodSeconds: 3
            failureThreshold: 1
          livenessProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
---
apiVersion: v1
kind: Service
metadata:
  name: probe-notes
  namespace: auwen-next-lab
spec:
  selector:
    app: auwen-probe-notes
  ports:
    - port: 80
      targetPort: http

kubectl config current-context
kubectl apply -f probe-notes.yaml
kubectl -n auwen-next-lab rollout status deployment/probe-notes --timeout=120s
kubectl -n auwen-next-lab get pods -l app=auwen-probe-notes
# Remove only the teaching readiness marker:
kubectl -n auwen-next-lab exec deployment/probe-notes -- rm /tmp/ready
kubectl -n auwen-next-lab get pods -l app=auwen-probe-notes
kubectl -n auwen-next-lab get endpointslices -l kubernetes.io/service-name=probe-notes -o yaml
# Restore the marker:
kubectl -n auwen-next-lab exec deployment/probe-notes -- touch /tmp/ready
kubectl -n auwen-next-lab get pods -l app=auwen-probe-notes

Run it

Wait for initial readiness, record restart count, remove the marker, then wait for a readiness check before inspecting again. EndpointSlices may retain an endpoint with ready: false rather than removing the address entirely. Restore the marker and watch readiness recover. Do not restart the Deployment to conceal the cause.

What to expect

Initially one pod is Running and 1/1 Ready. Removing /tmp/ready eventually changes it to 0/1 Ready while it remains Running and its restart count stays unchanged. HTTP liveness still succeeds. Restoring the file makes it ready again after a successful check. These are predicted cluster observations; no cluster execution was available here.

livenessProbe:
  exec:
    command: ["/bin/sh", "-c", "test -f /tmp/ready"]
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

# Apply the experimental manifest, wait for its new pod, then delete
# the marker. Repeated liveness failures trigger a container restart.
# The startup shell recreates the marker; the restart count increases.
# Restarting every client may worsen an external database outage.
# Use readiness for inability to serve; use liveness only where a
# restart is a justified recovery. Restore the HTTP liveness version.
# Checkpoint: predict Ready, phase and restart count independently.

Watch for: An exec probe runs in the container and needs the requested executable. Probe failures do not all have the same effect. Overaggressive liveness can amplify overload; a huge startup allowance can hide a broken start for too long. Default Service endpoint behavior has exceptions such as publishNotReadyAddresses, not used here.

Lesson references

• Kubernetes startup, readiness and liveness probes; checked 14 September 2026 ↗
• Kubernetes EndpointSlice conditions ↗

7. Reserve capacity and recover a stalled rollout

Before this lesson: Complete manifests and probes. Keep a dedicated local cluster and the notes Deployment from lesson 5; probe-notes is independent. Save this complete replacement as notes.yaml. Outcome: calculate requested capacity, distinguish Pending from a crashing process, and recover a rollout while restoring the source file.

A resource request tells the scheduler how much capacity to account for when placing a pod. A limit bounds runtime use under the platform's enforcement rules. 100m CPU means one tenth of a CPU unit; 32Mi is 32 × 2^20 bytes. CPU pressure can cause throttling; memory-limit failures may cause termination. These are not interchangeable outcomes, and a request is not a promise of dedicated low-latency hardware.

Two replicas requesting 100m and 32Mi each request 200m CPU and 64Mi total for these containers. The rolling strategy permits one extra pod (maxSurge: 1), so an update can temporarily need 300m CPU and 96Mi of requests, plus cluster/system overhead and other workloads. maxUnavailable: 0 keeps the controller from intentionally reducing available replicas during a normal rolling update, but does not guarantee uptime against node or application failure.

A readiness probe prevents a new pod from counting as ready merely because its process started. progressDeadlineSeconds reports a stalled rollout after the configured deadline; it does not automatically roll the Deployment back. A rollout-status timeout is the command giving up waiting, not proof that Kubernetes undid the change.

Save a known-good copy of notes.yaml before experimenting. A deliberately nonexistent image tag produces a controlled pull failure. Existing ready replicas can remain while the surge pod cannot start. Inspect events and the actual image string; increasing memory will not correct a misspelled image. Recovery requires both restoring runtime state and correcting the file that the next apply will use.

apiVersion: v1
kind: Namespace
metadata:
  name: auwen-next-lab
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
  namespace: auwen-next-lab
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  progressDeadlineSeconds: 120
  selector:
    matchLabels:
      app: auwen-notes
  template:
    metadata:
      labels:
        app: auwen-notes
    spec:
      containers:
        - name: web
          image: nginx:stable
          resources:
            requests:
              cpu: "100m"
              memory: "32Mi"
            limits:
              cpu: "500m"
              memory: "128Mi"
          readinessProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 3
          ports:
            - name: http
              containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: notes
  namespace: auwen-next-lab
spec:
  selector:
    app: auwen-notes
  ports:
    - port: 80
      targetPort: http

kubectl config current-context
kubectl apply -f notes.yaml
kubectl -n auwen-next-lab rollout status deployment/notes --timeout=120s
kubectl -n auwen-next-lab get pods -l app=auwen-notes
kubectl -n auwen-next-lab rollout history deployment/notes

# Controlled bad image, local lab only:
kubectl -n auwen-next-lab set image deployment/notes web=nginx:auwen-deliberately-missing-20260914
kubectl -n auwen-next-lab rollout status deployment/notes --timeout=60s
kubectl -n auwen-next-lab describe deployment notes
kubectl -n auwen-next-lab get events --sort-by=.metadata.creationTimestamp

# Restore the unchanged known-good source file:
kubectl apply -f notes.yaml
kubectl -n auwen-next-lab rollout status deployment/notes --timeout=120s

Run it

Use a valid pulled nginx:stable image and sufficient local-cluster capacity for the baseline. Record rollout history before the bad-image change. If your registry unexpectedly contains the deliberately unusual tag, choose a verified nonexistent lab tag instead. The intended test is image retrieval failure, not running an unknown image.

What to expect

The baseline should reach two ready replicas. The bad update normally creates an unready surge pod with image-pull events and stalls; old ready pods remain when capacity and the previous workload are healthy. Reapplying the known-good file restores the intended image and eventually the desired ready replica count. Event order, pod names and timing vary. No automatic rollback is claimed.

Three steady replicas: 300m CPU, 96Mi memory.
With one surge replica: 400m CPU, 128Mi memory, plus other overhead.

An unschedulable request prevents placement: Pending with scheduling
events is expected, not a running process repeatedly crashing.
Inspect the specific pod's events before changing limits.

kubectl rollout undo can restore an earlier Deployment pod template,
but does not rewrite your local YAML or reverse database migrations.
A subsequent apply of a bad file can reintroduce the failure.
Correct the file and verify the image, replica readiness and workload.
Cleanup AFTER all Kubernetes labs:
kubectl delete namespace auwen-next-lab
This removes every resource in that dedicated namespace.

Watch for: Do not use kubectl top as your only capacity evidence: it requires a metrics service and shows observations, not scheduling requests. A readiness endpoint returning 200 is not a full application acceptance test. Mutable image tags weaken rollback reproducibility; pin reviewed digests for real releases. No workload, policy or schema validation is implied by simply parsing YAML.

Lesson references

• Kubernetes resource requests and limits; checked 14 September 2026 ↗
• Kubernetes rolling updates, progress and rollback ↗