Shifting from Stateless to Stateful on DigitalOcean Kubernetes (DOKS)

When developers first dive into Kubernetes, they usually begin with stateless applications—web servers, APIs, or microservices that process incoming requests but don't care about memory once a container cycles. But what happens when your application must remember things? What happens when a database pod crashes or updates?

In this hands-on project, I built an Ephemeral Chat Application using Flask and Redis on DigitalOcean Kubernetes (DOKS) to explicitly test and analyze the massive architectural divide between Stateless and Stateful cluster configurations.

By cloning my repository, you can reproduce this exact DevOps experiment and watch the transition happen live in your browser.

🏗 The Project Architecture :

Our deployment environment maps out into two critical structural layers:

1. The Web Layer (Stateless): A Python Flask application scaled across multiple replicas to handle high availability. It processes incoming user traffic via an external cloud load balancer.

2. The Database Layer (Stateful): A Redis instance used to store and stream chat logs using Redis Lists.

🛠 Prerequisites :

Before running any deployment manifests, ensure you have the following prerequisites configured on your local machine:

• A DigitalOcean Account with an active Kubernetes (DOKS), Container Registry (DOCR) cluster provisioned.

• doctl CLI installed and authenticated to your DigitalOcean account.

• kubectl CLI installed locally for cluster orchestration.

• Docker Desktop or Docker Engine running locally.

📥 Step 1: Clone the Repository & Configure Context

Start by cloning the codebase directly to your local workspace:

git clone https://github.com/vishalparit10/doks-persistent-chat
cd doks-persistent-chat

Verify the active connection:

kubectl get nodes

Ensure you see your active DigitalOcean worker droplets listed before proceeding.

🧪 Step 2: Phase 1 — Deploying and testing "Stateless"

Behavior.

To understand the core problem that persistent storage solves, we must first watch a stateless system experience an unexpected failure.

1. Review the Stateless redis.yaml Configuration

In its initial format, the Redis container has no structural storage parameters attached to it. It stores data exclusively in volatile container memory:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:alpine
        ports:
        - containerPort: 6379
---
apiVersion: v1
kind: Service
metadata:
  name: redis-service
spec:
  selector:
    app: redis
  ports:
  - protocol: TCP
    port: 6379
    targetPort: 6379

2. Apply the Manifests to the Cluster

Execute the initial cluster configurations to deploy the application layer, the baseline Redis layer, and the external network routes:

kubectl apply -f redis.yaml
kubectl apply -f app.yaml

3. Fetch the External Load Balancer Route

DigitalOcean will provision a dedicated physical cloud load balancer. Track its provisioning state:

kubectl get svc chat-app-lb -w

Once the EXTERNAL-IP moves from <pending> to an active IP address, copy it and paste it directly into your web browser.

4. Run the Stateless Disruption Test

1. Type 3 or 4 messages into the chat interface (e.g., "Testing stateless cluster memory").

2. Open a separate browser tab to the same IP. Notice how the chat history syncs perfectly because both frontend pods point to the same internal redis-service endpoint.

Simulate an infrastructure crash in your terminal by killing the active Redis pod:

kubectl delete pod -l app=redis

1. Wait roughly 10 seconds for Kubernetes to spawn a replacement container, then refresh both of your active browser windows.

❌ The Stateless Result: The chat history vanishes completely. Because the container layer was destroyed, the data living in its RAM evaporated.

💾 Step 3: Phase 2 — Shifting the Architecture to "Stateful"

To ensure our database layer survives infrastructure lifecycle rotations, we must shift our architecture to use persistent storage.

1. Provision Cloud Storage Assets:

Create a new manifest file named redis-pvc.yaml to register a Persistent Volume Claim. This prompts DigitalOcean's native Storage Interface to spin up an independent 1GB cloud SSD:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: do-block-storage

Apply the claim profile to your environment:

kubectl apply -f redis-pvc.yaml

2. Refactor redis.yaml with Volume Mounts

Now, we must modify our redis.yaml file to pull that physical SSD into our deployment framework. Update your redis.yaml with three modifications:

• Add the command flag to force Redis to stream logs directly to disk (appendonly yes).

• Append volumeMounts within the container definition to assign the mounting path (/data).

• Declare a global pod volumes block linking the mount target back to your DigitalOcean PVC.

Here is the updated manifest file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:alpine
        ports:
        - containerPort: 6379
        # 1. Force Redis to record transactions directly to disk
        command: ["redis-server", "--appendonly", "yes"] 
        # 2. Attach the volume path inside the container environment
        volumeMounts:
        - name: redis-data
          mountPath: /data
      # 3. Mount the physical DigitalOcean PVC to the execution space
      volumes:
      - name: redis-data
        persistentVolumeClaim:
          claimName: redis-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: redis-service
spec:
  selector:
    app: redis
  ports:
  - protocol: TCP
    port: 6379
    targetPort: 6379

3. Apply the Structural Updates

Push the refactored code updates to your cluster. Kubernetes will perform an automated rolling update, safely detaching the stateless pod and bringing up your stateful configuration:

kubectl apply -f redis.yaml

4. Run the Stateful Disruption Test

1. Return to your web browser and submit fresh text logs into the app (e.g., "This message is permanent!").

2. Execute the crash simulation sequence again:

kubectl delete pod -l app=redis

1. While the replacement pod spins up, track the internal recovery lifecycle metrics directly from your container logs:

#Get the newly generated pod name
kubectl get pods

#Pull the system lifecycle logs
kubectl logs

You will explicitly see Redis locate the historical transaction file on your cloud block storage volume and load it into its system memory on bootup: * DB loaded from append only file: ...

4.Refresh your browser interface.

The Stateful Result: Your messages remain perfectly intact. Even though the underlying runtime environment was torn down, Kubernetes retained the storage assets, re-mapped the block volume to the new container, and preserved your data.

🧹 Cleaning Up Cloud Resources

Because managed public cloud elements run on a metered usage model, remember to tear down your staging configurations when your testing sequences conclude to avoid unexpected ongoing charges for the Load Balancer or the underlying Storage block infrastructure:

kubectl delete -f app.yaml
kubectl delete -f redis.yaml
kubectl delete -f redis-pvc.yaml

🎯 Wrap-Up

Shifting from an exclusively stateless design framework to managing persistent storage topologies marks a significant milestone in any cloud engineering path. By completely decoupling your execution tasks (the transient pods) from your structural state (DigitalOcean Block Storage via PVC definitions), you build production-ready systems engineered to withstand unpredictable physical node drops.

The configuration manifests and underlying code blocks are available on my GitHub repo: github.com/vishalparit10/doks-persistent-chat