Resize disk

🌇 Sunset Kubernetes deployments
This page covers our PostHog Kubernetes deployment, which we sunset and no longer support. We will continue to provide security updates for Kubernetes deployments until at least May 31, 2024.
For existing customers
We highly recommend migrating to PostHog Cloud (US or EU). Take a look at this guide for more information on the migration process.
Looking to continue self-hosting?
We still maintain our Open-source Docker Compose deployment. Instructions for deploying can be found here.

Requirements

You need to run a Kubernetes cluster with the Volume Expansion feature enabled. This feature is supported on the majority of volume types since Kubernetes version >= 1.11 (see docs).

Details

PersistentVolumes can be configured to be expandable. This feature when set to true, allows the users to resize the volume by editing the corresponding PersistentVolumeClaims object.

This can become useful in case your storage usage grows and you want to resize the disk on-the-fly without having to resync data across PVCs.

To verify if your storage class allows volume expansion you can run:

Terminal

kubectl get storageclass -o json | jq '.items[].allowVolumeExpansion'
true

In case it returns false, you can enable volume expansion capabilities for your storage class by running:

Terminal

DEFAULT_STORAGE_CLASS=$(kubectl get storageclass -o=jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")].metadata.name}')
kubectl patch storageclass "$DEFAULT_STORAGE_CLASS" -p '{"allowVolumeExpansion": true}'
storageclass.storage.k8s.io/gp2 patched

N.B:
expanding a persistent volume is a time consuming operation
some platforms have a per-volume quota of one modification every 6 hours
not all the volume types support this feature. Please take a look at the official docs for more info

How-to

Connect to the Clickhouse container to verify the data directory filesystem size (in this example 20G)

Terminal

kubectl -n posthog exec -it chi-posthog-posthog-0-0-0 -- /bin/bash
clickhouse@chi-posthog-posthog-0-0-0:/$ df -h /var/lib/clickhouse/
Filesystem                                                                Size  Used Avail Use% Mounted on
/dev/disk/by-id/scsi-0DO_Volume_pvc-f39035c1-c68c-4572-81f2-273de6eb088c   20G   50M   19G   1% /var/lib/clickhouse

In your Helm chart configuration, update the clickhouse.persistence.size value in value.yaml to the target size (40G in this example)
Run a helm upgrade

Connect to the ClickHouse container to verify the new filesystem size

Terminal

kubectl -n posthog exec -it chi-posthog-posthog-0-0-0 -- /bin/bash
clickhouse@chi-posthog-posthog-0-0-0:/$ df -h /var/lib/clickhouse/
Filesystem                                                                Size  Used Avail Use% Mounted on
/dev/disk/by-id/scsi-0DO_Volume_pvc-f39035c1-c68c-4572-81f2-273de6eb088c   40G  186M   38G   1% /var/lib/clickhouse

Troubleshooting

If the resize didn't work check for errors in the ClickHouse operator pod.

This procedure doesn't work to decrease a volume. If you try, the disk won't be resized and the following errors can be seen in the ClickHouse operator pod

posthog/posthog:ERROR unable to reconcile PVC(posthog/data-volumeclaim-template-chi-posthog-posthog-0-0-0) err: PersistentVolumeClaim "data-volumeclaim-template-chi-posthog-posthog-0-0-0" is in valid: spec.resources.requests.storage: Forbidden: field can not be less than previous value

For increases double check what the storage class is that's used for ClickHouse and make sure it's expandable as mentioned in requirements above.

Terminal

kubectl get pvc data-volumeclaim-template-chi-posthog-posthog-0-0-0 -o json | jq '.spec.storageClassName'

If a storage class with that name doesn't exist (kubectl get storageclass), then it's probably because major upgrade steps from 9.x.x weren't completed. Set installCustomStorageClass: true & run a helm upgrade first (without the storage increase request).

Requirements

How-to

Troubleshooting

Questions?

Was this page useful?

Restore