Scenario - Automatic storage management with Autopilot

In this scenario, we will configure Autopilot to grow a PVC when our virtual disk runs low on free space.

Autopilot is a rule-based engine that responds to changes from a monitoring source. Autopilot allows you to specify monitoring conditions along with actions it should take when those conditions occur.

With Autopilot, your cluster can react dynamically without your intervention to events such as:

  • Resizing PVCs when it is running out of capacity

  • Scaling Portworx storage pools to accommodate increasing usage

  • Rebalancing volumes across Portworx storage pools when they come unbalanced

In this scenario, we will use Autopilot to detect when a VM’s disk is filling up, and automatically expand that disk.

Configure our Autopilot Rule

Task 1: Create a new VM for Autopilot

Let’s create a new VM to test out Autopilot. This VM will have an additional data disk. We will mount this disk and start filling it up with space. We will also use the secret holding the public SSH key that we previously created.

cat << EOF | oc apply -f -
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: centos-stream9-autopilot
  namespace: vmtest
spec:
  dataVolumeTemplates:
  - metadata:
      name: centos-stream9-autopilot-ds-centos-stream9
      annotations:
        cdi.kubevirt.io/storage.usePopulator: "false"
    spec:
      sourceRef:
        kind: DataSource
        name: centos-stream9
        namespace: openshift-virtualization-os-images
      storage:
        storageClassName: px-csi-vm
        resources: {}
  - metadata:
      name: centos-stream9-autopilot-data-disk
      annotations:
        cdi.kubevirt.io/storage.usePopulator: "false"
    spec:
      preallocation: false
      source:
        blank: {}
      storage:
        resources:
          requests:
            storage: 5Gi
        storageClassName: px-csi-vm
  instancetype:
    name: u1.medium
  preference:
    name: centos.stream9
  runStrategy: Always
  template:
    spec:
      domain:
        devices: {}
        resources: {}
      terminationGracePeriodSeconds: 180
      volumes:
      - name: centos-stream9-autopilot-ds-centos-stream9
        dataVolume:
          name: centos-stream9-autopilot-ds-centos-stream9
      - name: centos-stream9-autopilot-data-disk
        dataVolume:
          name: centos-stream9-autopilot-data-disk
      - name: cloudinitdisk
        cloudInitNoCloud:
          userData: |-
            #cloud-config
            user: cloud-user
      accessCredentials:
      - sshPublicKey:
          propagationMethod:
            noCloud: {}
          source:
            secret:
              secretName: authorized-keys
EOF

Task 2: Create the Autopilot Rule

Create the Autopilot rule by running the following

cat << EOF | oc apply -f -
---
apiVersion: autopilot.libopenstorage.org/v1alpha1
kind: AutopilotRule
metadata:
 name: volume-resize
spec:
  ##### selector filters the objects affected by this rule given labels
  selector:
    matchLabels:
      expand: "true"
  ##### conditions are the symptoms to evaluate. All conditions are AND'ed
  conditions:
    # volume usage should be less than 30%
    expressions:
    - key: "100 * (px_volume_usage_bytes / px_volume_capacity_bytes)"
      operator: Gt
      values:
        - "30"
    for: 5
  ##### action to perform when condition is true
  actions:
  - name: openstorage.io.action.volume/resize
    params:
      # resize volume by scalepercentage of current size
      scalepercentage: "100"
      # volume capacity should not exceed 20GiB
      maxsize: "20Gi"
EOF

The rule will:

  • Line 9: Target PVCs with the Kubernetes label expand: true

  • Lines 14-17: Monitor if capacity usage grows to or above 30%

  • Line 24: Automatically grow the volume and underlying filesystem by 50% of the current volume size if usage above 30% is detected

  • Line 26: Not grow the volume to more than 20Gi

Task 3: Label our Virtual Machine PVC

Autopilot will expand PVCs that have the expand=true label applied. We will apply that label to our virtual machine’s PVC.

oc label pvc centos-stream9-autopilot-data-disk \
  expand=true --overwrite
oc get pvc centos-stream9-autopilot-data-disk

Take note of the size of our PVC!

Task 4: Format and mount our data disk

Because this is a new VM, let’s format our data disk and mount it to /data

# Wait for the VM to boot
until virtctl ssh cloud-user@centos-stream9-autopilot -i ~/.ssh/id_rsa -t "-o StrictHostKeyChecking=no" -c 'lsblk'; do
    echo "waiting for VM to boot"
    sleep 10
done

# Set up the filesystem and mount the disk as /data
virtctl ssh cloud-user@centos-stream9-autopilot \
  -i ~/.ssh/id_rsa \
  -t "-o StrictHostKeyChecking=no" \
  -c '(echo g; echo n; echo 1; echo ; echo ; echo w) | sudo fdisk /dev/vdb && sudo mkfs.ext4 /dev/vdb1 && sudo mkdir /data && sudo mount /dev/vdb1 /data'

Task 5: Add some storage space

We will use the shred command to add some storage space to our virtual machine.

We could of course log in to our VM though the console, but that would require that we log in to the virtual machine with the supplied password.

One of the advantages of an extensible framework like OpenShift is that much of the information about our environment is stored as metadata.

Task 6: Start filling the disk

Let’s execute a command to write data to the /data disk inside of our virtual machine:

virtctl ssh cloud-user@centos-stream9-autopilot \
  -i ~/.ssh/id_rsa \
  -t "-o StrictHostKeyChecking=no" \
  -c 'sudo touch /data/file; sudo shred -n 1 -s 4G /data/file'

Task 5: Observe the Portworx Autopilot events

Run the following command to observe the state changes for Portworx Autopilot:

watch oc get events --field-selector \
  involvedObject.kind=AutopilotRule,involvedObject.name=volume-resize \
  --all-namespaces --sort-by .lastTimestamp -o custom-columns=MESSAGE:.message

You will see Portworx Autopilot move through the following states as it monitors volumes and takes actions defined in Portworx Autopilot rules:

  • Initializing: Detected a volume to monitor via applied rule conditions

  • Normal: Volume is within defined conditions and no action is necessary

  • Triggered: Volume is no longer within defined conditions and action is necessary

  • ActiveActionsPending: Corrective action is necessary but not executed yet

  • ActiveActionsInProgress: Corrective action is under execution

  • ActiveActionsTaken: Corrective action is complete

Once you see ActiveActionsTaken in the event output, press CTRL+C to exit the watch command.

Task 6: Verify the Volume Expansion

Now let’s take a look at our PVC - note the automatic expansion of the volume occurred with no human interaction and no application interruption:

oc get pvc
You should now see the data volume size has now increased by 100%.

Let’s expand the virtual machine’s filesystem:

virtctl ssh cloud-user@centos-stream9-autopilot \
  -i ~/.ssh/id_rsa -t "-o StrictHostKeyChecking=no" \
  -c 'yes Fix | sudo parted --script --fix /dev/vdb print ; yes | sudo parted ---pretend-input-tty /dev/vdb resizepart 1 100% ; sudo resize2fs /dev/vdb1'

We can now observe the freespace in our virtual machine by running:

virtctl ssh cloud-user@centos-stream9-autopilot -i ~/.ssh/id_rsa -t "-o StrictHostKeyChecking=no" -c 'df -h'

Notice the size of the data disk mounted at /data

You’ve just configured Portworx Autopilot and observed how it can perform automated capacity management based on rules you configure, and be able to ``right-size'' your underlying persistent storage as it is needed!

Useful links:

https://docs.portworx.com/portworx-enterprise/operations/operate-kubernetes/storage-operations/manage-kubevirt-vms.html

https://docs.openshift.com/dedicated/virt/virtual_machines/virtual_disks/virt-expanding-vm-disks.html

https://kubevirt.io/user-guide/storage/disks_and_volumes/#disk-expansion