Skip to content
Cloudbricks.dev logo

Apache CloudStack - Operations & Troubleshooting

Part 8 of the Apache CloudStack series: troubleshooting, databases, usage billing, backup, GPU, webhooks, shared filesystems, and extensions.

Kishore 7 min read

A CloudStack cloud is a distributed system: a management server, a database, message queue, system VMs, hypervisor agents, and shared storage, all of which have to agree. Most incidents are a break in one of those links. This part is about finding which one.

It also covers the operational features that turn a working cloud into one you can run: usage billing, backups, GPU support, webhooks, shared filesystems, and the extension points.

The control path

When a user deploys a VM, the request travels:

UI / API → Management server → Database → Message queue → Host agent → Hypervisor

The response travels back the same way, as an asynchronous job. When something fails, walk the path in order:

  1. API — did the request validate? Check the job’s error text.
  2. Management server — is it healthy and can it reach the database and queue?
  3. Database — is it up, and is the disk full?
  4. Message queue — are agents connected and consuming?
  5. Host agent — is the agent running and is the host Up?
  6. Hypervisor — can it start the VM, and does it have the storage and network?

Logs and where to find them

ComponentLog location
Management server/var/log/cloudstack/management/management-server.log
Management server audit/var/log/cloudstack/management/audit.log
Agent/var/log/cloudstack/agent/agent.log
System VMs/var/log/cloudstack/management/ inside the VM, plus the console
Database/var/log/mysql/ or the distribution’s MariaDB log

The management log is verbose. Filter it by the job ID or the VM’s UUID:

sudo grep -i "<job-id>" /var/log/cloudstack/management/management-server.log
sudo grep -i "<vm-uuid>" /var/log/cloudstack/management/management-server.log

The audit log records who called which API command and the result, which is invaluable when the question is “who deleted this?” rather than “why did it fail?”.

Database

CloudStack stores its state in a relational database. MySQL 8 and MariaDB are the supported options; the installer creates the cloud and cloud_usage schemas.

Routine care:

  • Back it up. The database is the cloud’s source of truth. A backup of the database plus the secondary storage is a recoverable cloud; a backup of the VMs alone is not.
  • Monitor disk space. A full database disk takes the management server down and prevents new jobs.
  • Do not edit tables by hand. Use the API. Direct writes can leave the database inconsistent with the hypervisors.
  • Test restores. A backup you have never restored is a hypothesis.

A logical backup:

mysqldump --single-transaction --routines --triggers \
  -u cloud -p cloud > cloud-$(date +%F).sql

For larger installations, use a physical backup (Percona XtraBackup or the storage layer’s snapshot) and keep binlogs for point-in-time recovery.

Database HA

A single database is a single point of failure. Production deployments use a primary/replica pair or a managed database service. CloudStack supports database HA through the db.* configuration keys; the details depend on your database topology. Test failover before you need it.

Usage server and billing

The usage server is a separate service that samples resource usage and writes it to the cloud_usage schema. It is what turns a cloud into something you can bill for.

Enable and start it:

sudo systemctl enable --now cloudstack-usage

Usage records cover:

  • Running VM time
  • Allocated storage
  • Public IP addresses
  • Network usage
  • Snapshots and templates
  • Load balancer and VPN usage

Query usage through the API:

cmk list usagerecords startdate=2026-11-01 enddate=2026-11-30
cmk list usagerecords account=platform-team type=1

The usage server is not a billing system. It produces raw records; you aggregate them, apply prices, and invoice. If you charge for a service, decide early whether usage is metered hourly, daily, or monthly, and document how partial periods are rounded.

Backup and disaster recovery

A recoverable CloudStack deployment has four pieces:

PieceWhy it mattersHow to protect it
DatabaseDefines every object and relationshipDumps or physical backups, plus binlogs
Secondary storageTemplates, ISOs, snapshots, backupsSnapshot or replicate the NFS/S3 backend
Primary storageRunning VM disksStorage-level snapshots, replication, or volume backups
ConfigurationZone, network, and agent settingsVersion-controlled config and documented runbooks

Define two targets and test against them:

  • RPO — how much data you can lose. Drives backup frequency.
  • RTO — how long recovery may take. Drives whether you use restore-in-place or a warm standby.

A restore order that works: database first, then secondary storage, then primary storage, then start system VMs, then start user VMs.

High availability

CloudStack itself can run multiple management servers behind a load balancer. They share the database and the message queue, so the database and queue become the HA-critical components.

For hosts, offerha=true on a service offering restarts a VM on another host if its host fails. That is host-level HA, not application HA: the VM restarts, and anything in memory is lost.

For storage, use shared storage (NFS, iSCSI, Ceph) so a host failure does not take its VMs’ disks with it. Local storage ties a VM to one host and rules out live migration.

GPU and accelerated workloads

CloudStack supports GPU passthrough and vGPU on supported hypervisors. The shape of the configuration:

  1. Install the vendor driver on the host.
  2. Register the GPU or vGPU type with CloudStack.
  3. Create a compute offering that requests the GPU resource.
  4. Deploy the VM with that offering.
cmk list gpucards
cmk create serviceoffering \
  name=gpu-a10 \
  displaytext="1x A10" \
  cpunumber=8 \
  cpuspeed=2500 \
  memory=32768 \
  gputype=<gpu-type> \
  vgpu=<vgpu-profile>

GPU offerings are scarce resources. Use limits and dedicated clusters so a single tenant cannot exhaust the pool. Confirm the exact parameter names against your CloudStack version; GPU support has evolved across releases.

Webhooks

Webhooks let CloudStack call an external HTTP endpoint when an event occurs, instead of you polling the API. They are useful for chat notifications, ticketing integration, and audit pipelines.

Configure a webhook under Global Settings → Webhooks (or the webhook.* keys), then subscribe to events such as VM creation, destruction, or failure. Treat the receiver as untrusted input: validate the payload, authenticate the caller, and make the handler idempotent because deliveries can be retried.

Shared filesystems

CloudStack can present shared filesystems to instances, backed by a supported storage provider. This is different from a block volume: multiple VMs mount the same filesystem, so the filesystem itself must handle concurrent access.

Use it for:

  • Shared application content
  • Home directories
  • Build caches

Do not use it as a database’s data directory unless the database explicitly supports shared storage. Most do not, and corruption is the usual result.

Extensions and plugins

CloudStack is extensible. The common extension points are:

  • Network plugins — new SDN controllers or network backends.
  • Storage plugins — new primary or secondary storage providers.
  • Hypervisor plugins — new compute backends.
  • Usage plugins — custom metering.
  • API plugins — additional commands.

Plugins are version-sensitive. Pin the plugin to the CloudStack release, read its compatibility matrix, and test upgrades in a lab. A plugin that silently stops working after an upgrade is worse than one that refuses to load.

A troubleshooting checklist

When a deployment fails:

  1. Read the job result, not just the HTTP status.
  2. Find the job ID in the management log.
  3. Check the host agent log on the target host.
  4. Confirm the host is Up and the agent is connected.
  5. Confirm the template is Ready and available in the zone.
  6. Confirm the service offering’s storage type matches the pool.
  7. Confirm the network offering is compatible with the zone.
  8. Check database and storage capacity.
  9. Check the system VM that mediates the operation (router, console proxy, secondary storage VM).
  10. If all else fails, reproduce with the API and the smallest possible object.

When the cloud is slow rather than broken:

  • Check database query latency and disk I/O.
  • Check the message queue for a backlog.
  • Check management server heap and GC pauses.
  • Check storage latency; CloudStack waits on storage for many operations.
  • Check the number of concurrent jobs and the management.server.worker.threads setting.

Operational hygiene

  • Monitor the management server, database, queue, hosts, system VMs, and storage from outside CloudStack.
  • Alert on disk space before it is full, not after.
  • Keep the database backup and restore runbook current.
  • Review audit logs for unexpected admin actions.
  • Test a host failure and a management server failure on a schedule.
  • Keep the CloudStack version and plugins on a supported, documented combination.