TECH PREVIEW

Deploy Trove

OpenStack Trove is the Database as a Service (DBaaS) component of OpenStack. It provisions and manages database instances (running as Nova guest VMs) without the operator having to hand-build and administer each database. This document describes how Trove is deployed in Genestack.

Unlike a plain Helm install, Trove needs a good deal of surrounding scaffolding to actually work: install-script managed secret prerequisites, a dedicated management overlay network, security groups, a guest image that carries the Trove guest agent, a datastore registration, and a per-chassis bridge that lets guest VMs reach RabbitMQ and Keystone. In Genestack that entire lifecycle is driven by the trove_enablement_techpreview Ansible role, which wraps the bin/install-trove.sh Helm deployment.

Genestack facilitates the deployment by leveraging Kubernetes' orchestration capabilities together with an Ansible role that stitches Trove into Nova, Neutron, Cinder, Glance and Swift.

Reference the full online OpenStack Trove documentation.

Overview of the enablement flow

The trove_enablement_techpreview role performs the end-to-end enablement. At a high level it:

Ensures secret prerequisites — chart-derived Trove passwords are owned by bin/install-trove.sh and bin/services/trove.yaml; the role ensures they exist before pre-install work needs them. The role owns only the trove-ssh ed25519 keypair secret.
Creates the Trove management overlay network — a geneve network (trove-mgmt-net), its subnet, and a router with an external gateway to the public provider network.
Creates the security groups — trove-access-secgroup (ICMP, SSH, 3306 for guests) and trove-services-secgroup (5672 RabbitMQ, 5000 Keystone, 8080 Swift for the services VIP).
Deep-merges the Helm overrides — network ID, security group ID, keypair name and related conf.trove settings are merged into trove-helm-overrides.yaml. It also wires up the trove-mgmt-bridge DaemonSet, guest cloud-init/guest-agent ConfigMaps, and a MySQL config-template override.
Configures the gateway, kustomize overlay and endpoints — Envoy listener, HTTPRoute, the /etc/genestack/kustomize/trove overlay, and Trove stanzas in global_overrides/endpoints.yaml.
Builds and uploads the guest image — a Debian bookworm qcow2 built with diskimage-builder that carries MySQL, the Trove guest agent, Docker, and pre-baked datastore images. The image is uploaded to Glance and shared with the admin and service projects.
Registers the datastore version (post-deploy) — links the Glance image to the Trove mysql datastore and loads the configuration validation rules.
Installs python-troveclient, creates the Nova keypair as the trove service user, and distributes the guest SSH key to the compute nodes.
Deploys Swift (swift_all_in_one) for backup/restore support, unless disabled.

This is a tech preview focused on MySQL 8.4

The role currently builds and registers a MySQL 8.4 datastore on Debian bookworm. The datastore, MySQL version, and OpenStack release used for the guest agent are all configurable (see Role variables), but MySQL 8.4 is the tested path.

Prerequisites

Keystone, Nova, Neutron, Cinder and Glance deployed and operational.
A public provider network (default name flat) for the management router's external gateway.
The genestack virtualenv at /home/ubuntu/.venvs/genestack and genestack.rc present (the standard Genestack jump-host layout).
Ansible >= 2.15.8 with the kubernetes.core collection.

Create secrets

The role ensures Trove secrets for you as its first step, so no manual secret creation is normally required.

Information about the secrets used

Trove chart-derived secrets such as trove-rabbitmq-password, trove-db-password, and trove-admin are managed by bin/install-trove.sh through the shared bin/services/trove.yaml schema. Existing secret values are reused.

The techpreview role also manages the trove-ssh keypair secret used by the post-install access workflow. trove_force_recreate_secrets=true and force_full_recreation=true recreate only this SSH keypair secret; they do not rotate Trove database, RabbitMQ, or Keystone service-user passwords.

Define policy configuration

Information about the default policy rules used

The default RabbitMQ policy sets the quorum queues target group size to 3 for the trove vhost. This can be changed in base-kustomize/trove/base/policies.yaml.

Default RabbitMQ policy

apiVersion: rabbitmq.com/v1beta1
kind: Policy
metadata:
  name: trove-quorum-three-replicas
  namespace: openstack
spec:
  name: trove-quorum-three-replicas
  vhost: "trove"
  pattern: ".*"
  applyTo: queues
  definition:
    target-group-size: 3
  priority: 0
  rabbitmqClusterReference:
    name: rabbitmq

Run the enablement

The recommended path is to run the Ansible role end to end. It performs the pre-install scaffolding, calls bin/install-trove.sh for the Helm release, and then finishes the post-install steps (datastore registration, keypair, SSH key distribution).

Run the Trove enablement playbook

cd /opt/genestack/ansible/playbooks
ansible-playbook trove-enablement-techpreview.yaml

The playbook is a thin wrapper around the role:

- name: Trove (Database as a Service) Enablement
  hosts: localhost
  roles:
    - role: trove_enablement_techpreview

Running specific phases with tags

The role's tasks are tagged so you can run just part of the flow. This is useful when the Trove API is not yet up (defer the post-install steps) or when you only need to rebuild one piece.

Tag	What runs
`trove_pre_install`	Install-script managed secrets, mgmt network, security groups, Helm config, gateway/kustomize
`trove_secrets`	Install-script managed secret prerequisites
`trove_mgmt_network`	Management overlay network, subnet, router
`trove_security_groups`	`trove-access` and `trove-services` security groups
`trove_helm_config`	Deep-merge Helm overrides, mgmt-bridge DaemonSet, guest ConfigMaps
`trove_gateway`	Envoy listener, HTTPRoute, kustomize overlay, endpoints merge
`trove_post_install`	Image build, datastore setup, client install, keypair, SSH key
`trove_image_build`	Build/upload the guest image only
`trove_datastore`	Register the datastore version (requires the Trove API to be up)
`trove_client`	Install `python-troveclient`
`trove_keypair`	Create the Nova keypair as the `trove` service user
`trove_ssh_key_distribute`	Copy the guest SSH key to compute nodes
`deploy_swift`	Deploy Swift for backup/restore

Run only the post-install steps after Trove is up

ansible-playbook trove-enablement-techpreview.yaml --tags trove_post_install

Register the datastore version only

ansible-playbook trove-enablement-techpreview.yaml --tags trove_datastore

Force / recreation flags

By default the role is idempotent — it skips work that is already done (for example, it will not rebuild the guest image if it already exists in Glance). Pass these flags to force specific work:

Flag	Effect
`-e force_rebuild_image=true`	Delete and rebuild the guest image, then re-upload to Glance
`-e force_create_dsv=true`	Recreate the datastore version and reload configuration parameters
`-e trove_force_recreate_secrets=true`	Recreate only the `trove-ssh` keypair secret
`-e trove_force_recreate_mgmt_network=true`	Delete and recreate the management network/subnet/router
`-e trove_force_recreate_security_groups=true`	Delete and recreate the security groups
`-e force_full_recreation=true`	Nuclear option — recreate SSH keypair, network, security groups, image and datastore

Force a guest image rebuild

ansible-playbook trove-enablement-techpreview.yaml -e force_rebuild_image=true

The Helm deployment script

The role installs the Trove chart with bin/install-trove.sh. You can also run it directly if you only need to (re)deploy the chart after the scaffolding is in place.

Run the Trove deployment script /opt/genestack/bin/install-trove.sh

#!/bin/bash
# Description: Fetches the version for SERVICE_NAME_DEFAULT from the specified
# YAML file and executes a helm upgrade/install command with dynamic values files.

# Disable SC2124 (unused array), SC2145 (array expansion issue), SC2294 (eval)
# shellcheck disable=SC2124,SC2145,SC2294

# Service
SERVICE_NAME_DEFAULT="trove"
SERVICE_NAMESPACE="openstack"

# Helm
HELM_REPO_NAME_DEFAULT="openstack-helm"
HELM_REPO_URL_DEFAULT="https://tarballs.opendev.org/openstack/openstack-helm"

# Base directories provided by the environment
GENESTACK_BASE_DIR="${GENESTACK_BASE_DIR:-/opt/genestack}"
GENESTACK_OVERRIDES_DIR="${GENESTACK_OVERRIDES_DIR:-/etc/genestack}"

# Common secret helpers. Missing secrets are generated in Kubernetes and
# existing secrets are never overwritten.
# shellcheck source=helpers.sh
source "${GENESTACK_BASE_DIR}/bin/helpers.sh"
trap cleanup_tmp EXIT

# Define service-specific override directories based on the framework
SERVICE_BASE_OVERRIDES="${GENESTACK_BASE_DIR}/base-helm-configs/${SERVICE_NAME_DEFAULT}"
SERVICE_CUSTOM_OVERRIDES="${GENESTACK_OVERRIDES_DIR}/helm-configs/${SERVICE_NAME_DEFAULT}"

# Define the Global Overrides directory used in the original script
GLOBAL_OVERRIDES_DIR="${GENESTACK_OVERRIDES_DIR}/helm-configs/global_overrides"

# Read the desired chart version from VERSION_FILE
VERSION_FILE="${GENESTACK_OVERRIDES_DIR}/helm-chart-versions.yaml"

if [ ! -f "$VERSION_FILE" ]; then
    echo "Error: helm-chart-versions.yaml not found at $VERSION_FILE" >&2
    exit 1
fi

# Extract version dynamically using the SERVICE_NAME_DEFAULT variable
SERVICE_VERSION=$(grep "^[[:space:]]*${SERVICE_NAME_DEFAULT}:" "$VERSION_FILE" | sed "s/.*${SERVICE_NAME_DEFAULT}: *//")

if [ -z "$SERVICE_VERSION" ]; then
    echo "Error: Could not extract version for '$SERVICE_NAME_DEFAULT' from $VERSION_FILE" >&2
    exit 1
fi

echo "Found version for $SERVICE_NAME_DEFAULT: $SERVICE_VERSION"

# Load chart metadata from custom override YAML if defined
for yaml_file in "${SERVICE_CUSTOM_OVERRIDES}"/*.yaml; do
    if [ -f "$yaml_file" ]; then
        HELM_REPO_URL=$(yq eval '.chart.repo_url // ""' "$yaml_file")
        HELM_REPO_NAME=$(yq eval '.chart.repo_name // ""' "$yaml_file")
        SERVICE_NAME=$(yq eval '.chart.service_name // ""' "$yaml_file")
        break  # use the first match and stop
    fi
done

# Fallback to defaults if variables not set
: "${HELM_REPO_URL:=$HELM_REPO_URL_DEFAULT}"
: "${HELM_REPO_NAME:=$HELM_REPO_NAME_DEFAULT}"
: "${SERVICE_NAME:=$SERVICE_NAME_DEFAULT}"


# Determine Helm chart path
if [[ "$HELM_REPO_URL" == oci://* ]]; then
    # OCI registry path
    HELM_CHART_PATH="$HELM_REPO_URL/$HELM_REPO_NAME/$SERVICE_NAME"
else
    # --- Helm Repository and Execution ---
    helm repo add --force-update "$HELM_REPO_NAME" "$HELM_REPO_URL" 2>/dev/null || true
    helm repo update
    HELM_CHART_PATH="$HELM_REPO_NAME/$SERVICE_NAME"
fi

# Debug output
echo "[DEBUG] HELM_REPO_URL=$HELM_REPO_URL"
echo "[DEBUG] HELM_REPO_NAME=$HELM_REPO_NAME"
echo "[DEBUG] SERVICE_NAME=$SERVICE_NAME"
echo "[DEBUG] HELM_CHART_PATH=$HELM_CHART_PATH"

# Prepare an array to collect -f arguments
overrides_args=()

# Include all YAML files from the BASE configuration directory
# NOTE: Files in this directory are included first.
if [[ -d "$SERVICE_BASE_OVERRIDES" ]]; then
    echo "Including base overrides from directory: $SERVICE_BASE_OVERRIDES"
    for file in "$SERVICE_BASE_OVERRIDES"/*.yaml; do
        # Check that there is at least one match
        if [[ -e "$file" ]]; then
            echo " - $file"
            overrides_args+=("-f" "$file")
        fi
    done
else
    echo "Warning: Base override directory not found: $SERVICE_BASE_OVERRIDES"
fi

# Include all YAML files from the GLOBAL configuration directory
# NOTE: Files here override base settings and are applied before service-specific ones.
if [[ -d "$GLOBAL_OVERRIDES_DIR" ]]; then
    echo "Including global overrides from directory: $GLOBAL_OVERRIDES_DIR"
    for file in "$GLOBAL_OVERRIDES_DIR"/*.yaml; do
        if [[ -e "$file" ]]; then
            echo " - $file"
            overrides_args+=("-f" "$file")
        fi
    done
else
    echo "Warning: Global override directory not found: $GLOBAL_OVERRIDES_DIR"
fi

# Include all YAML files from the custom SERVICE configuration directory
# NOTE: Files here have the highest precedence.
if [[ -d "$SERVICE_CUSTOM_OVERRIDES" ]]; then
    echo "Including overrides from service config directory:"
    for file in "$SERVICE_CUSTOM_OVERRIDES"/*.yaml; do
        if [[ -e "$file" ]]; then
            echo " - $file"
            overrides_args+=("-f" "$file")
        fi
    done
else
    echo "Warning: Service config directory not found: $SERVICE_CUSTOM_OVERRIDES"
fi

echo

# Collect all --set arguments, executing commands and quoting safely
# NOTE: This array contains OpenStack-specific secret retrievals and MUST be updated
#       with the necessary --set arguments for your target SERVICE_NAME_DEFAULT.
# Collect secret-backed --set arguments from bin/services/${SERVICE_NAME_DEFAULT}.yaml.
collect_service_secret_set_args "$SERVICE_NAME_DEFAULT"
set_args=("${SECRET_HELM_SET_ARGS[@]}")

helm_command=(
    helm upgrade --install "$SERVICE_NAME_DEFAULT" "$HELM_CHART_PATH"
    --version "${SERVICE_VERSION}"
    --namespace="$SERVICE_NAMESPACE"
    --timeout 120m
    --create-namespace

    "${overrides_args[@]}"
    "${set_args[@]}"

    # Post-renderer configuration
    --post-renderer "$GENESTACK_OVERRIDES_DIR/kustomize/kustomize.sh"
    --post-renderer-args "$SERVICE_NAME_DEFAULT/overlay"

    "$@"
)

echo "Executing Helm command (arguments are quoted safely):"
printf '%q ' "${helm_command[@]}"
echo

# Execute the command directly from the array
"${helm_command[@]}"

What it does: the script reads the pinned Trove chart version from helm-chart-versions.yaml, then runs helm upgrade --install trove in the openstack namespace. It layers Helm values from three directories in order of increasing precedence — base-helm-configs/trove, helm-configs/global_overrides, then the operator's helm-configs/trove overrides — and injects service passwords with --set by reading the Keystone, Nova, Neutron, Cinder, MariaDB, memcached and RabbitMQ secrets out of Kubernetes. Finally it runs the Trove kustomize overlay as a Helm post-renderer.

Tip

For a multi-region deployment, see the Multi-Region Support guide for the recommended workflow.

Guest image and datastore

Trove instances are Nova VMs booted from a purpose-built guest image that contains the database engine and the Trove guest agent. Genestack builds this image with diskimage-builder rather than by customizing a cloud image.

The build (task file trove_guest_image_builder.yml) clones the upstream Trove repo at the release branch (stable/2025.2 by default), adds this role's custom DIB elements (debian-guest, debian-docker, image-pre-load), and runs disk-image-create to produce a Debian bookworm qcow2 named trove-mysql-8.4-bookworm. The image-pre-load element bakes the mysql:8.4 datastore image and a mysql-backup:8.4 image into the guest's Docker cache so instances boot with a warm cache and can perform backups.

After the Trove API is running, trove_datastore_setup.yml registers the datastore version and links it to the Glance image via image tags:

openstack datastore version create 8.4 mysql mysql "" \
    --image-tags trove,mysql,8.4,bookworm --active --default

Datastore image loader scripts inside the guest

The guest image ships two systemd units installed by the image-pre-load/install.d/32-install-datastore-image-loader element. On boot, trove-load-datastore-images.service loads only the datastore image (blocking, ordered before the guest agent — a sub-second no-op when already cached) and trove-load-backup-image.service loads the backup image in the background. Splitting the work this way keeps a reboot from delaying the guest agent past the taskmanager's RPC reply timeout.

For details on customizing the image build, see the Building MySQL Images for Trove guide.

Guest connectivity: the trove-mgmt-bridge

Trove guest VMs live on the trove-mgmt-net geneve overlay and must reach the in-cluster RabbitMQ and Keystone services. There is no physnet or MetalLB involved. Instead, the role deploys the trove-mgmt-bridge DaemonSet: one anchor pod per control-plane chassis, each bound to a per-chassis Neutron port and running haproxy that proxies 5672 (RabbitMQ) and 5000 (Keystone) into the cluster. Guest cloud-init points the rabbitmq and keystone hostnames at the services anchor IP (10.0.0.10 by default).

The per-chassis anchor ports are created by create_trove_mgmt_ports.sh.

How the role invokes it

create_trove_mgmt_ports.sh \
    trove-mgmt-net \                # network name
    <trove-services-secgroup-id> \  # security group allowing 5672/5000 in
    trove-mgmt-subnet \             # subnet name
    10 \                            # first host octet for anchor IPs
    default                         # os-cloud name

What it does: it lists the control-plane nodes, then creates one Neutron port per node on trove-mgmt-net, each bound to its specific chassis with --host= and pinned to a deterministic fixed IP below the subnet's allocation pool (mirroring the Octavia health-manager port pattern). The trove-mgmt-bridge init container later resolves "its" port by node-name suffix and plumbs the LSP into the pod network namespace.

Validate the deployment

After deployment, verify that the Trove services are running:

kubectl --namespace openstack get pods -l application=trove

List the Trove services and datastores:

openstack database service list
openstack datastore list
openstack datastore version list mysql

Database instance management

Create a database instance

openstack database instance create my-database \
    --flavor <flavor-id> \
    --size 10 \
    --datastore mysql \
    --datastore-version 8.4 \
    --nic net-id=<tenant-network-id>

The --nic should reference a tenant network the instance can use for client access; the management NIC on trove-mgmt-net is attached automatically by Trove.

List and inspect instances

openstack database instance list
openstack database instance show my-database

Create databases and users

# Create a database
openstack database db create my-database myapp_db

# Create a user with access to the database
openstack database user create my-database myapp_user myapp_password --databases myapp_db

# List databases and users
openstack database db list my-database
openstack database user list my-database

Backups

Backups require an object store. The role deploys Swift (swift_all_in_one) by default so that backups work out of the box. Set -e trove_deploy_swift=false to skip Swift deployment, in which case backups will not be available.

openstack database backup create my-database my-backup
openstack database backup list

Supported datastores

This tech preview builds and registers MySQL 8.4. The datastore name, version, and guest OS/OpenStack release are configurable through the role variables below, but MySQL 8.4 on Debian bookworm is the tested configuration.

Role variables

Common variables (see ansible/roles/trove_enablement_techpreview/defaults/main.yml for the full list):

Variable	Default	Description
`trove_mysql_version`	`8.4`	MySQL version to build and register
`trove_guest_image_os_release`	`bookworm`	Debian release for the guest image
`trove_guest_image_name`	`trove-mysql-8.4-bookworm`	Glance image name (derived from the two above)
`trove_openstack_release`	`2025.2`	Trove branch cloned for DIB elements + guest agent
`trove_datastore_name`	`mysql`	Trove datastore type name
`trove_datastore_version_name`	`8.4`	Trove datastore version
`trove_keypair_name`	`trove-access-keypair`	Nova keypair for instance access
`trove_secgroup_name`	`trove-access-secgroup`	Security group applied to guest instances
`trove_services_secgroup_name`	`trove-services-secgroup`	Security group for the services (RabbitMQ/Keystone) VIP
`trove_mgmt_public_network_name`	`flat`	Public provider network for the mgmt router's gateway
`trove_mgmt_network_name`	`trove-mgmt-net`	Management overlay network name
`trove_mgmt_subnet_cidr`	`10.0.0.0/20`	Management subnet CIDR
`trove_services_anchor_ip`	`10.0.0.10`	Anchor IP guests use for RabbitMQ/Keystone
`trove_deploy_swift`	`true`	Deploy Swift for backup/restore support

Guest agent and server release must match

trove_openstack_release (the guest agent branch) must be compatible with the Trove server-side image in trove-helm-overrides.yaml. A mismatch across the 23.x/25.x boundary can cause silently dropped RPC heartbeats and guest-agent build timeouts. See the comments in defaults/main.yml for the full rationale.

Troubleshooting

Check Trove logs

# API logs
kubectl --namespace openstack logs -l application=trove,component=api

# Conductor logs
kubectl --namespace openstack logs -l application=trove,component=conductor

# Taskmanager logs
kubectl --namespace openstack logs -l application=trove,component=taskmanager

Verify guest connectivity plumbing

# The mgmt-bridge DaemonSet should have one running pod per control-plane node
kubectl --namespace openstack get ds trove-mgmt-bridge
kubectl --namespace openstack rollout status ds/trove-mgmt-bridge

Common issues

Instance stuck in BUILD then errors with a guest-agent timeout — usually a guest agent/server release mismatch (see the note above) or the guest cannot reach the services anchor IP. Check the trove-mgmt-bridge pods and the guest's console log.
Datastore setup skipped — trove_datastore tasks are skipped when the Trove API is not reachable. Re-run with --tags trove_datastore once Trove is up.
Image not found — ensure the guest image built and uploaded to Glance (openstack image show trove-mysql-8.4-bookworm); rebuild with -e force_rebuild_image=true.
Instance creation fails on dependencies — ensure Nova, Neutron, Cinder and Glance are healthy and the trove service user has the roles it needs.

Configuration options

Key configuration options merged into trove-helm-overrides.yaml by the role (conf.trove.DEFAULT):

default_datastore: default database engine (mysql)
management_networks: the trove-mgmt-net network ID
management_security_groups: the trove-access-secgroup ID
nova_keypair: the keypair injected into guest instances
network_driver: trove.network.neutron.NeutronDriver

For advanced configuration, refer to the OpenStack Trove documentation.