Deployment Kubespray

Currently only the k8s provider kubespray is supported and included as submodule into the code base.

Info

Existing OpenStack Ansible inventory can be converted using the /opt/genestack/scripts/convert_osa_inventory.py script which provides a hosts.yml

Before you Deploy

Kubespray will be using OVN for all of the network functions, as such, you will need to ensure your hosts are ready to receive the deployment at a low level. While the Kubespray tooling will do a lot of prep and setup work to ensure success, you will need to prepare your networking infrastructure and basic storage layout before running the playbooks.

Minimum system requirements

2 Network Interfaces

Note

While we would expect the environment to be running with multiple bonds in a production cloud, two network interfaces is all that's required. This can be achieved with vlan tagged devices, physical ethernet devices, macvlan, or anything else. Have a look at the netplan example file found here for an example of how you could setup the network.

Ensure we're running kernel 5.17+

Tip

While the default kernel on most modern operating systems will work, we recommend running with Kernel 6.2+.

Kernel modules

Warning

The Kubespray tool chain will attempt to deploy a lot of things, one thing is a set of sysctl options which will include bridge tunings. Given the tooling will assume bridging is functional, you will need to ensure the br_netfilter module is loaded or you're using a kernel that includes that functionality as a built-in.

Executable /tmp

Warning

The /tmp directory is used as a download and staging location within the environment. You will need to make sure that the /tmp is executable. By default, some kick-systems set the mount option noexec, if that is defined you should remove it before running the deployment.

Create your Inventory

A default inventory file for kubespray is provided at /etc/genestack/inventory and must be modified.

Checkout the openstack-flex/prod-inventory-example.yaml file for an example of a target environment.

Note

Before you deploy the kubernetes cluster you should define the kube_override_hostname option in your inventory. This variable will set the node name which we will want to be an FQDN. When you define the option, it should have the same suffix defined in our cluster_name variable.

However, any Kubespray compatible inventory will work with this deployment tooling. The official Kubespray documentation can be used to better understand the inventory options and requirements. Within the ansible/playbooks/inventory directory there is a directory named openstack-flex and openstack-enterprise. These directories provide everything we need to run a successful Kubernetes environment for genestack at scale. The difference between enterprise and flex are just target environment types.

Ensure systems have a proper FQDN Hostname

Before running the Kubernetes deployment, make sure that all hosts have a properly configured FQDN.

source /opt/genestack/scripts/genestack.rc
ansible -m shell -a 'hostnamectl set-hostname {{ inventory_hostname }}' --become all
ansible -m shell -a "grep 127.0.0.1 /etc/hosts | grep -q {{ inventory_hostname }} || sed -i 's/^127.0.0.1.*/127.0.0.1 {{ inventory_hostname }} localhost.localdomain localhost/' /etc/hosts" --become all

Note

In the above command I'm assuming the use of cluster.local this is the default cluster_name as defined in the group_vars k8s_cluster file. If you change that option, make sure to reset your domain name on your hosts accordingly.

The ansible inventory is expected at /etc/genestack/inventory

Prepare hosts for installation

source /opt/genestack/scripts/genestack.rc
cd /opt/genestack/ansible/playbooks

Note

The RC file sets a number of environment variables that help ansible to run in a more easily to understand way.

While the ansible-playbook command should work as is with the sourced environment variables, sometimes it's necessary to set some overrides on the command line. The following example highlights a couple of overrides that are generally useful.

Example host setup playbook

ansible-playbook host-setup.yml

Example host setup playbook with overrides

Confirm openstack-flex-inventory.yaml matches what is in /etc/genestack/inventory. If it does not match update the command to match the file names.

# Example overriding things on the CLI
ansible-playbook host-setup.yml --inventory /etc/genestack/inventory/openstack-flex-inventory.yaml \
                                --private-key ${HOME}/.ssh/openstack-flex-keypair.key

Run the cluster deployment

This is used to deploy kubespray against infra on an OpenStack cloud. If you're deploying on baremetal you will need to setup an inventory that meets your environmental needs.

Change the directory to the kubespray submodule.

cd /opt/genestack/submodules/kubespray

Source your environment variables

source /opt/genestack/scripts/genestack.rc

Note

The RC file sets a number of environment variables that help ansible to run in a more easy to understand way.

Once the inventory is updated and configuration altered (networking etc), the Kubernetes cluster can be initialized with

ansible-playbook cluster.yml

The cluster deployment playbook can also have overrides defined to augment how the playbook is executed. Confirm openstack-flex-inventory.yaml matches what is in /etc/genestack/inventory. If it does not match update the command to match the file names.

ansible-playbook --inventory /etc/genestack/inventory/openstack-flex-inventory.yaml \
                 --private-key /home/ubuntu/.ssh/openstack-flex-keypair.key \
                 --user ubuntu \
                 --become \
                 cluster.yml

Tip

Given the use of a venv, when running with sudo be sure to use the full path and pass through your environment variables; sudo -E /home/ubuntu/.venvs/genestack/bin/ansible-playbook.

Once the cluster is online, you can run kubectl to interact with the environment.