Skip to content

Deployment Kubespray

Currently only the k8s provider kubespray is supported and included as submodule into the code base.

Before you Deploy

Kubespray will be using OVN for all of the network functions, as such, you will need to ensure your hosts are ready to receive the deployment at a low level. While the Kubespray tooling will do a lot of prep and setup work to ensure success, you will need to prepare your networking infrastructure and basic storage layout before running the playbooks.

Minimum system requirements

  • 2 Network Interfaces

Note

While we would expect the environment to be running with multiple bonds in a production cloud, two network interfaces is all that's required. This can be achieved with vlan tagged devices, physical ethernet devices, macvlan, or anything else. Have a look at the netplan example file found here for an example of how you could setup the network.

  • Ensure we're running kernel 5.17+

Tip

While the default kernel on most modern operating systems will work, we recommend running with Kernel 6.2+.

  • Kernel modules

Warning

The Kubespray tool chain will attempt to deploy a lot of things, one thing is a set of sysctl options which will include bridge tunings. Given the tooling will assume bridging is functional, you will need to ensure the br_netfilter module is loaded or you're using a kernel that includes that functionality as a built-in.

  • Executable /tmp

Warning

The /tmp directory is used as a download and staging location within the environment. You will need to make sure that the /tmp is executable. By default, some kick-systems set the mount option noexec, if that is defined you should remove it before running the deployment.

Create your Inventory

A default inventory file for kubespray is provided at /etc/genestack/inventory and must be modified.

Checkout the inventory.yaml.example file for an example of a target environment.

Note

Before you deploy the kubernetes cluster you should define the kube_override_hostname option in your inventory. This variable will set the node name which we will want to be an FQDN. When you define the option, it should have the same suffix defined in our cluster_name variable.

However, any Kubespray compatible inventory will work with this deployment tooling. The official Kubespray documentation can be used to better understand the inventory options and requirements.

Ensure systems have a proper FQDN Hostname

Before running the Kubernetes deployment, make sure that all hosts have a properly configured FQDN.

source /opt/genestack/scripts/genestack.rc
ansible -m shell -a 'hostnamectl set-hostname {{ inventory_hostname }}' --become all
ansible -m shell -a "grep 127.0.0.1 /etc/hosts | grep -q {{ inventory_hostname }} || sed -i 's/^127.0.0.1.*/127.0.0.1 {{ inventory_hostname }} localhost.localdomain localhost/' /etc/hosts" --become all

Note

In the above command I'm assuming the use of cluster.local this is the default cluster_name as defined in the group_vars k8s_cluster file. If you change that option, make sure to reset your domain name on your hosts accordingly.

The ansible inventory is expected at /etc/genestack/inventory and automatically loaded once genestack.rc is sourced.

Prepare hosts for installation

source /opt/genestack/scripts/genestack.rc
cd /opt/genestack/ansible/playbooks

Note

The rc file sets a number of environment variables that help ansible to run in a more easily to understand way.

While the ansible-playbook command should work as is with the sourced environment variables, sometimes it's necessary to set some overrides on the command line. The following example highlights a couple of overrides that are generally useful.

Example host setup playbook

ansible-playbook host-setup.yml

Example host setup playbook with overrides

Confirm inventory.yaml matches what is in /etc/genestack/inventory. If it does not match update the command to match the file names.

source /opt/genestack/scripts/genestack.rc
# Example overriding things on the CLI
ansible-playbook host-setup.yml

Note

The RC file sets a number of environment variables that help ansible to run in a more easy to understand way.

Run the cluster deployment

This is used to deploy kubespray against infra on an OpenStack cloud. If you're deploying on baremetal you will need to setup an inventory that meets your environmental needs. Change the directory to the kubespray submodule.

The cluster deployment playbook can also have overrides defined to augment how the playbook is executed. Confirm openstack-flex-inventory.yaml matches what is in /etc/genestack/inventory. If it does not match update the command to match the file names.

cd /opt/genestack/submodules/kubespray
ansible-playbook --inventory /etc/genestack/inventory/openstack-flex-inventory.ini \
                --private-key /home/ubuntu/.ssh/openstack-flex-keypair.key \
                --user ubuntu \
                --become \
                cluster.yml

The private-key option can be used to instruct ansible to use a custom SSH key for the SSH connection

    --private-key ${HOME}/.ssh/openstack-keypair.key

source /opt/genestack/scripts/genestack.rc
ansible-playbook setup-kubernetes.yml

Tip

Given the use of a venv, when running with sudo be sure to use the full path and pass through your environment variables; sudo -E /home/ubuntu/.venvs/genestack/bin/ansible-playbook.

Once the cluster is online, you can run kubectl to interact with the environment.