Prometheus

We are taking advantage of the prometheus community kube-prometheus-stack as well as other various components for monitoring and alerting. For more information, take a look at Prometheus Kube Stack.

Tip

You may need to provide custom values to configure prometheus. For a simple single region or lab deployment you can supply an additional overrides flag using the example found at base-helm-configs/aio-example-openstack-overrides.yaml. In other cases such as a multi-region deployment you may want to view the Multi-Region Support guide to for a workflow solution.

Install the Prometheus Stack

Run the Prometheus deployment Script bin/install-prometheus.sh

#!/bin/bash

GENESTACK_DIR="${GENESTACK_DIR:-/opt/genestack}"
GENESTACK_CONFIG_DIR="${GENESTACK_CONFIG_DIR:-/etc/genestack}"

GENESTACK_PROMETHEUS_DIR=\
"${GENESTACK_PROMETHEUS_DIR:-$GENESTACK_DIR/base-helm-configs/prometheus}"
GENESTACK_PROMETHEUS_CONFIG_DIR=\
"${GENESTACK_PROMETHEUS_CONFIG_DIR:-$GENESTACK_CONFIG_DIR/helm-configs/prometheus}"

VALUES_BASE_FILENAMES=(
    "prometheus-helm-overrides.yaml"
    "alerting_rules.yaml"
    "alertmanager_config.yaml"
)

# Though it probably wouldn't make any difference for all of the
# $GENESTACK_CONFIG_DIR files to come last, this takes care to fully preserve
# the order
echo "Including overrides in order:"
values_args=()
for BASE_FILENAME in "${VALUES_BASE_FILENAMES[@]}"
do
    for DIR in "$GENESTACK_PROMETHEUS_DIR" "$GENESTACK_PROMETHEUS_CONFIG_DIR"
    do
        ABSOLUTE_PATH="$DIR/$BASE_FILENAME"
        if [[ -e "$ABSOLUTE_PATH" ]]
        then
            echo "    $ABSOLUTE_PATH"
            values_args+=("--values" "$ABSOLUTE_PATH")
        fi
    done
done
echo

helm repo add prometheus-community \
    https://prometheus-community.github.io/helm-charts
helm repo update
helm upgrade --install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
    --create-namespace --namespace=prometheus --timeout 10m \
    "${values_args[@]}" \
    --post-renderer "$GENESTACK_DIR/base-kustomize/kustomize.sh" \
    --post-renderer-args prometheus/base "$@"

Success

If the installation is successful, you should see the related exporter pods in the prometheus namespace.

kubectl -n prometheus get pods -l "release=kube-prometheus-stack"

Update Alertmanager Configuration

In this example, we supply a Teams webhook URL to send all open alerts to a teams channel. However, there are a plethora of other receivers available. For a full list, review prometheus documentation: receiver-integration-settings.

Example

You can ignore this step if you don't want to send alerts to Teams, the alertmanager will still deploy and provide information.

read -p "webhook_url: " webhook_url;
sed -i -e "s#https://webhook_url.example#$webhook_url#" \
/etc/genestack/helm-configs/prometheus/alertmanager_config.yaml