Gateway API

Gateway API is L4 and L7 layer routing project in Kubernetes. It represents next generation of k8s Ingress, LB and Service Mesh APIs. For more information on the project see: Gateway API SIG.

Move from Ingress to Gateway APIs

Since Gateway APIs are successor to Ingress Controllers there needs to be a one time migration from Ingress to GW API resources.

Resource Models in Gateway API

There are 3 main resource models in gateway apis:

1. GatewayClass - Mostly managed by a controller.
2. Gateway - An instance of traffic handling infra like a LB.
3. Routes - Defines HTTP-specific rules for mapping traffic from a Gateway listener to a representation of backend network endpoints.

While both sound the same, API Gateway is a more of a general concept that defines a set of resources that exposes capabilities of a backend service but also provide other functionalities like traffic management, rate limiting, authentication and more. It is geared towards commercial API management and monetisation.

From the gateway api sig:

Controller Selection

There are various implementations of the Gateway API. In this document, we will cover two of them:

kubectl create ns nginx-gateway

kubectl kustomize "https://github.com/nginxinc/nginx-gateway-fabric/config/crd/gateway-api/standard?ref=v1.4.0" | kubectl apply -f -

kubectl kustomize "https://github.com/nginxinc/nginx-gateway-fabric/config/crd/gateway-api/experimental?ref=v1.4.0" | kubectl apply -f -

cd /opt/genestack/submodules/nginx-gateway-fabric/charts

helm upgrade --install nginx-gateway-fabric ./nginx-gateway-fabric \
            --namespace=nginx-gateway \
            --wait \
            --timeout 10m \
            -f /etc/genestack/helm-configs/nginx-gateway-fabric/helm-overrides.yaml \
            --post-renderer /etc/genestack/kustomize/kustomize.sh \
            --post-renderer-args gateway/base

cd /opt/genestack/submodules/nginx-gateway-fabric/charts

helm upgrade --install nginx-gateway-fabric . \
            --namespace=nginx-gateway \
            -f /etc/genestack/helm-configs/nginx-gateway-fabric/helm-overrides.yaml \
            --set nginxGateway.gwAPIExperimentalFeatures.enable=true

kubectl rollout restart deployment cert-manager --namespace cert-manager

kubectl kustomize /etc/genestack/kustomize/gateway/nginx-gateway-fabric | kubectl apply -f -

kubectl kustomize /etc/genestack/kustomize/gateway/nginx-gateway-fabric | kubectl apply -f -

kubectl kustomize --enable-helm /etc/genestack/kustomize/envoyproxy-gateway/base | kubectl apply -f -

kubectl kustomize /etc/genestack/kustomize/gateway/envoyproxy | kubectl apply -f -

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: test_application
namespace: test_app
spec:
parentRefs:
- name: flex-gateway
  sectionName: http
  namespace: envoy-gateway-system
hostnames:
- "test_application.sjc.your.domain.tld"
rules:
  - backendRefs:
    - name: test_application
      port: 8774

mkdir -p /etc/genestack/gateway-api
sed 's/your.domain.tld/<YOUR_DOMAIN>/g' /opt/genestack/etc/gateway-api/gateway-envoy-http-routes.yaml > /etc/genestack/gateway-api/gateway-envoy-http-routes.yaml
kubectl apply -f /etc/genestack/gateway-api/gateway-envoy-http-routes.yaml

Deploy with Let's Encrypt Certificates

By default, certificates are issued by an instance of the selfsigned-cluster-issuer. This section focuses on replacing that with a Let's Encrypt issuer to ensure valid certificates are deployed in our cluster.

Apply the Let's Encrypt Cluster Issuer

Before we can have Cert Manager start coordinating Let's Encrypt certificate requests for us, we need to add an ACME issuer with a valid, monitored email (for expiration reminders and other important ACME related information).

read -p "Enter a valid email address for use with ACME: " ACME_EMAIL; \
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ${ACME_EMAIL}
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRefs:
            - group: gateway.networking.k8s.io
              kind: Gateway
              name: flex-gateway
              namespace: nginx-gateway
EOF

Patch Gateway with valid listeners

By default, a generic Gateway is created using a hostname of *.cluster.local. To add specific hostnames/listeners to the gateway, you can either create a patch or update the gateway YAML to include your specific hostnames and then apply the patch/update. Each listener must have a unique name.

[
    {
        "op": "add",
        "path": "/spec/listeners/-",
        "value": {
            "name": "http-wildcard-listener",
            "port": 80,
            "protocol": "HTTP",
            "hostname": "*.your.domain.tld",
            "allowedRoutes": {
                "namespaces": {
                    "from": "All"
                }
            }
        }
    }
]

mkdir -p /etc/genestack/gateway-api/listeners
for listener in $(ls -1 /opt/genestack/etc/gateway-api/listeners); do
    sed 's/your.domain.tld/<YOUR_DOMAIN>/g' /opt/genestack/etc/gateway-api/listeners/$listener > /etc/genestack/gateway-api/listeners/$listener
done

kubectl patch -n nginx-gateway gateway flex-gateway \
              --type='json' \
              --patch="$(jq -s 'flatten | .' /etc/genestack/gateway-api/listeners/*)"

Apply Related Gateway routes

Another example with most of the OpenStack services is located at /opt/genestack/etc/gateway-api/routes/http-wildcard-listener.yaml. Similarly, you must modify and apply them as shown below, or apply your own.

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: http2https-route
  namespace: openstack
  labels:
    application: gateway-api
    service: HTTPRoute
    route: http2https
spec:
  parentRefs:
  - name: flex-gateway
    sectionName: http-wildcard-listener
    namespace: nginx-gateway
  hostnames:
  - "*.your.domain.tld"
  rules:
    - filters:
      - type: RequestRedirect
        requestRedirect:
          scheme: https
          statusCode: 301

All routes can be found at /etc/genestack/gateway-api/routes.

mkdir -p /etc/genestack/gateway-api/routes
for route in $(ls -1 /opt/genestack/etc/gateway-api/routes); do
    sed 's/your.domain.tld/<YOUR_DOMAIN>/g' /opt/genestack/etc/gateway-api/routes/$route > /etc/genestack/gateway-api/routes/$route
done

kubectl apply -f /etc/genestack/gateway-api/routes

Patch Gateway with Let's Encrypt Cluster Issuer

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: flex-gateway
  namespace: nginx-gateway
  annotations:
    acme.cert-manager.io/http01-edit-in-place: "true"
    cert-manager.io/cluster-issuer: letsencrypt-prod

kubectl patch --namespace nginx-gateway \
              --type merge \
              --patch-file /etc/genestack/gateway-api/gateway-letsencrypt.yaml \
              gateway flex-gateway

Example Implementation with Prometheus UI (NGINX Gateway Fabric)

In this example we will look at how Prometheus UI is exposed through the gateway. For other services the gateway kustomization file for the service.

First, create the shared gateway and then the httproute resource for prometheus.

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: prometheus-gateway-route
  namespace: prometheus
spec:
  parentRefs:
  - name: flex-gateway
    sectionName: http
    namespace: nginx-gateway
  hostnames:
  - "prometheus.your.domain.tld"
  rules:
    - backendRefs:
      - name: kube-prometheus-stack-prometheus
        port: 9090

mkdir -p /etc/genestack/gateway-api
sed 's/your.domain.tld/<YOUR_DOMAIN>/g' /opt/genestack/etc/gateway-api/gateway-prometheus.yaml > /etc/genestack/gateway-api/gateway-prometheus.yaml

kubectl apply -f /etc/genestack/gateway-api/gateway-prometheus.yaml

At this point, flex-gateway has a listener pointed to the port 80 matching *.your.domain.tld hostname. The HTTPRoute resource configures routes for this gateway. Here, we match all path and simply pass any request from the matching hostname to kube-prometheus-stack-prometheus backend service.

Example Implementation from Rackspace

This example is not required and is only intended to show how Rackspace deploys specific gateway kustomization files.

kubectl kustomize /etc/genestack/kustomize/gateway/nginx-gateway-fabric | kubectl apply -f -

Exposing Flex Services

We have a requirement to expose a service

1. Internally for private consumption (Management and Administrative Services)
2. Externally to customers (mostly Openstack services)

For each externally exposed service, example: keystone endpoint, we have a GatewayAPI resource setup to use listeners on services with matching rules based on hostname, for example keystone.your.domain.tld. When a request comes in to the f5 vip for this the vip is setup to pass the traffic to the Metallb external vip address. Metallb then forwards the traffic to the appropriate service endpoint for the gateway controller which matches the hostname and passes the traffic onto the right service. The same applies to internal services. Anything that matches your.domain.tld hostname can be considered internal and handled accordingly.

flowchart LR
    External --> External_VIP_Address --> MetalLB_VIP_Address --> Gateway_Service

This setup can be expended to have multiple MetalLB VIPs with multiple Gateway Services listening on different IP addresses as required by your setup.

Cross Namespace Routing

Gateway API has support for multi-ns and cross namespace routing. Routes can be deployed into different Namespaces and Routes can attach to Gateways across Namespace boundaries. This allows user access control to be applied differently across Namespaces for Routes and Gateways, effectively segmenting access and control to different parts of the cluster-wide routing configuration.

More information on cross namespace routing can be found here.