Prometheus: exposing Prometheus/Grafana as Ingress for kube-prometheus-stack

The kube-prometheus-stack bundles Prometheus, Grafana, and AlertManager for monitoring a Kubernetes cluster.

By default, the Ingress of these services is disabled.  In this article I will show you how to expose these services with NGINX Ingress either via subdomain (e.g. prometheus.my.domain) or web context (e.g. my.domain/prometheus).

You would not want to expose these monitoring applications on the same Ingress as your public end-user traffic, but if you had a secondary NGINX ingress tied to a private internal-only network, then exposing these applications could be appropriate.

Prerequisites

Kubernetes cluster with NGINX Ingress

You will need a Kubernetes cluster with NGINX Ingress installed in order to go through this article.

You can use my article on installing a K3s with MetalLB and NGINX Ingress available here.

Certificates available for TLS

Depending on whether we will be exposing these services at subdomains, for example:

• alertmanager.k3s.local
• grafana.k3s.local
• prometheus.local

OR at a web context path, for example:

• k3s.local/alertmanager
• k3s.local/grafana
• k3s.local/prometheus

Certificates can be created using OpenSSL for the required name(s).  It can be as simple as a self-signed SAN, or custom CA with SAN, or you can get them from a commercial certificate provider.

Ultimately, the TLS key+certificate for the domain name(s) must be loaded as a secret into the cluster.

secret_name="tls-credential"

# create Kubernetes secret from local file named my.key and my.crt
kubectl create secret tls $secret_name \
  --key=my.key \
  --cert=my.crt \
  -n default

Option #1: Exposing monitoring applications at subdomain

Exposing Prometheus, AlertManager, and Grafana with their own subdomain requires an override of the Helm chart defaults by creating your own custom values file.

This section assumes we are exposing the applications at the subdomains below and the ‘tls-credential’ secret has all these subdomain names in its SAN certificate:

• alertmanager.k3s.local
• grafana.k3s.local
• prometheus.local

AlertManager ingress enabled

Here is the proper syntax for enabling AlertManager ingress at a subdomain (https://alertmanager.k3s.local)

alertmanager:
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
    hosts: ['alertmanager.k3s.local']
    paths: ['/']
    tls:
    - secretName: tls-credential
      hosts:
      - alertmanager.k3s.local
...
  alertmanagerSpec:
    externalUrl: https://alertmanager.k3s.local/
    routePrefix: /

Grafana ingress enabled

Here is the proper syntax for enabling Grafana ingress at a subdomain (https://grafana.k3s.local)

grafana:
  # username is 'admin'
  adminPassword: prom-operator
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
    hosts: ['grafana.k3s.local']
    path: "/"
    tls:
    - secretName: tls-credential
      hosts:
      - grafana.k3s.local

Prometheus ingress enabled

Here is the syntax for enabling the Prometheus Ingress at a subdomain (https://prometheus.k3s.local)

prometheus:
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
    hosts: ['prometheus.k3s.local']
    paths: ['/']
    tls:
    - secretName: tls-credential
      hosts:
      - prometheus.k3s.local
...
  prometheusSpec:
    externalUrl: "https://prometheus.k3s.local/"
    routePrefix: /

Option #2: Exposing monitoring applications at web context

Exposing Prometheus, AlertManager, and Grafana at the same domain (but different web context) requires an override of the Helm chart defaults by creating your own custom values file.

This section assumes we are exposing the applications at the ‘k3s.local’ domain and the ‘tls-credential’ secret has this domain name as is common name or SAN.

• k3s.local/alertmanager
• k3s.local/grafana
• k3s.local/prometheus

AlertManager ingress enabled

Here is the proper syntax for enabling AlertManager ingress at a web context path (https://k3s.local/alertmanager)

alertmanager:
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/rewrite-target: /$2
    hosts: ['k3s.local']
    paths: ['/alertmanager(/|$)(.*)']
    tls:
    - secretName: tls-credential
      hosts:
      - k3s.local
... 
  alertmanagerSpec: 
    externalUrl: https://k3s.local/alertmanager
    routePrefix: "/" # does not need alertmanager suffix

Grafana ingress enabled

Here is the proper syntax for enabling Grafana ingress at a web context path (https://k3s.local/grafana).  Notice we had to add a couple of environment variables to get Grafana to properly process requests at a context path.

grafana:
  env:
    GF_SERVER_ROOT_URL: https://k3s.local/grafana
    GF_SERVER_SERVE_FROM_SUB_PATH: 'true'
  # username is 'admin'
  adminPassword: prom-operator
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/rewrite-target: /$2
    hosts: ['k3s.local']
    path: "/grafana(/|$)(.*)"
    tls:
    - secretName: tls-credential
      hosts:
      - k3s.local

Prometheus ingress enabled

Here is the syntax for enabling the Prometheus Ingress at a web context path (https://k3s.local/prometheus)

prometheus:
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
    hosts: ['k3s.local']
    paths: ['/prometheus'] # does not need regex capture like others
    tls:
    - secretName: tls-credential
      hosts:
      - k3s.local
...
  prometheusSpec:
    externalUrl: "https://k3s.local/prometheus"
    routePrefix: /prometheus

Full custom values file

I have a full example values file for a K3s cluster that exposes these monitoring services by subdomain at prom-sparse.expanded.yaml.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# create the namespace
kubectl create ns prom

# install|upgrade
helm install \
  --namespace prom \
  -f prom-sparse.yaml \
  prom-stack prometheus-community/kube-prometheus-stack

REFERENCES

kube-prometheus-stack, values.yaml containing all values