Prometheus: external template for AlertManager html email with kube-prometheus-stack

The kube-prometheus-stack bundles AlertManager for taking action on Prometheus alerts.

And if you are customizing the Heml custom values file to configure email alerting, there are multiple options available. The simplest is to allow the system to fallback to using the default subject and html templates.

But if you need to tailor the email content to your specific needs, then you need to supply override values. These values can be placed inline to the Helm custom values file, or you can provide an external file.

If you would like to skip the incremental decisions that lead us to externalized templates, and go straight to the Helm command and AlertManager template directive to accomplish this, then click here to go to the last section.

Option #1: Use fallback defaults

In its simplest form, we configure our SMTP server settings, but allow AlertManager to use its default subject line and html content.

alertmanager:
  config:
    global:
      resolve_timeout: 5m
      smtp_from: amgr@k3s
      smtp_smarthost: 10.43.235.116:1025
      smtp_require_tls: false

    route:
      group_by: ['job']
      receiver: email_platform
      routes:
      - receiver: 'null'
        matchers:
          - alertname =~ "InfoInhibitor|Watchdog"
      - receiver: email_platform
        continue: true
    receivers:
    - name: email_platform

      # allow fallback to default for subject line and html content
      email_configs:
      - to: platform@k3s
        send_resolved: true
    - name: 'null'
    templates:
    - '/etc/alertmanager/config/*.tmpl'

Option #2: Use direct inline values

In the simplest form of customization, we provide override inline values for subject and html content.

The subject line below follows an AlertManager template specification that is evaluated at runtime, where the double curly brackets denote values to be substituted.

The html and text inline values uses yaml block scalar syntax to inline their template content.

alertmanager:
  config:
    global:
      resolve_timeout: 5m
      smtp_from: amgr@k3s
      smtp_smarthost: 10.43.235.116:1025
      smtp_require_tls: false

    route:
      group_by: ['job']
      receiver: email_platform
      routes:
      - receiver: 'null'
        matchers:
          - alertname =~ "InfoInhibitor|Watchdog"
      - receiver: email_platform
        continue: true
    receivers:
    - name: email_platform

      # override subject line and html content 
      email_configs:
      - to: platform@k3s
        send_resolved: true
        headers:
          subject: "{{ .Status | toUpper }} {{ .CommonLabels.env }}:{{ .CommonLabels.cluster }} {{ .CommonLabels.alertname }}"

        html: |-
          <h3>You have the following alerts:</h3>
          {{ range .Alerts }}
          <p><b>{{.Labels.alertname}}</b>
            <ul>{{ range .Annotations.SortedPairs }}
            <li>{{ .Name }} = {{ .Value }}</li>
            {{ end }}</ul>
            <ul>{{ range .Labels.SortedPairs }}
            <li>{{ .Name }} = {{ .Value }}</li>
            {{ end }}</ul>
            {{ .GeneratorURL }}</p>
          {{ end }}
         
        text: |-
          You have the following alerts:
          {{ range .Alerts }}
          * {{.Labels.alertname}}
            {{ range .Annotations.SortedPairs }}
            {{ .Name }} = {{ .Value }}
            {{ end }}
            {{ range .Labels.SortedPairs }}
            {{ .Name }} = {{ .Value }}
            {{ end }}
            {{ .GeneratorURL }}
          {{ end }}

    - name: 'null'
    templates:
    - '/etc/alertmanager/config/*.tmpl'

Option #3: refer to inline template files

Although I don’t see any benefit to this hybrid approach, I am including it because it introduces the concept of AlertManager named templates.

If you define inline files at ‘alertmanager.templateFiles’, Helm will place them into the ‘/etc/alertmanager/config’ directory in the AlertManager pod. But this does not mean that AlertManager knows how to use them. AlertManager needs templates with a defined name, and that is why you see the files start with “{{ define <templateName> }}”, paired with an additional “{{ end }}”

From the AlertManager side, the “{{ template <templateName> . }}” directive we use for ‘html’ and ‘text’ elements tells AlertManager at runtime to use the named templates.

alertmanager:

  # places files into '/etc/alertmanager/config' directory on AlertManager
  templateFiles:
    email.default.html.tmpl: |-
      {{ define "emaildefaulthtml" }}
      <h3>You have the following alerts:</h3>
      {{ range .Alerts }}
      <p><b>{{.Labels.alertname}}</b>
        <ul>{{ range .Annotations.SortedPairs }}
        <li>{{ .Name }} = {{ .Value }}</li>
        {{ end }}</ul>
        <ul>{{ range .Labels.SortedPairs }}
        <li>{{ .Name }} = {{ .Value }}</li>
        {{ end }}</ul>
        {{ .GeneratorURL }}</p>
      {{ end }}
      {{ end }}

    email.default.txt.tmpl: |-
      {{ define "emaildefaulttext" }}
      You have the following alerts:
      {{ range .Alerts }}
      * {{.Labels.alertname}}
        {{ range .Annotations.SortedPairs }}
        {{ .Name }} = {{ .Value }}
        {{ end }}
        {{ range .Labels.SortedPairs }}
        {{ .Name }} = {{ .Value }}
        {{ end }}
        {{ .GeneratorURL }}
      {{ end }}
      {{ end }}

  config:
    global:
      resolve_timeout: 5m
      smtp_from: amgr@k3s
      smtp_smarthost: 10.43.235.116:1025
      smtp_require_tls: false

    route:
      group_by: ['job']
      receiver: email_platform
      routes:
      - receiver: 'null'
        matchers:
          - alertname =~ "InfoInhibitor|Watchdog"
      - receiver: email_platform
        continue: true
    receivers:
    - name: email_platform

      # override subject line and html content 
      email_configs:
      - to: platform@k3s
        send_resolved: true
        headers:
          subject: "{{ .Status | toUpper }} {{ .CommonLabels.env }}:{{ .CommonLabels.cluster }} {{ .CommonLabels.alertname }}"

        # refers to named template evaluated by AlertManager at runtime
        html: '{{ template "emaildefaulthtml" . }}'
        text: '{{ template "emaildefaulttext" . }}'

    - name: 'null'
    templates:
    - '/etc/alertmanager/config/*.tmpl'

Option #4: externalized templates using Helm and AlertManager template directive

Going one step further, let’s avoid the messiness of inline files altogether. If we create external files, then we can use the Helm ‘set-files’ flag to have it set the ‘alertmanager.templateFiles’ for us.

Assuming a local file named “email.default.html” with the content below:

{{ define "emaildefaulthtml" }}

<h3>You have the following alerts:</h3>
{{ range .Alerts }}<p>
<b>{{.Labels.alertname}}</b>
<ul>{{ range .Annotations.SortedPairs }}
<li>{{ .Name }} = {{ .Value }}</li>
{{ end }}</ul> 
<ul>{{ range .Labels.SortedPairs }} 
<li>{{ .Name }} = {{ .Value }}</li> 
{{ end }}</ul> 
{{ .GeneratorURL }}</p> 
{{ end }} 

{{ end }}

And a local file named “email.default.txt” with the content below:

{{ define "emaildefaulttext" }} 

You have the following alerts: {{ range .Alerts }} 
* {{.Labels.alertname}} 
{{ range .Annotations.SortedPairs }} 
{{ .Name }} = {{ .Value }} 
{{ end }} 
{{ range .Labels.SortedPairs }} 
{{ .Name }} = {{ .Value }} 
{{ end }} 
{{ .GeneratorURL }} 
{{ end }} 

{{ end }}

There is no need to inline ‘alertmanager.templateFiles’ anymore, you simply use the “{{ template <templateName> . }}” directive that AlertManager will load at runtime.

alertmanager:
  config:
    global:
      resolve_timeout: 5m
      smtp_from: amgr@k3s
      smtp_smarthost: 10.43.235.116:1025
      smtp_require_tls: false

    route:
      group_by: ['job']
      receiver: email_platform
      routes:
      - receiver: 'null'
        matchers:
          - alertname =~ "InfoInhibitor|Watchdog"
      - receiver: email_platform
        continue: true
    receivers:
    - name: email_platform

      # override subject line and html content 
      email_configs:
      - to: platform@k3s
        send_resolved: true
        headers:
          subject: "{{ .Status | toUpper }} {{ .CommonLabels.env }}:{{ .CommonLabels.cluster }} {{ .CommonLabels.alertname }}"

        # refers to named template evaluated by AlertManager at runtime
        html: '{{ template "emaildefaulthtml" . }}'
        text: '{{ template "emaildefaulttext" . }}'

    - name: 'null'
    templates:
    - '/etc/alertmanager/config/*.tmpl'

And when you invoke the Helm install/upgrade of kube-prometheus-stack, you provide the “set-file” flags that makes Helm provide the content in the ‘alertmanager.templateFiles’ location.

helm upgrade --namespace prom \
  -f prom-sparse.yaml prom-stack prometheus-community/kube-prometheus-stack \
  --set-file "alertmanager.templateFiles.email\.default\.html\.tmpl"=email.default.html \
  --set-file "alertmanager.templateFiles.email\.default\.txt\.tmpl"=email.default.txt

REFERENCES

AlertManager page

AlertManager, email_config

kube-prometheus-stack, values.yaml containing all values

github AlertManager, default email template

github, slack templates

Monzo github, other slack templates

kifarunux.com, alertmanager and smtp alerts

robustperception.io, alertmanager and pagerduty alerts

junosnotes.com, good explanations of alertmanager routes and continuing attribute

robustperception.io, pagerduty with alertmanager

velenux.wordpress, alertmanager email templates