Notifications

☰

Overview

Proxmox VE emits Notification Events in case of storage replication failures, node fencing, finished/failed backups and other events. These events are processed based on the global notification settings. Each notification event includes metadata, such as a timestamp, severity level, type, and additional event-specific fields.
Notification Matchers route a notification event to one or more notification targets. A matcher can have match rules to selectively route based on the metadata of a notification event.
Notification Targets are a destination to which a notification event is routed to by a matcher. There are multiple types of target, mail-based (Sendmail and SMTP) and Gotify.

The global notification settings can be configured in the GUI under Datacenter → Notifications. The configuration is stored in /etc/pve/notifications.cfg and /etc/pve/priv/notifications.cfg - the latter contains sensitive configuration options such as passwords or authentication tokens for notification targets and can only be read by root.

Notification Targets

Proxmox VE offers multiple types of notification targets.

Sendmail

The sendmail binary is a program commonly found on Unix-like operating systems that handles the sending of email messages. It is a command-line utility that allows users and applications to send emails directly from the command line or from within scripts.

The sendmail notification target uses the sendmail binary to send emails to a list of configured users or email addresses. If a user is selected as a recipient, the email address configured in user’s settings will be used. For the root@pam user, this is the email address entered during installation. A user’s email address can be configured in Datacenter → Permissions → Users. If a user has no associated email address, no email will be sent.

In standard Proxmox VE installations, the sendmail binary is provided by Postfix. It may be necessary to configure Postfix so that it can deliver mails correctly - for example by setting an external mail relay (smart host). In case of failed delivery, check the system logs for messages logged by the Postfix daemon.

The configuration for Sendmail target plugins has the following options:

mailto: E-Mail address to which the notification shall be sent to. Can be set multiple times to accommodate multiple recipients.
mailto-user: Users to which emails shall be sent to. The user’s email address will be looked up in users.cfg. Can be set multiple times to accommodate multiple recipients.
author: Sets the author of the E-Mail. Defaults to Proxmox VE.
from-address: Sets the from address of the E-Mail. If the parameter is not set, the plugin will fall back to the email_from setting from datacenter.cfg. If that is also not set, the plugin will default to root@$hostname, where $hostname is the hostname of the node. The From header in the email will be set to $author <$from-address>.
comment: Comment for this target

Example configuration (/etc/pve/notifications.cfg):

sendmail: example
        mailto-user root@pam
        mailto-user admin@pve
        mailto max@example.com
        from-address pve1@example.com
        comment Send to multiple users/addresses

SMTP

SMTP notification targets can send emails directly to an SMTP mail relay. This target does not use the system’s MTA to deliver emails. Similar to sendmail targets, if a user is selected as a recipient, the user’s configured email address will be used.

Unlike sendmail targets, SMTP targets do not have any queuing/retry mechanism in case of a failed mail delivery.

The configuration for SMTP target plugins has the following options:

mailto: E-Mail address to which the notification shall be sent to. Can be set multiple times to accommodate multiple recipients.
mailto-user: Users to which emails shall be sent to. The user’s email address will be looked up in users.cfg. Can be set multiple times to accommodate multiple recipients.
author: Sets the author of the E-Mail. Defaults to Proxmox VE.
from-address: Sets the From-address of the email. SMTP relays might require that this address is owned by the user in order to avoid spoofing. The From header in the email will be set to $author <$from-address>.
username: Username to use during authentication. If no username is set, no authentication will be performed. The PLAIN and LOGIN authentication methods are supported.
password: Password to use when authenticating.
mode: Sets the encryption mode (insecure, starttls or tls). Defaults to tls.
server: Address/IP of the SMTP relay.
port: The port to connect to. If not set, the used port defaults to 25 (insecure), 465 (tls) or 587 (starttls), depending on the value of mode.
comment: Comment for this target

Example configuration (/etc/pve/notifications.cfg):

smtp: example
        mailto-user root@pam
        mailto-user admin@pve
        mailto max@example.com
        from-address pve1@example.com
        username pve1
        server mail.example.com
        mode starttls

The matching entry in /etc/pve/priv/notifications.cfg, containing the password:

smtp: example
        password somepassword

Gotify

Gotify is an open-source self-hosted notification server that allows you to send and receive push notifications to various devices and applications. It provides a simple API and web interface, making it easy to integrate with different platforms and services.

The configuration for Gotify target plugins has the following options:

server: The base URL of the Gotify server, e.g. http://<ip>:8888
token: The authentication token. Tokens can be generated within the Gotify web interface.
comment: Comment for this target

The Gotify target plugin will respect the HTTP proxy settings from the datacenter configuration

Example configuration (/etc/pve/notifications.cfg):

gotify: example
        server http://gotify.example.com:8888
        comment Send to multiple users/addresses

The matching entry in /etc/pve/priv/notifications.cfg, containing the secret token:

gotify: example
        token somesecrettoken

Webhook

Webhook notification targets perform HTTP requests to a configurable URL.

The following configuration options are available:

url: The URL to which to perform the HTTP requests. Supports templating to inject message contents, metadata and secrets.
method: HTTP Method to use (POST/PUT/GET)
header: Array of HTTP headers that should be set for the request. Supports templating to inject message contents, metadata and secrets.
body: HTTP body that should be sent. Supports templating to inject message contents, metadata and secrets.
secret: Array of secret key-value pairs. These will be stored in a protected configuration file only readable by root. Secrets can be accessed in body/header/URL templates via the secrets namespace.
comment: Comment for this target.

For configuration options that support templating, the Handlebars syntax can be used to access the following properties:

{{ title }}: The rendered notification title
{{ message }}: The rendered notification body
{{ severity }}: The severity of the notification (info, notice, warning, error, unknown)
{{ timestamp }}: The notification’s timestamp as a UNIX epoch (in seconds).
{{ fields.<name> }}: Sub-namespace for any metadata fields of the notification. For instance, fields.type contains the notification type - for all available fields refer to Notification Events.
{{ secrets.<name> }}: Sub-namespace for secrets. For instance, a secret named token is accessible via secrets.token.

For convenience, the following helpers are available:

{{ url-encode <value/property> }}: URL-encode a property/literal.
{{ escape <value/property> }}: Escape any control characters that cannot be safely represented as a JSON string.
{{ json <value/property> }}: Render a value as JSON. This can be useful to pass a whole sub-namespace (e.g. fields) as a part of a JSON payload (e.g. {{ json fields }}).

Examples

ntfy.sh

Method: POST
URL: https://ntfy.sh/{{ secrets.channel }}
Headers:
- Markdown: Yes
Body:

```
{{ message }}
```

Secrets:
- channel: <your ntfy.sh channel>

Discord

Method: POST
URL: https://discord.com/api/webhooks/{{ secrets.token }}
Headers:
- Content-Type: application/json
Body:

{
  "content": "``` {{ escape message }}```"
}

Secrets:
- token: <token>

Slack

Method: POST
URL: https://hooks.slack.com/services/{{ secrets.token }}
Headers:
- Content-Type: application/json
Body:

{
  "text": "``` {{escape message}}```",
  "type": "mrkdwn"
}

Secrets:
- token: <token>

Notification Matchers

screenshot/gui-datacenter-notification-matcher.png

Notification matchers route notifications to notification targets based on their matching rules. These rules can match certain properties of a notification, such as the timestamp, the severity of the notification or metadata fields, as well as arbitrary combinations of these. If a notification is matched by a matcher, all targets configured for the matcher will receive the notification.

An arbitrary number of matchers can be created, each with their own match rules and targets to notify. Every target is notified at most once for every notification, even if the target is used in multiple matchers.

Match Rules

Notification matchers can be configured in the Proxmox VE Web UI under Datacenter → Notifications. Matchers can be created, modified and deleted from this view.

When editing a matcher, match rules can be edited under the Match Rules tab. Rules are represented as a tree. Each tree node can either match on properties of a notification (fields, severity or timestamp) or combine the results of child nodes (e.g. All sub-rules match). Combinator nodes can be nested, which makes it possible to compose arbitrarily complex matching behavior.

The following node types are available:

Always match: Match any notification.
All sub-rules match: All direct children of this node must match.
At least one sub-rule matches: At least one direct child of this node must match.
Exactly one sub-rule matches: Exactly one direct child of this node must match.
Match Field: Match a metadata field of a notification. This node can either match if the field value matches one of the values provided, or, alternatively, if it matches a regular expression. Available fields are listed in Notification Events.
Match Calendar: Check if the timestamp of a notification is within a configured time range, based on a selection of weekdays and one contiguous time range within each day (e.g. Monday to Friday, 09:00 to 17:00). If multiple time ranges within a day must be matched, use an At least one sub-rule matches node with one Match Calendar child per time range.
Match Severity: Match the severity of a notification. The following severity levels are available: info, notice, warning, error and unknown. Each Match Severity node can specify multiple severity levels it matches on, for instance info and notice.

Each node in the tree can be inverted by checking Invert Rule. For instance, for a Match Severity node matching on info, checking Invert Rule lets the node match on notice, warning, error and unknown.

Under the Targets to Notify tab, one or more notification targets can be selected. All selected targets are notified if a notification is matched.

Matcher Configuration Options

This section gives an overview of the options used by matchers in the configuration file and in the API.

The following options are used in configuration entries for matchers:

target: Determine which target should be notified if the matcher matches. Can be used multiple times to notify multiple targets.
disable: Don’t evaluate this matcher.
comment: Comment for this matcher.
The matching behavior is defined either by the expression option or by the deprecated options below; the two cannot be combined.
- Expression-based matchers (preferred):
  - expression: Ruleset for this matcher as a single JSON-serialized object (Match Expression Reference).
- Deprecated matcher options:
  - invert-match: Inverts the result of the whole matcher.
  - mode: Determines how the individual match rules are evaluated to compute the result for the whole matcher. If set to all, all matching rules must match. If set to any, at least one rule must match. Defaults to all.
  - match-calendar: Match the notification’s timestamp against a schedule. Days can be specified as an inclusive range (e.g. mon..wed) or enumerated (e.g. mon,wed,sat). If no days are specified, the time range applies to every day of the week (e.g. 08:00-12:00).
    Examples:
    match-calendar mon..wed 9-12
    match-calendar mon,wed,sat 08:15-12:35
    match-calendar 04:15-16:35
  - match-field: Match a notification’s metadata field, either against a list of matched values, or as a regular expression. If the field does not exist, the notification will not be matched.
    Examples:
    match-field exact:type=vzdump
    match-field exact:type=replication,fencing
    match-field regex:hostname=^pve[0-9]+$
  - match-severity: Match the notification’s severity.
    Examples:
    match-severity error
    match-severity warning,error,unknown

A matcher without any matching rules always matches; the configured targets will always be notified.

Starting with Proxmox VE 9.3, match rules created and edited via the UI are stored as a single option expression in the respective matcher configuration entry. The previously used options (match-field, match-calendar, match-severity, mode and invert-match) will continue to be supported at least until the next major release.

Any existing matcher will be migrated to use the newer expression option when edited through the UI.

Match Expression Reference

This section describes the representation used to encode rules in the expression parameter.

The expression option contains a JSON-formatted tree structure of rules, with leaf nodes matching on the properties of a notification (field, severity, calendar) or representing constants (constant). Intermediate nodes and the root node encode the relationship between them (all-of, any-of, one-of, not).

Node Type	Description	Leaf Node	Example
all-of	Match if all child nodes match	No	{"all-of": [<expr>]}
any-of	Match if any child node matches	No	{"any-of": [<expr>]}
one-of	Match if exactly one child node matches	No	{"one-of": [<expr>]}
not	Invert the result of a single child node	No	{"not": <expr>}
constant	Always or never matches, depending on the boolean value	Yes	{"constant": true\|false}
match	Match on a property of a notification	Yes	{"match": <match-expr>}

Node Type

Description

Leaf Node

Example

all-of

Match if all child nodes match

{"all-of": [<expr>]}

any-of

Match if any child node matches

{"any-of": [<expr>]}

one-of

Match if exactly one child node matches

{"one-of": [<expr>]}

not

Invert the result of a single child node

{"not": <expr>}

constant

Always or never matches, depending on the boolean value

Yes

{"constant": true|false}

match

Match on a property of a notification

Yes

{"match": <match-expr>}

match leaf nodes can have the following variants (<match-expr>):

Match Type	Description	Example
calendar	Check if the notification’s timestamp is within a configured time range	{"type": "calendar", "schedule": "mon..fri 9-17"}
field	Check a notification’s fields against a regex or a list of permitted values	{"type": "field", "field": "job-id", "values": ["a", "b"]} or {"type": "field", "field": "hostname", "regex": "^pve[0-9]+$"}
severity	Check a notification’s severity against a list of permitted values	{"type": "severity", "severities": ["info", "notice"]}

Match Type

Description

Example

calendar

Check if the notification’s timestamp is within a configured time range

{"type": "calendar", "schedule": "mon..fri 9-17"}

field

Check a notification’s fields against a regex or a list of permitted values

{"type": "field", "field": "job-id", "values": ["a", "b"]} or {"type": "field", "field": "hostname", "regex": "^pve[0-9]+$"}

severity

Check a notification’s severity against a list of permitted values

{"type": "severity", "severities": ["info", "notice"]}

A pretty-printed, full example of a non-trivial expression is shown below. It encodes an expression matching on backup jobs (vzdump), but only within a given time range (mon..fri 09:00-17:00) or if it has a severity of warning or error. In the API and configuration file, the JSON is expected to be stored as a single line, as the value of the expression option; the line breaks and indentation are only for demonstration purposes.

{
  "all-of": [
    {
      "any-of": [
        {
          "match": {
            "type": "calendar",
            "schedule": "mon..fri 09:00-17:00"
          }
        },
        {
          "match": {
            "type": "severity",
            "severities": [
              "warning",
              "error"
            ]
          }
        }
      ]
    },
    {
      "match": {
        "type": "field",
        "field": "type",
        "values": [
          "vzdump"
        ]
      }
    }
  ]
}

Notification Events

Event	type	Severity	Metadata fields (in addition to type)
System updates available	package-updates	info	hostname
Cluster node fenced	fencing	error	hostname
Storage replication job failed	replication	error	hostname, job-id
Backup succeeded	vzdump	info	hostname, job-id (only for backup jobs)
Backup failed	vzdump	error	hostname, job-id (only for backup jobs)
Mail for root	system-mail	unknown	hostname

Event

type

Severity

Metadata fields (in addition to type)

System updates available

package-updates

info

hostname

Cluster node fenced

fencing

error

hostname

Storage replication job failed

replication

error

hostname, job-id

Backup succeeded

vzdump

info

hostname, job-id (only for backup jobs)

Backup failed

vzdump

error

hostname, job-id (only for backup jobs)

Mail for root

system-mail

unknown

hostname

Field name	Description
type	Type of the notification
hostname	Hostname, without domain (e.g. pve1)
job-id	Job ID

Field name

Description

type

Type of the notification

hostname

Hostname, without domain (e.g. pve1)

job-id

Job ID

Backup job notifications only have job-id set if the backup job was executed automatically based on its schedule, but not if it was triggered manually by the Run now button in the UI.

System Mail Forwarding

Certain local system daemons, such as smartd, generate notification emails that are initially directed to the local root user. These mails are converted into notification events with the type system-mail and a severity of unknown and are processed based on the global notification settings.

When the email is forwarded to a sendmail target, the mail’s content and headers are forwarded as-is. For all other targets, the system tries to extract both a subject line and the main text body from the email content. In instances where emails solely consist of HTML content, they will be transformed into plain text format during this process.

Permissions

To modify/view the configuration for notification targets, the Mapping.Modify/Mapping.Audit permissions are required for the /mapping/notifications ACL node.

Testing a target requires Mapping.Use, Mapping.Audit or Mapping.Modify permissions on /mapping/notifications

Notification Mode

Backup jobs allow to choose between two modes for sending backup related notifications. This is controlled by the notification-mode option in the backup job configuration.

Send notifications based on the global notification settings (notification-system).
Send notification emails via the system’s sendmail command to the email address configured in the backup job (legacy-sendmail). This mode also allows to select whether the email should be sent always or only on failure of the backup. Any targets or matchers from the global notification settings are ignored. This mode is equivalent to the notification behavior for Proxmox VE versions before 8.1. This mode might be removed in a later release of Proxmox VE.

Overriding Notification Templates

Proxmox VE uses Handlebars templates to render notifications. The original templates provided by Proxmox VE are stored in /usr/share/pve-manager/templates/default/.

Notification templates can be overridden by providing a custom template file in the override directory at /etc/pve/notification-templates/default/. When rendering a notification of a given type, Proxmox VE will first attempt to load a template from the override directory. If this one does not exist or fails to render, the original template will be used.

The template files follow the naming convention of <type>-<body|subject>.<html|txt>.hbs. For instance, the file vzdump-body.html.hbs contains the template for rendering the HTML version for backup notifications, while package-updates-subject.txt.hbs is used to render the subject line of notifications for available package updates.

Email-based notification targets, such as sendmail and smtp, always send multi-part messages with an HTML and a plain text part. As a result, both the <type>-body.html.hbs as well as the <type>-body.txt.hbs template will be used when rendering the email message. All other notification target types only use the <type>-body.txt.hbs template.