Alertmanager with Telegram for Prometheus-Stack in K8S

How to create a Telegram bot To create a Telegram bot, you'll need to have the Telegram app installed on your computer. If you don't have it already, you can download it from the Telegram website. Connect to BotFather and hit the /newbot and follow the instruction BotFather is a bot created by Telegram that allows you to create and manage your own bots. To connect to BotFather, search for "@botfather" in the Telegram app and click on the result to start a conversation. In case you’d like a more detailed walkthrough, we suggest you take a look at this amazing tutorial prepared by Telegram. Save the bot access token, it will be used in seting up telegram receiver in alertmanager.Also, join the bot to a channel/group to get the messages. To get a info about the bot The bot prefix has to be in the URL before the bot. BOT_ACCESS_TOKEN: it is provided by the "@BotFather". curl https://api.telegram.org/bot/getMe | jq . { "ok": true, "result": { "id": , "is_bot": true, "first_name": , "username": , "can_join_groups": true, "can_read_all_group_messages": false, "supports_inline_queries": false, "can_connect_to_business": false, "has_main_web_app": false } } CHAT_ID: After joining the bot to a channel/group , get the chat_id which you want to send message on behalf of the bot to that channel/group . curl https://api.telegram.org/bot/getUpdates | jq . To Send a test message curl -X POST "https://api.telegram.org/bot/sendMessage" -d "chat_id=" -d "text=Hello, World!" Let's say the prometheus-stack is installed via helm or its operator in K8S cluster. The release name will be used in many objects like AlertmanagerConfig, ServiceMonitor, PodMonitor, etc ... which the prometheus operator be able to select objects and add configs to the prometheus or alertmanager . get release list in the prometheus namespace helm list -n prometheus NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION prometheus-stack prometheus 1 2025-03-08 23:34:47.082601345 +0330 +0330 deployed kube-prometheus-stack-68.4.4 v0.79.2 AlertmanagerConfig to define the Telegram receiver GROUP: monitoring.coreos.com KIND: AlertmanagerConfig VERSION: v1alpha1 DESCRIPTION: AlertmanagerConfig configures the Prometheus Alertmanager, specifying how alerts should be grouped, inhibited and notified to external systems. telegramConfigs kubectl explain AlertmanagerConfig.spec.receivers.telegramConfigs GROUP: monitoring.coreos.com KIND: AlertmanagerConfig VERSION: v1alpha1 FIELD: telegramConfigs DESCRIPTION: List of Telegram configurations. TelegramConfig configures notifications via Telegram. See https://prometheus.io/docs/alerting/latest/configuration/#telegram_config botToken Telegram bot token. It is mutually exclusive with `botTokenFile`. The secret needs to be in the same namespace as the AlertmanagerConfig object and accessible by the Prometheus Operator. Either `botToken` or `botTokenFile` is required. chatID -required- The Telegram chat ID. parseMode enum: MarkdownV2, Markdown, HTML Parse mode for telegram message sendResolved Whether to notify about resolved alerts. The botToken has to be a Secret in the same namespace as the AlertmanagerConfig. lets say we name the secret file name telegram-bot-token-secret.yaml apiVersion: v1 kind: Secret metadata: name: telegram-bot-token-secret # secret name will be referred in botToken in AlertmanagerConfig namespace: prometheus # same `namespace` as the `AlertmanagerConfig`. type: Opaque data: # bot-token will be used as key for botToken in AlermanagerConfig. bot-token: # example # echo -n "" | base64 (Token without bot prefix, exactly the token get from bot_father) Create the Secret kubectl apply -f telegram-bot-token-secret.yaml -n prometheus AlertmanagerConfig Create a new file named alertmanagerconfig-telegram.yaml with contents: --- apiVersion: monitoring.coreos.com/v1alpha1 kind: AlertmanagerConfig metadata: name: alert-config-telegram namespace: prometheus labels: release: prometheus-stack # the release label should be the exact label of the prometheus helm release label alertmanagerConfig: alertmanager-telegram # custom label and also set this label to alertmanager in the prometheus-stack values.yaml spec: route: groupBy: ["alertname","job","namespace"] # it could be other labels to be grouped by groupWait: 30s groupInterval: 5m repeatInterval: 1h receiver: "telegram" routes: - receiver: "telegram" matchers: - name: severity matchType: "=~" value: "warning|critical" receivers: - name

Mar 9, 2025 - 13:23

Alertmanager with Telegram for Prometheus-Stack in K8S

How to create a Telegram bot

To create a Telegram bot, you'll need to have the Telegram app installed on your computer. If you don't have it already, you can download it from the Telegram website.
Connect to BotFather and hit the /newbot and follow the instruction
BotFather is a bot created by Telegram that allows you to create and manage your own bots. To connect to BotFather, search for "@botfather" in the Telegram app and click on the result to start a conversation.

In case you’d like a more detailed walkthrough, we suggest you take a look at this amazing tutorial prepared by Telegram.

Save the bot access token, it will be used in seting up telegram receiver in alertmanager.Also, join the bot to a channel/group to get the messages.

To get a info about the bot

The bot prefix has to be in the URL before the
bot.

BOT_ACCESS_TOKEN: it is provided by the "@BotFather".

curl https://api.telegram.org/bot/getMe | jq .

{
    "ok": true,
  "result": {
      "id": ,
    "is_bot": true,
    "first_name": <"bot_name">,
    "username": <"bot_username">,
    "can_join_groups": true,
    "can_read_all_group_messages": false,
    "supports_inline_queries": false,
    "can_connect_to_business": false,
    "has_main_web_app": false
  }
}

CHAT_ID: After joining the bot to a channel/group , get the chat_id which you want to send message on behalf of the bot to that channel/group .

curl https://api.telegram.org/bot/getUpdates | jq .

To Send a test message

curl -X POST "https://api.telegram.org/bot/sendMessage" -d "chat_id=" -d "text=Hello, World!"

Let's say the prometheus-stack is installed via helm or its operator in K8S cluster.
The release name will be used in many objects like AlertmanagerConfig, ServiceMonitor, PodMonitor, etc ... which the prometheus operator be able to select objects and add configs to the prometheus or alertmanager .

get release list in the prometheus namespace

helm list -n prometheus

NAME                NAMESPACE   REVISION    UPDATED                                     STATUS      CHART                           APP VERSION
prometheus-stack    prometheus  1       2025-03-08 23:34:47.082601345 +0330 +0330   deployed    kube-prometheus-stack-68.4.4    v0.79.2

`AlertmanagerConfig` to define the `Telegram` receiver

GROUP:      monitoring.coreos.com
KIND:       AlertmanagerConfig
VERSION:    v1alpha1

DESCRIPTION:
    AlertmanagerConfig configures the Prometheus Alertmanager,
    specifying how alerts should be grouped, inhibited and notified to external
    systems.

`telegramConfigs`

kubectl explain AlertmanagerConfig.spec.receivers.telegramConfigs

GROUP:      monitoring.coreos.com
KIND:       AlertmanagerConfig
VERSION:    v1alpha1

FIELD: telegramConfigs <[]Object>


DESCRIPTION:
    List of Telegram configurations.
    TelegramConfig configures notifications via Telegram.
    See
    https://prometheus.io/docs/alerting/latest/configuration/#telegram_config

  botToken