5 min to read

Installing Prometheus and Thanos with Helm

A step-by-step guide to setting up Prometheus and Thanos monitoring stack

Featured image

Image Reference


Overview

Following our previous post about Prometheus and Thanos, let’s explore how to install and configure these tools using Helm charts. We’ll cover both single-cluster and multi-cluster setups.

Prerequisites
  • Kubernetes cluster with storage provisioner configured
  • Object storage (AWS S3, GCP GCS, MinIO, Ceph, etc.)
  • Helm installed
  • kubectl configured


Installing Prometheus

Prometheus examines the installation method using the kube-prometheus-stack chart and the prometheus-community chart.

And the values file below is an example of installing only the components I need. Therefore, you just need to check values.yaml and add or remove the necessary components.


Installing Prometheus CRDs

# Add repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install CRDs
helm install prometheus-operator-crds -n monitoring prometheus-community/prometheus-operator-crds

# Verify installation
kubectl get crd | grep monitoring

Installing Prometheus Community Chart

Create values file (values/somaz.yaml):

server:
  name: server
  image:
    repository: quay.io/prometheus/prometheus
  persistentVolume:
    enabled: true
    accessModes:
      - ReadWriteOnce
    storageClass: "default"
  replicaCount: 1
  statefulSet:
    enabled: false
  service:
    enabled: true
    type: NodePort

alertmanager:
  enabled: true
  persistence:
    size: 2Gi

kube-state-metrics:
  enabled: true

prometheus-node-exporter:
  enabled: true

prometheus-pushgateway:
  enabled: true
  serviceAnnotations:
    prometheus.io/probe: pushgateway

Install Prometheus:

# Install
helm install prometheus-community . -n monitoring -f ./values/somaz.yaml

# Upgrade if needed
helm upgrade prometheus-community . -n monitoring -f ./values/somaz.yaml

Installing Kube-Prometheus-Stack

Create values file (values/somaz.yaml):

alertmanager:
  enabled: true
  config:
    global:
      resolve_timeout: 5m
  service:
    type: NodePort

grafana:
  enabled: false

prometheus:
  enabled: true
  thanosService:
    enabled: true
  thanosServiceMonitor:
    enabled: true
  thanosServiceExternal:
    enabled: true
    type: NodePort
  ingress:
    enabled: true
    ingressClassName: nginx
    hosts:
      - prometheus.somaz.link
  prometheusSpec:
    replicas: 1
    thanos:
      baseImage: quay.io/thanos/thanos
      version: v0.36.1
      objectStorageConfig:
        existingSecret:
          name: thanos-objstore
          key: objstore.yml
    storageSpec: 
      volumeClaimTemplate:
        spec:
          storageClassName: default
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 100Gi
    externalLabels:
      provider: somaz
      region: seoul
      cluster: mgmt

Install Kube-Prometheus-Stack:

# Verify configuration
helm lint --values ./values/somaz.yaml

# Install
helm install kube-prometheus-stack . -n monitoring -f ./values/somaz.yaml --create-namespace

# Upgrade if needed
helm upgrade kube-prometheus-stack . -n monitoring -f ./values/somaz.yaml


Installing Thanos

Thanos learns how to import data from an internal Prometheus Thanos Discovery Sidecar, how to import data from an external cluster’s Prometheus Thanos Discovery Sidecar, and how to integrate and manage it as a single Thanos.

Internal Cluster Setup

Create values file (values/somaz.yaml):

global:
  defaultStorageClass: "default"

fullnameOverride: "thanos"
clusterDomain: somaz-cluster.local

existingObjstoreSecret: "thanos-objstore"

query:
  enabled: true
  logLevel: debug
  replicaLabel:
    - prometheus_replica
  stores:
    - dnssrv+_grpc._tcp.kube-prometheus-stack-thanos-discovery.monitoring.svc.somaz-cluster.local
  ingress:
    enabled: true
    hostname: thanos-query.somaz.link
    ingressClassName: nginx

queryFrontend:
  enabled: true
  ingress:
    enabled: true
    hostname: thanos.somaz.link

compactor:
  enabled: true
  retentionResolutionRaw: 60d
  retentionResolution5m: 60d
  retentionResolution1h: 1y
  persistence:
    enabled: true
    size: 10Gi

storegateway:
  enabled: true
  persistence:
    enabled: true
    size: 10Gi

External Cluster Setup

Create values file (values/somaz-externalcluster.yaml):

query:
  enabled: true
  stores:
    - dnssrv+_grpc._tcp.kube-prometheus-stack-thanos-discovery.monitoring.svc.external-cluster.local
  ingress:
    grpc:
      enabled: true
      hostname: external-cluster-thanos-query.somaz.link
      ingressClassName: "nginx"
      annotations:
        nginx.ingress.kubernetes.io/backend-protocol: "GRPC"

Multi-cluster Setup

Update main Thanos values to include external cluster:

query:
  stores:
    - dnssrv+_grpc._tcp.kube-prometheus-stack-thanos-discovery.monitoring.svc.somaz-cluster.local
    - dnssrv+_grpc._tcp.thanos-multicluster-query-grpc.monitoring.svc.somaz-cluster.local

Install Thanos:

# Verify configuration
helm lint --values ./values/somaz.yaml

# Install
helm install thanos . -n monitoring -f ./values/somaz.yaml --create-namespace

# Upgrade if needed
helm upgrade thanos . -n monitoring -f ./values/somaz.yaml

Debugging Tips

# Check DNS resolution:
kubectl run -it --rm --image=nicolaka/netshoot dns-test --restart=Never -- dig _grpc._tcp.kube-prometheus-stack-thanos-discovery.monitoring.svc.somaz-cluster.local

# Verify endpoints:
kubectl get ep -n monitoring | grep thanos-discovery


Next Steps

In our next post, we’ll explore setting up:


References