k8s kafka

梦想歌の网络日志 - K8s 采集 Kafka 监控指标

使用 StatefulSet 创建 kafka-exporter

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/name: kafka-exporter
    app.kubernetes.io/component: prometheus-exporter
  name: kafka-exporter
  namespace: tools
spec:
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/name: kafka-exporter
      app.kubernetes.io/component: prometheus-exporter
  serviceName: ""
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kafka-exporter
        app.kubernetes.io/component: prometheus-exporter
    spec:
      containers:
      - args:
        - --kafka.server=127.0.0.1:9092
        env:
        - name: TZ
          value: Asia/Shanghai
        image: ccr.ccs.tencentyun.com/rig-agent/kafka-exporter:v1.3.0
        imagePullPolicy: IfNotPresent
        name: kafka-exporter
        ports:
        - containerPort: 9308
          name: metric-port
          protocol: TCP
        resources:
          limits:
            cpu: 250m
            memory: 512Mi
          requests:
            cpu: 250m
            memory: 512Mi
        securityContext:
          privileged: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: qcloudregistrykey
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate

确认 kafka-exporter 的端点访问正常：http://127.0.0.1:9308/metrics

创建 Service 绑定 kafka-exporter

apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/kafka-exporter-prometheus-enabled: "true"
    prometheus.io/port: "9308"
  name: kafka-exporter
  namespace: tools
spec:
  clusterIP: none
  ports:
  - name: tcp-9308-9308
    port: 9308
    protocol: TCP
    targetPort: 9308
  selector:
    app.kubernetes.io/name: kafka-exporter
    app.kubernetes.io/component: prometheus-exporter
  sessionAffinity: None
  type: ClusterIP

加入 Prometheus 监控配置

基于 K8s 服务发现机制解析 kafka-exporter 的 endpoints，从而获得所有 Pod 的实例信息。

global:
  scrape_interval: 15s
  evaluation_interval: 15s

  - job_name: "kafka-exporter"
    scrape_interval: 5s
    kubernetes_sd_configs:
    - role: endpoints
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_kafka_exporter_prometheus_enabled]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
      action: replace
      target_label: __scheme__
      regex: (https?)
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
      action: replace
      target_label: __address__
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: namespace
    - source_labels: [__meta_kubernetes_service_name]
      action: replace
      target_label: service_name

确认 Prometheus 的目标 Kafka 节点状态为 UP

本文由 Ivan Dong 创作，采用 知识共享署名4.0 国际许可协议进行许可
本站文章除注明转载/出处外，均为本站原创或翻译，转载前请务必署名
最后编辑时间为: Jun 13, 2023 at 10:07 am