Home

這個是非常方便可以HPA自動橫向縮、放的功能

完成圖

 kubectl get hpa -w
NAME                       REFERENCE        TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-app-keda-scaler   Deployment/app   0/120 (avg)   2         4         2          13m
keda-hpa-app-keda-scaler   Deployment/app   22m/120 (avg)   2         4         2          16m
keda-hpa-app-keda-scaler   Deployment/app   0/120 (avg)     2         4         2          17m
keda-hpa-app-keda-scaler   Deployment/app   22m/120 (avg)   2         4         2          18m
keda-hpa-app-keda-scaler   Deployment/app   0/120 (avg)     2         4         2          19m
keda-hpa-app-keda-scaler   Deployment/app   22m/120 (avg)   2         4         2          19m
keda-hpa-app-keda-scaler   Deployment/app   0/120 (avg)     2         4         2          20m
keda-hpa-app-keda-scaler   Deployment/app   45m/120 (avg)   2         4         2          23m
keda-hpa-app-keda-scaler   Deployment/app   0/120 (avg)     2         4         2          24m
keda-hpa-app-keda-scaler   Deployment/app   22m/120 (avg)   2         4         2          25m
keda-hpa-app-keda-scaler   Deployment/app   67m/120 (avg)   2         4         2          25m

因為我弄了自動，所以需要加載k3s 權限

先加上

1 prometheus-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus-sa
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-cluster-role
rules:
- apiGroups: [""]
  resources: ["nodes", "nodes/metrics", "nodes/proxy"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-cluster-role-binding
subjects:
- kind: ServiceAccount
  name: prometheus-sa
  namespace: default
roleRef:
  kind: ClusterRole
  name: prometheus-cluster-role
  apiGroup: rbac.authorization.k8s.io

2 prometheus-deploy.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: default
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s

    scrape_configs:
      - job_name: 'kubernetes-pods-waf'
        # 🟢 1. 叫 Prometheus 自動去捞叢集裡所有的 Pod
        kubernetes_sd_configs:
          - role: pod

        relabel_configs:
          # 🟢 2. 只保留標籤為 app=waf 的 WAF Pod，其餘無關的 Pod 全部過濾掉
          - source_labels: [__meta_kubernetes_pod_label_app]
            regex: 'waf'
            action: keep

          # 🟢 3. 因為 WAF 開啟了 hostNetwork，直接將抓取目標導向它所在的實體機 IP 與 2019 管理 Port [|]
          - source_labels: [__meta_kubernetes_pod_ip]
            regex: '(.*)'
            replacement: '${1}:2019'
            target_label: __address__

          # 🟢 4. 【核心關鍵】自動提取這個 Pod 所在的 K8s 節點名稱，強制命名為 `waf_node`
          # 這樣不論機器叫什麼名字、未來增減多少台，標籤都會動態生成！
          - source_labels: [__meta_kubernetes_pod_node_name]
            target_label: waf_node    
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: light-prometheus
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: light-prometheus
  template:
    metadata:
      labels:
        app: light-prometheus
    spec:
      serviceAccountName: prometheus-sa # 🟢 關鍵：帶上剛才建立的權限帳號
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: prometheus
        image: prom/prometheus:v2.54.1
        args:
          - "--config.file=/etc/prometheus/prometheus.yml"
          - "--storage.tsdb.retention.time=2h" # 只保留 2 小時暫存，防硬碟爆滿
        ports:
        - containerPort: 9090
        volumeMounts:
        - name: config-volume
          mountPath: /etc/prometheus
      volumes:
      - name: config-volume
        configMap:
          name: prometheus-config
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus-service
  namespace: default
spec:
  selector:
    app: light-prometheus
  ports:
    - protocol: TCP
      port: 9090
      targetPort: 9090

3 keda-app-scaler.yaml

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: app-keda-scaler
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app                     
  minReplicaCount: 2               #  離峰時最少維持 2 個 Pod (K8s 會自動分配成 1+1)
  maxReplicaCount: 4               #  尖峰時最多長到 4 個 Pod (K8s 會自動分配成 2+2)
  pollingInterval: 15              
  cooldownPeriod: 120              
  

  triggers:
  - type: prometheus
    metadata:
      # 1. 補上先前漏掉的 Prometheus 內部 Service 短網址
      serverAddress: http://prometheus-service.default.svc.cluster.local:9090
      # 2. 指標名稱
      metricName: caddy_http_requests_total_dynamic
      # 3. 全叢集總流量 PromQL 查詢句
      query: sum(rate(caddy_http_requests_total[1m]))
      # 4.  你的總流量門檻：
      # 因為 minReplicaCount 是 2，如果設定門檻為 120 RPS，
      # 代表全叢集總流量每秒超過 120 下（平均每個 Pod 承受 60 下）時，就會觸發擴展，最高幫長到 4 個 Pod！
      threshold: '120'

之所以這麼麻煩是因為我的環境是沒有互通的，也是是k3s是沒有互相調度的，只鎖定入口後，流量就在該台worker運作，不會跨worker。

因為資源稀缺，所以刻意拿掉了KUBE-PROXY這一塊。

正常來說怖了keda，它是會自動調度資源的。所以這個是特例。

上面的抓取指標，我因為是用CADDY，BUILD的waf，所以我就是寫抓CADDY裡面的指標。它的功能非常多，並且和雲端環境高度適合，再自行挑選。

#svc name

kubectl get svc -n default
NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
app-headless         ClusterIP   None          <none>        3000/TCP         10d
gitea-service        NodePort    10.43.40.54   <none>        3000:30080/TCP   8d
kubernetes           ClusterIP   10.43.0.1     <none>        443/TCP          10d
postgres             ClusterIP   None          <none>        6432/TCP         4d22h
prometheus-service   ClusterIP   10.43.154.4   <none>        9090/TCP         4h44m
redis                ClusterIP   None          <none>        6379/TCP         4d22h

忘了最重要的

請下載、安裝keda

照順序安裝
1 keda-2.20.2-crds.yaml 2  keda-2.20.2-core.yaml