來把預設需多次加、解密封包的flannel，更換一下成host-gw
因為我的環境剛好就有WG，所以就沒有使用內建的host-gw, 不然就更加快速和方便
不過最好是開始安怖建環境就先規劃好，而不是像我中途更換，因為會中斷業務
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{.metadata.annotations}{"\n\n"}{end}' | grep flannel
{"alpha.kubernetes.io/provided-node-ip":"10.99.99.3","flannel.alpha.coreos.com/backend-data":"null","flannel.alpha.coreos.com/backend-type":"host-gw","flannel.alpha.coreos.com/kube-subnet-manager":"true","flannel.alpha.coreos.com/public-ip":"10.99.99.3","k3s.io/hostname":"chunk01","k3s.io/internal-ip":"10.99.99.3","k3s.io/node-args":"[\"server\",\"--token\",\"********\",\"--flannel-backend\",\"host-gw\",\"--flannel-iface\",\"wg0\",\"--node-ip\",\"10.99.99.3\",\"--bind-address\",\"10.99.99.3\",\"--kubelet-arg\",\"allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding,net.core.somaxconn,net.ipv4.tcp_max_syn_backlog\"]","k3s.io/node-config-hash":"IOSXWZITEQ256E6O2AAIPGV5WYEFL3IEWCNHDPOC1SBYOC5EDV3RA====","k3s.io/node-env":"{}","node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}
{"alpha.kubernetes.io/provided-node-ip":"10.99.99.1","flannel.alpha.coreos.com/backend-data":"null","flannel.alpha.coreos.com/backend-type":"host-gw","flannel.alpha.coreos.com/kube-subnet-manager":"true","flannel.alpha.coreos.com/public-ip":"192.227.175.204","k3s.io/hostname":"linux.53247a.com","k3s.io/internal-ip":"10.99.99.1","k3s.io/node-args":"[\"agent\",\"--server\",\"https://10.99.99.3:6443\",\"--token\",\"********\",\"--kubelet-arg\",\"allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding,net.core.somaxconn,net.ipv4.tcp_max_syn_backlog\",\"--node-ip\",\"10.99.99.1\"]","k3s.io/node-config-hash":"XMULZZBZM75HQFXUD6ATULQCYUYPOWQMMPG2AZH34ZSGTIEK35V6A====","k3s.io/node-env":"{\"K3S_TOKEN\":\"********\",\"K3S_URL\":\"https://191.96.11.162:6443\"}","node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}
{"alpha.kubernetes.io/provided-node-ip":"10.99.99.2","flannel.alpha.coreos.com/backend-data":"null","flannel.alpha.coreos.com/backend-type":"host-gw","flannel.alpha.coreos.com/kube-subnet-manager":"true","flannel.alpha.coreos.com/public-ip":"107.173.182.29","k3s.io/hostname":"linux.53247b.com","k3s.io/internal-ip":"10.99.99.2","k3s.io/node-args":"[\"agent\",\"--server\",\"https://10.99.99.3:6443\",\"--token\",\"********\",\"--kubelet-arg\",\"allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding,net.core.somaxconn,net.ipv4.tcp_max_syn_backlog\",\"--node-ip\",\"10.99.99.2\"]","k3s.io/node-config-hash":"6HAIZYGWKSMJKPEWE2CMRZTEEE2ZXYLE6CEGC2NQJDUDUMH35XONA====","k3s.io/node-env":"{\"K3S_TOKEN\":\"********\",\"K3S_URL\":\"https://191.96.11.162:6443\"}","node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}WG的部份，必需全部打通，不是只有主、副通而已，這個設定如果機器多要花很多時間
先從master 開始
清理 Systemd 殘留：
- 檢查
/etc/systemd/system/k3s.service，確保ExecStart後方只有乾淨的server \，沒有寫死任何舊公網 IP 參數。 - 檢查並清空
/etc/systemd/system/k3s.service.env中帶有舊公網 IP 的環境變數。
寫入正確的 /etc/rancher/k3s/config.yaml：
token: "您的TOKEN"
flannel-backend: "host-gw"
flannel-iface: "wg0"
node-ip: "10.99.99.3"
bind-address: "10.99.99.3" # 強制 API Server 監聽 WireGuard 網卡
tls-san:
- "127.0.0.1"
- "localhost"
- "10.99.99.3"
kubelet-arg:
- "allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding,net.core.somaxconn,net.ipv4.tcp_max_syn_backlog"
大清理1
sudo systemctl stop k3s
sudo killall -9 k3s containerd-shim containerd 2>/dev/null
sudo rm -rf /var/lib/rancher/k3s/server/tls/* # 粉碎舊憑證，強迫重新製證
sudo rm -rf /var/lib/rancher/k3s/agent/patched-certs/*
sudo rm -rf /run/flannel/*
sudo ip link delete flannel.1 2>/dev/null # 刪除舊 VXLAN 網卡
大清理2
sudo systemctl daemon-reload && sudo systemctl start k3s
sudo sed -i 's/127.0.0.1/10.99.99.3/g' /etc/rancher/k3s/k3s.yaml
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
再來改舊的worker 我只有2台，記得照正確的替換上去
編輯 /etc/systemd/system/k3s-agent.service.env，將舊公網 K3S_URL 修正為內網地址
K3S_URL=https://10.99.99.3:6443更新 /etc/rancher/k3s/config.yaml
server: "https://10.99.99.3:6443" # 永遠認準內網 Master
token: "您的TOKEN"
node-ip: "10.99.99.1" # 另一台填 10.99.99.2 (對齊自己 wg0 的 IP)
kubelet-arg:
- "allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding,net.core.somaxconn,net.ipv4.tcp_max_syn_backlog"
整理一下
sudo systemctl stop k3s-agent
sudo ip link delete flannel.1 2>/dev/null
sudo rm -rf /var/lib/rancher/k3s/agent/patched-certs
sudo rm -rf /run/flannel/
sudo systemctl daemon-reload && sudo systemctl start k3s-agent
因為上面有跑一些服務沒停，所以也需要整理一下，讓它們重啟
# 在 Master 執行，強制重構所有業務 Pod、ArgoCD 與系統組件
kubectl delete pods --all -n default
kubectl delete pods --all -n argocd
kubectl delete pods --all -n kube-system
最後檢查 先看有沒有成功替換成內網IP，再來看POD有沒有都起來?
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
chunk01 Ready control-plane 15d v1.36.3+k3s1 10.99.99.3 <none> Debian GNU/Linux 12 (bookworm) 6.17.7-joeyblog-bbrv3 (amd64) containerd://2.3.2-k3s2
linux.53247a.com Ready <none> 15d v1.36.3+k3s1 10.99.99.1 <none> Debian GNU/Linux 12 (bookworm) 6.18.18-x64v3-xanmod1 (amd64) containerd://2.3.2-k3s2
linux.53247b.com Ready <none> 15d v1.36.3+k3s1 10.99.99.2 <none> Debian GNU/Linux 12 (bookworm) 6.18.18-x64v3-xanmod1 (amd64) containerd://2.3.2-k3s2
kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
argocd argocd-application-controller-0 1/1 Running 0 2m41s
argocd argocd-applicationset-controller-579c4c54b8-c2tbx 1/1 Running 0 2m44s
argocd argocd-dex-server-744c5d4467-rszbz 1/1 Running 0 2m44s
argocd argocd-notifications-controller-dd4ff84c-6fckz 1/1 Running 0 2m44s
argocd argocd-redis-7c85dbf67b-t2lft 1/1 Running 0 2m44s
argocd argocd-repo-server-78666d6999-2kxrc 1/1 Running 0 2m43s
argocd argocd-server-7d95c55dd-jblpg 1/1 Running 0 2m43s
default app-6695dc87c5-g47rc 1/1 Running 0 3m26s
default app-6695dc87c5-hlt89 1/1 Running 0 3m25s
default gitea-c6d9dc9fc-rzscq 1/1 Running 0 3m26s
default gitea-runner-78c8f68958-9vtpz 1/1 Running 1 (3m23s ago) 3m26s
default light-prometheus-555b45dd5c-npvr7 1/1 Running 0 3m26s
default pgbouncer-dnc86 1/1 Running 0 2m54s
default pgbouncer-m8t69 1/1 Running 0 2m55s
default redis-4hj7g 1/1 Running 0 3m24s
default redis-8sn2q 1/1 Running 0 3m24s
default telegram-reports-unified-29791680-jtcgj 0/1 ErrImageNeverPull 0 37s
default waf-2wqkr 1/1 Running 0 3m23s
default waf-c4xt5 1/1 Running 0 3m23s
keda keda-metrics-apiserver-585f4dc5dd-vjrlj 1/1 Running 1 (32m ago) 3d19h
keda keda-operator-848d96d78c-jxw4m 1/1 Running 2 (177m ago) 3d19h
kube-system coredns-c578f7647-nf749 1/1 Running 0 2m29s
kube-system local-path-provisioner-6fbf84fdd9-mrlzr 1/1 Running 0 2m29s
kube-system local-registry-fc6bb97c7-d79d2 1/1 Running 0 2m29s
kube-system metrics-server-6dc596dfb8-hhk7p 1/1 Running 0 2m29s之所以換這個是因為最省資源了，Calico 如果我要用"關閉 IP-in-IP 或 VXLAN 封裝"，其實就和HOST-GW一樣效率；Cilium 效能更佳，但我的worker 記憶體讓不出這幾百MB。後續如果有高階一點機器再來試。主要我這三台機器也都是跨WAN端，原先就不是最佳條件了。maser 和 worker 還跨不同的VPS廠商。