手搓跨云k3s集群

根据我个人跨云的经验来手搓的，有些改动。

准备操作

Debian环境准备

安装基础环境

apt install --no-install-recommends --no-install-suggests -y -qq nfs-common iptables conntrack jq socat bash-completion open-iscsi rsync ipset ipvsadm htop net-tools wget psmisc git curl nload ebtables ethtool
systemctl enable --now iscsid
mkdir -pv /etc/systemd/system.conf.d
cat > /etc/systemd/system.conf.d/30-k8s-ulimits.conf <<EOF
[Manager]
DefaultLimitCORE=infinity
DefaultLimitNOFILE=100000
DefaultLimitNPROC=100000
EOF

cat > /etc/modules-load.d/10-k8s-modules.conf <<EOF
br_netfilter
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF

systemctl daemon-reload
systemctl restart systemd-modules-load
systemctl restart systemd-journald

k3s准备

# 国外
wget https://github.com/k3s-io/k3s/releases/download/v1.27.4%2Bk3s1/k3s
# 国内
wget https://ghproxy.com/https://github.com/k3s-io/k3s/releases/download/v1.27.4%2Bk3s1/k3s
chmod +x k3s
mv k3s /usr/local/bin/k3s

mysql部署

version: '2.1'

services:
  mysql:
    image: docker.io/bitnami/mysql:8.0
    container_name: mysql
    ports:
      - '3306:3306'
    volumes:
      - '/k8sdata/mysql/data:/bitnami/mysql/data'
    environment:
      # ALLOW_EMPTY_PASSWORD is recommended only for development.
      - MYSQL_ROOT_PASSWORD=cholei7quohyeaduk9OhTee1faisodaikouv8eep1iuraibuegeijiew7gue5euf
      - MYSQL_DATABASE=teil4Ir5
      - MYSQL_USER=AoQui4gion
      - MYSQL_PASSWORD=shooC9iecoh8aevaingoh2aiNeim3poorahY7too6ahru8saewaih0LizieNevie
      - MYSQL_CHARACTER_SET=utf8mb4
      - MYSQL_COLLATE=utf8mb4_general_ci
    healthcheck:
      test: ['CMD', '/opt/bitnami/scripts/mysql/healthcheck.sh']
      interval: 15s
      timeout: 5s
      retries: 6

volumes:
  mysql_data:
    driver: local

启动mysql

docker-compose up -d

k3s 部署

• 虽然新版本k3s内置tailscale支持，但是我发现基于跨云tailscale组网，不支持headscale组网，且导致哪吒监控数据流量比较大，暂时就不考虑了。
• 容器的runtime最好都一致，要不都用docker，要不都是默认内置的containerd
• 数据存储可以选择熟悉的数据库，我这里选择mysql，master高可用

master节点配置

给个示例的 /etc/systemd/system/k3s.service

[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=notify
EnvironmentFile=-/etc/systemd/system/k3s.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s server \
                --tls-san t1.slb.ns.local \
                --tls-san t2.slb.ns.local \
                --tls-san t1.clb.ns.local \
                --tls-san t2.clb.ns.local \
                --datastore-endpoint mysql://root:cholei7quohyeaduk9OhTee1faisodaikouv8eep1iuraibuegeijiew7gue5euf@tcp(db.slb.ns.local:3306)/ \
                --node-external-ip <公网ip> \
                --https-listen-port 26443 \
                --cluster-cidr 10.142.0.0/16 \
                --service-cidr 10.143.0.0/16 \
                --flannel-external-ip \
                --flannel-backend wireguard-native \
                --server https://t1.slb.ns.local:26443 \
                --token we3ohCh1ahjeng6goo9hahke0Agoox5a \
                --docker \
                --multi-cluster-cidr \
                --disable-network-policy \
                --disable-helm-controller \
                --disable servicelb,traefik \
                --kube-proxy-arg "proxy-mode=ipvs" "masquerade-all=true" \
                --kube-proxy-arg "metrics-bind-address=0.0.0.0"

worker 节点配置

[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=exec
EnvironmentFile=-/etc/systemd/system/k3s.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s agent \
		--node-external-ip <节点公网ip> \
		--server https://t1.clb.ns.local:26443 \
		--token we3ohCh1ahjeng6goo9hahke0Agoox5a \
		--docker \
		--kube-proxy-arg "proxy-mode=ipvs" "masquerade-all=true" \
		--kube-proxy-arg "metrics-bind-address=0.0.0.0"

开机并启动

systemctl enable k3s --now

其他组件安装

• 跨云分布式存储
• 内网负载均衡

如果对这些有想法可以再写过文档

最后

不想手搓的可以用rancher或者 https://github.com/cnrancher/autok3s 这两个都是有UI的，我推荐用后者