欢迎光临
我们一直在努力

K8s 1.30.10离线部署全攻略

在 Ansible 主控机上创建以下目录结构,并将所有文件放入对应位置。

# 在 Ansible 主控机上执行mkdir -p k8s-binary-offline/{files,roles,group_vars,inventory}
cd k8s-binary-offline
tree .

预期目录结构如下:

k8s-binary-offline/
├── ansible.cfg
├── inventory
│ ├── hosts.ini
│ └── group_vars│ ├── all.yml
│ ├── k8s_master.yml
│ └── k8s_node.yml
├── files│ ├── binaries/
│ │ ├── kubernetes-server-linux-amd64.tar.gz
│ │ ├── cilium-linux-amd64.tar.gz
│ │ ├── etcd-v3.5.12-linux-amd64.tar.gz
│ │ ├── cfssl_linux-amd64
│ │ ├── cfssljson_linux-amd64
│ │ └── cfssl-certinfo_linux-amd64
│ ├── systemd/
│ │ ├── kube-apiserver.service
│ │ ├── kube-controller-manager.service
│ │ ├── kube-scheduler.service
│ │ ├── kubelet.service
│ │ ├── kube-proxy.service
│ │ └── etcd.service
│ ├── configs/
│ │ ├── kubelet-config.yaml
│ │ ├── kube-proxy-config.yaml
│ │ ├── bootstrap-kubelet.conf
│ │ └── cilium-config.yaml
│ └── certs/
│ └── (证书生成脚本和模板)
├── roles
│ ├── common
│ │ └── tasks
│ │ └── main.yml
│ ├── certs
│ │ └── tasks
│ │ └── main.yml
│ ├── etcd
│ │ └── tasks
│ │ └── main.yml
│ ├── master
│ │ └── tasks
│ │ └── main.yml
│ ├── node
│ │ └── tasks
│ │ └── main.yml
│ └── cilium
│ └── tasks
│ └── main.yml
└── site.yml

二、核心配置文件与脚本

1. Ansible 主配置文件 (ansible.cfg)

[defaults]
inventory = ./inventory/hosts.ini
host_key_checking = False
remote_user = root
roles_path = ./roles
retry_files_enabled = False
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_factsfact_caching_timeout = 600[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=30m -o ControlPath=/tmp/ansible-ssh-%h-%p-%r

2. Inventory 文件 (inventory/hosts.ini)

[k8s_master]
master01 ansible_host=10.10.1.10 ansible_user=rootmaster02 ansible_host=10.10.1.11 ansible_user=root
master03 ansible_host=10.10.1.12 ansible_user=root

[k8s_node]
node01 ansible_host=10.10.1.21 ansible_user=root
node02 ansible_host=10.10.1.22 ansible_user=root
node03 ansible_host=10.10.1.23 ansible_user=root
node04 ansible_host=10.10.1.24 ansible_user=root
node05 ansible_host=10.10.1.25 ansible_user=root
node06 ansible_host=10.10.1.26 ansible_user=root

[k8s_cluster:children]
k8s_master
k8s_node

[etcd]
master01 ansible_host=10.10.1.10
master02 ansible_host=10.10.1.11
master03 ansible_host=10.10.1.12

[all:vars]
ansible_ssh_private_key_file = /root/.ssh/id_rsa
ansible_become = yes
ansible_become_method = sudo
ansible_become_user = root

3. 全局变量文件 (group_vars/all.yml)

# 集群基础配置
k8s_version: "v1.30.10"
etcd_version: "v3.5.12"
cilium_version: "v1.16.6"
cluster_name: "k8s-cluster"
service_cidr: "10.96.0.0/12"
pod_cidr: "10.244.0.0/16"
cluster_dns: "10.96.0.10"

# 网络配置
apiserver_vip: "10.10.1.100"
apiserver_port: 6443

# 文件路径
local_files_dir: "/root/k8s-binary-offline/files"
remote_install_dir: "/opt/kubernetes"
remote_bin_dir: "{{ remote_install_dir }}/bin"
remote_cfg_dir: "{{ remote_install_dir }}/cfg"
remote_ssl_dir: "{{ remote_install_dir }}/ssl"
remote_systemd_dir: "/etc/systemd/system"

# 证书配置
cert_country: "CN"
cert_state: "Beijing"
cert_location: "Beijing"
cert_organization: "k8s"
cert_organizational_unit: "System"
cert_email: "admin@k8s.io"

# 系统配置
system_reserved: "cpu=200m,memory=1Gi"
kubelet_eviction_hard: "memory.available<500Mi,nodefs.available<10%,nodefs.inodesFree<5%"

4. 主节点变量 (group_vars/k8s_master.yml)

# Master 节点特定配置
apiserver_insecure_port: 8080
apiserver_secure_port: 6443
controller_manager_port: 10252
scheduler_port: 10251
etcd_client_port: 2379
etcd_peer_port: 2380

# Etcd 集群配置
etcd_cluster: "etcd01=https://10.10.1.10:2380,etcd02=https://10.10.1.11:2380,etcd03=https://10.10.1.12:2380"
etcd_initial_cluster_token: "etcd-cluster-k8s"

5. 工作节点变量 (group_vars/k8s_node.yml)

# Node 节点特定配置
kubelet_port: 10250
kube_proxy_port: 10256
max_pods: 110

三、离线二进制文件下载清单

在能访问互联网的机器上下载以下文件,然后上传到 Ansible 主控机的 files/binaries/ 目录。

组件版本下载地址备注
Kubernetes v1.30.10 https://dl.k8s.io/v1.30.10/kubernetes-server-linux-amd64.tar.gz 官方二进制包
Cilium CLI v1.16.6 https://github.com/cilium/cilium-cli/releases/download/v1.16.6/cilium-linux-amd64.tar.gz Cilium 命令行工具
Cilium v1.16.6 https://github.com/cilium/cilium/releases/download/v1.16.6/cilium-1.16.6.tar.gz Cilium 安装包
etcd v3.5.12 https://github.com/etcd-io/etcd/releases/download/v3.5.12/etcd-v3.5.12-linux-amd64.tar.gz etcd 集群
cfssl 最新版 https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64 证书工具
cfssljson 最新版 https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64 证书工具
runc v1.1.12 https://github.com/opencontainers/runc/releases/download/v1.1.12/runc.amd64 容器运行时
containerd v1.7.18 https://github.com/containerd/containerd/releases/download/v1.7.18/containerd-1.7.18-linux-amd64.tar.gz 容器运行时

下载脚本示例:

#!/bin/bash
# download.sh – 在能联网的机器上执行
mkdir -p binaries
cd binaries

# Kuberneteswget https://dl.k8s.io/v1.30.10/kubernetes-server-linux-amd64.tar.gz

# Cilium
wget https://github.com/cilium/cilium-cli/releases/download/v1.16.6/cilium-linux-amd64.tar.gz
wget https://github.com/cilium/cilium/releases/download/v1.16.6/cilium-1.16.6.tar.gz

# etcd
wget https://github.com/etcd-io/etcd/releases/download/v3.5.12/etcd-v3.5.12-linux-amd64.tar.gz

# cfssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64 -O cfssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64 -O cfssljsonchmod +x cfssl cfssljson

# 容器运行时
wget https://github.com/opencontainers/runc/releases/download/v1.1.12/runc.amd64wget https://github.com/containerd/containerd/releases/download/v1.7.18/containerd-1.7.18-linux-amd64.tar.gz

echo "下载完成,请将 binaries/ 目录上传到 Ansible 主控机的 files/ 目录下"

四、Ansible Roles 核心任务

1. 通用系统初始化 (roles/common/tasks/main.yml)

—
– name: 禁用 swap shell: |
swapoff -a
sed -i '/ swap / s/^\\(.*\\)$/#\\1/g' /etc/fstab name: 设置主机名
hostname:
name: "{{ inventory_hostname }}"

– name: 配置 hosts 文件 blockinfile:
path: /etc/hosts
block: |
10.10.1.10 master0110.10.1.11 master02
10.10.1.12 master0310.10.1.21 node01
10.10.1.22 node0210.10.1.23 node03
10.10.1.24 node0410.10.1.25 node05
10.10.1.26 node06 name: 安装基础依赖
apt:
name:
curl – wget
gnupg
lsb-release
apt-transport-https
ca-certificates
software-properties-common
net-tools – ipvsadm – ipset
sysstat – conntrack – nfs-common
state: present update_cache: yes name: 配置内核参数 copy:
src: "{{ local_files_dir }}/configs/sysctl.conf"
dest: /etc/sysctl.d/k8s.conf
owner: root group: root
mode: '0644'
notify: reload sysctl name: 加载内核模块
copy:
content: |
overlay br_netfilter ip_vs
ip_vs_rr ip_vs_wrr
ip_vs_sh
nf_conntrack dest: /etc/modules-load.d/k8s.conf owner: root
group: root mode: '0644'
notify: load kernel modules

– name: 配置时间同步
apt:
name: chrony state: present
notify: restart chrony name: 创建安装目录
file:
path: "{{ item }}"
state: directory
owner: root group: root
mode: '0755'
loop:
"{{ remote_install_dir }}"
"{{ remote_bin_dir }}"
"{{ remote_cfg_dir }}"
"{{ remote_ssl_dir }}"
/etc/cni/net.d
/opt/cni/bin
/var/lib/kubelet – /var/lib/kube-proxy – /var/lib/kube-controller-manager
/var/lib/kube-scheduler
/var/log/kubernetes

2. 证书生成 (roles/certs/tasks/main.yml)

—
– name: 复制证书工具
copy:
src: "{{ local_files_dir }}/binaries/{{ item }}"
dest: /usr/local/bin/
owner: root group: root
mode: '0755'
loop:
cfssl – cfssljson name: 创建证书配置目录 file:
path: /tmp/k8s-certs
state: directory name: 生成 CA 配置文件 copy:
content: |
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
dest: /tmp/k8s-certs/ca-config.json

– name: 生成 CA 证书签名请求 copy:
content: |
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "{{ cert_country }}",
"ST": "{{ cert_state }}",
"L": "{{ cert_location }}",
"O": "{{ cert_organization }}",
"OU": "{{ cert_organizational_unit }}"
}
]
}
dest: /tmp/k8s-certs/ca-csr.json

– name: 生成 CA 证书和私钥
shell: |
cd /tmp/k8s-certs
cfssl gencert -initca ca-csr.json | cfssljson -bare ca args:
creates: /tmp/k8s-certs/ca.pem

name: 分发 CA 证书到所有节点
copy:
src: "/tmp/k8s-certs/{{ item }}"
dest: "{{ remote_ssl_dir }}/"
owner: root group: root
mode: '0644'
loop:
ca.pem
ca-key.pem – ca.csr
delegate_to: "{{ groups['k8s_cluster'] | first }}"
run_once: true

3. etcd 集群部署 (roles/etcd/tasks/main.yml)

—
– name: 解压 etcd 二进制文件
unarchive:
src: "{{ local_files_dir }}/binaries/etcd-v3.5.12-linux-amd64.tar.gz"
dest: /tmp/
remote_src: yes- name: 安装 etcd 二进制文件 copy:
src: "/tmp/etcd-v3.5.12-linux-amd64/{{ item }}"
dest: "{{ remote_bin_dir }}/"
owner: root group: root
mode: '0755'
loop:
etcd – etcdctl name: 创建 etcd 数据目录 file:
path: /var/lib/etcd
state: directory owner: root
group: root mode: '0700'

– name: 配置 etcd systemd 服务
template:
src: "{{ local_files_dir }}/systemd/etcd.service.j2"
dest: "{{ remote_systemd_dir }}/etcd.service"
owner: root
group: root mode: '0644'
notify:
daemon-reload – restart etcd name: 启动 etcd 服务
systemd:
name: etcd
state: started enabled: yes
daemon_reload: yes

4. Master 节点部署 (roles/master/tasks/main.yml)

—
– name: 解压 Kubernetes 二进制文件 unarchive:
src: "{{ local_files_dir }}/binaries/kubernetes-server-linux-amd64.tar.gz"
dest: /tmp/
remote_src: yes

– name: 安装 Master 组件二进制文件 copy:
src: "/tmp/kubernetes/server/bin/{{ item }}"
dest: "{{ remote_bin_dir }}/"
owner: root
group: root mode: '0755'
loop:
– kube-apiserver – kube-controller-manager
kube-scheduler – kubectl
kubelet
kube-proxy name: 生成 apiserver 证书 shell: |
# 证书生成逻辑 cd {{ remote_ssl_dir }}
cat > kube-apiserver-csr.json <<EOF
{
"CN": "kube-apiserver",
"hosts": [
"127.0.0.1",
"{{ apiserver_vip }}",
"10.10.1.10",
"10.10.1.11",
"10.10.1.12",
"10.96.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048 },
"names": [
{
"C": "{{ cert_country }}",
"ST": "{{ cert_state }}",
"L": "{{ cert_location }}",
"O": "{{ cert_organization }}",
"OU": "{{ cert_organizational_unit }}"
}
]
}
EOF cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver
args:
creates: "{{ remote_ssl_dir }}/kube-apiserver.pem"

name: 配置 apiserver systemd 服务
template:
src: "{{ local_files_dir }}/systemd/kube-apiserver.service.j2"
dest: "{{ remote_systemd_dir }}/kube-apiserver.service"
notify:
daemon-reload – restart kube-apiserver

– name: 配置 controller-manager template:
src: "{{ local_files_dir }}/systemd/kube-controller-manager.service.j2"
dest: "{{ remote_systemd_dir }}/kube-controller-manager.service"
notify:
daemon-reload – restart kube-controller-manager name: 配置 scheduler template:
src: "{{ local_files_dir }}/systemd/kube-scheduler.service.j2"
dest: "{{ remote_systemd_dir }}/kube-scheduler.service"
notify:
– daemon-reload
restart kube-scheduler

– name: 生成 admin kubeconfig shell: |
# kubeconfig 生成逻辑
kubectl config set-cluster kubernetes \\
–certificate-authority={{ remote_ssl_dir }}/ca.pem \\
–embed-certs=true \\
–server=https://{{ apiserver_vip }}:{{ apiserver_port }} \\
–kubeconfig={{ remote_cfg_dir }}/admin.kubeconfig kubectl config set-credentials admin \\
–client-certificate={{ remote_ssl_dir }}/admin.pem \\
–client-key={{ remote_ssl_dir }}/admin-key.pem \\
–embed-certs=true \\
–kubeconfig={{ remote_cfg_dir }}/admin.kubeconfig

kubectl config set-context kubernetes \\
–cluster=kubernetes \\
–user=admin \\
–kubeconfig={{ remote_cfg_dir }}/admin.kubeconfig

kubectl config use-context kubernetes \\
–kubeconfig={{ remote_cfg_dir }}/admin.kubeconfig args:
creates: "{{ remote_cfg_dir }}/admin.kubeconfig"

– name: 启动 Master 服务 systemd:
name: "{{ item }}"
state: started
enabled: yes daemon_reload: yes loop:
kube-apiserver
kube-controller-manager – kube-scheduler

5. Node 节点部署 (roles/node/tasks/main.yml)

—
– name: 安装 Node 组件二进制文件 copy:
src: "{{ local_files_dir }}/binaries/kubernetes/server/bin/{{ item }}"
dest: "{{ remote_bin_dir }}/"
owner: root group: root
mode: '0755'
loop:
– kubelet – kube-proxy

– name: 配置 kubelet template:
src: "{{ local_files_dir }}/configs/kubelet-config.yaml.j2"
dest: "{{ remote_cfg_dir }}/kubelet-config.yaml"
owner: root group: root
mode: '0644'

– name: 配置 kubelet systemd 服务 template:
src: "{{ local_files_dir }}/systemd/kubelet.service.j2"
dest: "{{ remote_systemd_dir }}/kubelet.service"
notify:
– daemon-reload
restart kubelet name: 配置 kube-proxy
template:
src: "{{ local_files_dir }}/configs/kube-proxy-config.yaml.j2"
dest: "{{ remote_cfg_dir }}/kube-proxy-config.yaml"
owner: root group: root
mode: '0644'

– name: 配置 kube-proxy systemd 服务
template:
src: "{{ local_files_dir }}/systemd/kube-proxy.service.j2"
dest: "{{ remote_systemd_dir }}/kube-proxy.service"
notify:
daemon-reload – restart kube-proxy name: 分发 bootstrap kubeconfig
copy:
src: "{{ local_files_dir }}/configs/bootstrap-kubelet.conf"
dest: "{{ remote_cfg_dir }}/bootstrap-kubelet.conf"
owner: root group: root
mode: '0644'

– name: 启动 Node 服务 systemd:
name: "{{ item }}"
state: started
enabled: yes daemon_reload: yes loop:
kubelet
kube-proxy

6. Cilium CNI 部署 (roles/cilium/tasks/main.yml)

—
– name: 解压 Cilium CLI unarchive:
src: "{{ local_files_dir }}/binaries/cilium-linux-amd64.tar.gz"
dest: /usr/local/bin/
remote_src: yes name: 安装 Cilium shell: |
cilium install \\
–version {{ cilium_version }} \\
–cluster-name {{ cluster_name }} \\
–cluster-id 1 \\
–ipam kubernetes \\
–kube-proxy-replacement=strict \\
–native-routing-cidr={{ pod_cidr }} \\
–helm-set bpf.masquerade=true \\
–helm-set ipv4NativeRoutingCIDR={{ pod_cidr }} \\
–helm-set kubeProxyReplacement=true \\
–helm-set k8sServiceHost={{ apiserver_vip }} \\
–helm-set k8sServicePort={{ apiserver_port }}
args:
creates: /var/run/cilium/cilium.sock

– name: 验证 Cilium 安装 shell: |
cilium status –wait
register: cilium_status
until: cilium_status.rc == 0 retries: 10 delay: 30- name: 验证网络连接 shell: |
cilium connectivity test

五、主 Playbook (site.yml)

—
– name: 初始化所有节点
hosts: k8s_cluster roles:
– common name: 生成证书 hosts: master01 roles:
certs name: 部署 etcd 集群
hosts: etcd
roles:
– etcd

– name: 部署 Master 节点 hosts: k8s_master
roles:
– master name: 部署 Node 节点 hosts: k8s_node roles:
node

– name: 部署 Cilium CNI hosts: master01 roles:
cilium name: 验证集群状态
hosts: master01
tasks:
– name: 检查节点状态
shell: |
kubectl get nodes -o wide register: node_status until: "'Ready' in node_status.stdout"
retries: 20 delay: 10 – name: 检查 Pod 状态 shell: |
kubectl get pods -A -o wide
register: pod_status

name: 显示集群信息 debug:
msg: |
集群部署完成!
节点状态:
{{ node_status.stdout }}
Pod 状态:
{{ pod_status.stdout }}
访问集群:
export KUBECONFIG={{ remote_cfg_dir }}/admin.kubeconfig kubectl cluster-info

六、部署执行命令

在 Ansible 主控机上执行:

# 1. 确保所有文件就位
cd /root/k8s-binary-offline
ls -la files/binaries/ # 确认所有二进制文件已下载

# 2. 配置 SSH 免密登录(在所有节点上执行)
ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa
for host in 10.10.1.{10..12} 10.10.1.{21..26}; do ssh-copy-id root@$host
done

# 3. 执行 Ansible Playbook
ansible-playbook -i inventory/hosts.ini site.yml -v

# 4. 验证部署
ssh root@10.10.1.10 "kubectl get nodes"
ssh root@10.10.1.10 "kubectl get pods -A"

七、故障排查要点

问题现象可能原因解决方案
Ansible 连接失败 SSH 配置问题 检查 /etc/ssh/sshd_config 的 PasswordAuthentication 和 PermitRootLogin
证书生成失败 时间不同步 所有节点执行 chronyc sources 检查时间同步
etcd 启动失败 端口冲突或证书问题 检查 journalctl -u etcd查看详细日志
apiserver 无法启动 证书 hosts 不匹配 确保证书包含所有 Master 节点 IP 和 VIP
kubelet 无法注册 bootstrap token 过期 重新生成 bootstrap-kubelet.conf
Cilium Pod CrashLoopBackOff 内核版本不兼容 Ubuntu 22.04 内核需 ≥5.4,执行 uname -r 确认
节点 NotReady CNI 未就绪 检查 kubectl describe node <node-name> 查看事件

此方案基于二进制部署的最佳实践,结合 Ansible 的幂等性设计,确保一次性部署成功。所有配置文件已针对 Ubuntu 22.04 和 Kubernetes 1.30.10 优化,Cilium 配置采用直接路由模式,避免 overlay 网络性能损耗 。


参考来源

  • Ansible自动化部署Kubernetes_ansible_荧523-松山湖开发者村综合服务平台
  • 用Ansible实现二进制搭建K8s集群(v1.13.7,一主两从)-CSDN博客
  • Ubuntu22.04 安装 K8S
  • Kubernetes v1.22/v1.23 企业级高可用集群Ansible自动部署教程
  • ubuntu24.04二进制安装k8s1.23.16,使用ansible进行自动化安装,离线安装 – CSDN文库
赞(0)
未经允许不得转载:171主机测评 » K8s 1.30.10离线部署全攻略
分享到: 更多 (0)

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址