最近在复习Golang，写了一个web应用，本地跑通以后想在k8s集群测试一番，苦于机器配置原因， 搭建一套完整的k8s集群还是有一些力不从心，记得之前有朋友说过docker里也可以跑k8s，于是这就折腾了一番。

今天的主角是kind,那么kind是什么呢？kind可以用来干什么呢？

一、kind介绍

kind是Kubernetes In Docker缩写，一个使用docker容器节点来运行本地的kubernetes集群工具，kind主要用于测试kubernetes本身， 适用于本地开发或者CI。

对于kind目前几乎支持kubernetes官方所有版本，对于运行时目前仅支持docker，未来会逐步支持CRI里常用运行时， 相信未来一些日子，会慢慢支持conterned。

其实，kind管理集群过程，是在使用kubeadm、kustomize等开源组件来进行工作。

废话不多说，下面我们就快速安装使用kind。

二、kind安装

由于我系统是mac osx所以，我这里直接使用brew安装：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# brew install kind
==> Downloading https://mirrors.ustc.edu.cn/homebrew-bottles/kind-0.17.0.arm64_monterey.bottle.tar.gz
Already downloaded: /Users/wanzi/Library/Caches/Homebrew/downloads/bcd419997297730492f5cebc36be86fb51f21061a6ddb2e066e1e4d8ad33ddf3--kind-0.17.0.arm64_monterey.bottle.tar.gz
==> Pouring kind-0.17.0.arm64_monterey.bottle.tar.gz
==> Caveats
zsh completions have been installed to:
  /opt/homebrew/share/zsh/site-functions
==> Summary
🍺  /opt/homebrew/Cellar/kind/0.17.0: 8 files, 8.7MB
==> Running `brew cleanup kind`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
# kind version
kind v0.17.0 go1.19.2 darwin/arm64

对于linux安装：

1
2
3

curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.17.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

当然，如果你是windows，也可以使用Chocolatey安装；或者使用二进制方式，这些官方都支持。

三、创建第一个集群

在操作之前，docker和kubectl是我们需要提前准备，具体安装我这里就先忽略。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# kind help
kind creates and manages local Kubernetes clusters using Docker container 'nodes'

Usage:
  kind [command]

Available Commands:
  build       Build one of [node-image]
  completion  Output shell completion code for the specified shell (bash, zsh or fish)
  create      Creates one of [cluster]
  delete      Deletes one of [cluster]
  export      Exports one of [kubeconfig, logs]
  get         Gets one of [clusters, nodes, kubeconfig]
  help        Help about any command
  load        Loads images into nodes
  version     Prints the kind CLI version

Flags:
  -h, --help              help for kind
      --loglevel string   DEPRECATED: see -v instead
  -q, --quiet             silence all stderr output
  -v, --verbosity int32   info log verbosity, higher value produces more output

Use "kind [command] --help" for more information about a command.

对于kind help命令，我们知道kind支持build,create,delete,get,load等操作， 对于子命令可以通过-h获取更多讯息。

kind创建一个集群， 在创建集群之前我们需要确保~/.kube目录存在，因为创建过程中，kind会将集群API地址、证书等信息写入kubeconfig。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

# kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a nice day! 👋

# kubectl get node
NAME                 STATUS   ROLES           AGE     VERSION
kind-control-plane   Ready    control-plane   9m21s   v1.25.3
# kubectl get pods -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-565d847f94-kp6l7                     1/1     Running   0          9m15s
kube-system          coredns-565d847f94-ml28b                     1/1     Running   0          9m15s
kube-system          etcd-kind-control-plane                      1/1     Running   0          9m32s
kube-system          kindnet-hfsc8                                1/1     Running   0          9m15s
kube-system          kube-apiserver-kind-control-plane            1/1     Running   0          9m31s
kube-system          kube-controller-manager-kind-control-plane   1/1     Running   0          9m33s
kube-system          kube-proxy-m9zp5                             1/1     Running   0          9m15s
kube-system          kube-scheduler-kind-control-plane            1/1     Running   0          9m33s
local-path-storage   local-path-provisioner-684f458cdd-d7f29      1/1     Running   0          9m15s

这样我们第一个集群就用kind创建完成。

四、集群操作：

1、kind自定义集群名称：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# kind create cluster --name ci-cluster
Creating cluster "ci-cluster" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-ci-cluster"
You can now use your cluster with:

2、切换集群：

1
2
3
4
5
6
7

# kubectl cluster-info --context kind-ci-cluster
Kubernetes control plane is running at https://127.0.0.1:56527
CoreDNS is running at https://127.0.0.1:56527/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Have a nice day! 👋

3、查看kind所有集群

1
2
3

# kind get clusters
ci-cluster
kind

4、导入镜像

默认kind node节点是无法读取宿主机镜像，需要手动导入到kind

node节点：

1
2
3
4

# kind load docker-image --name ci-cluster --nodes  ci-cluster-control-plane  traefik:v2.9.5
# docker exec -it ci-cluster-control-plane bash
root@ci-cluster-control-plane:/# crictl images  |grep traefik
docker.io/library/traefik                  v2.9.5               a1252ce6bfaaa       132MB

五、集群配置：

1、多节点集群

对于前面，我们介绍了kind创建集群，默认集群只有一个节点，下面介绍如何创建多个节点。

创建config.yaml内容如下：

1
2
3
4
5
6

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker

重新创建集群

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# kind create cluster --config config.yaml
# kubectl get node
NAME                 STATUS   ROLES           AGE    VERSION
kind-control-plane   Ready    control-plane   110s   v1.25.3
kind-worker          Ready    <none>          74s    v1.25.3
kind-worker2         Ready    <none>          88s    v1.25.3

kind本身也可以支持多master高可用部署，这里我主要测试就不简单介绍。

### 2、自定义kubernetes集群版本

```Yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
  images: kindest/node:v1.24.7@sha256:577c630ce8e509131eab1aea12c022190978dd2f745aac5eb1fe65c0807eb315

3、挂载宿主机目录到节点容器持久化存储

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraMounts:
  - hostPath: /Users/wanzi/tools/kind/files
    containerPath: /files
  - hostPath: /Users/wanzi/tools/kind/other-files/
    containerPath: /other-files
    readOnly: true
    selinuxRelabel: false
    propagation: None

4、端口映射

宿主机端口80映射到nodePort 30950，kind节点containerPort需要和Service的nodePort端口相同

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30950
    hostPort: 80

kind: Pod
apiVersion: v1
metadata:
  name: foo
  labels:
    app: foo
spec:
  containers:
  - name: foo
    image: hashicorp/http-echo:0.2.3
    args:
    - "-text=foo"
    ports:
    - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: foo
spec:
  type: NodePort
  ports:
  - name: http
    nodePort: 30950
    port: 5678
  selector:
    app: foo

5、自定义label

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
  extraPortMappings:
  - containerPort: 30950
    hostPort: 80
  labels:
    tier: frontend
- role: worker
  labels:
    tier: backend

6、kubeadm自定义配置：

kind用kubeadm配置集群，他会在第一个控制平面节点上运行kubeadm init,我们可以对它进行自定义：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "my-label=true"

如果你想做更多的定制，有四种配置类型可供选择kubeadm init

  • InitConfiguration
  • ClusterConfiguration
  • KubeProxyConfiguration
  • KubeletConfiguration

例如，我们可以使用 kubeadm ClusterConfiguration ( spec ) 覆盖 apiserver 标志:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
        extraArgs:
          enable-admission-plugins: NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook

在 KIND管理HA的集群中配置的每个附加节点上，KIND运行kubeadm join可以使用JoinConfiguration自定义配置：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "my-label2=true"    
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "my-label3=true"

7、配置ingress

允许本地主机通过端口80/443向Ingress控制器发出请求，并配置控制器特定label

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"    
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
1

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml

六、完整集群配置

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
#增加多结点集群,并自定义node的集群版本
- role: control-plane
  images: kindest/node:v1.24.7@sha256:577c630ce8e509131eab1aea12c022190978dd2f745aac5eb1fe65c0807eb315
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"    
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
- role: worker
  images: kindest/node:v1.24.7@sha256:577c630ce8e509131eab1aea12c022190978dd2f745aac5eb1fe65c0807eb315
  labels:
    app: front
  extraMounts:
  - hostPath: /Users/wanzi/tools/kind/wwwroot
    containerPath: /wwwroot
- role: worker
  images: kindest/node:v1.24.7@sha256:577c630ce8e509131eab1aea12c022190978dd2f745aac5eb1fe65c0807eb315
  labels:
    app: backend
  extraMounts:
  - hostPath: /Users/wanzi/tools/kind/wwwroot
    containerPath: /wwwroot
networking:
  #自定义配置APIServer和网络
  apiServerAddress: "127.0.0.1"
  apiServerPort: 6443
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"
  #disableDefaultCNI: true #默认CNI插件为kindnetd，也可以禁用使用其他插件比如calico
  kubeProxyMode: "ipvs" #设置kubeproxy模式为lvs,如果要禁用可以设置none

至此，使用kind来快速构建集群算是告一段落，平常就可以用它跑测试，验证功能了。

参考文档： https://kind.sigs.k8s.io https://github.com/kubernetes-sigs/kind