使用kind搭建kubernetes

2024-01-28 13:01

kind是一个非常方便的用于创建kubernetes测试集群的工具，可以使用kind创建的集群来对kubernetes进行测试。

kind架构

kind的架构如下，它将docker容器作为一个kubernetes的"node"，并在该"node"中安装kubernetes组件。

虚拟机和kind的一个"node"之间的网络模型如下，多"node"之间的通信走bridge即可。

虚拟机到node容器使用了docker的bridge网络(见下面的kind网桥)


# docker network ls

NETWORK ID          NAME                DRIVER              SCOPE

0912809efeab        bridge              bridge              local

3b3ab0cdd358        host                host                local

198d0909f9cf        kind                bridge              local

b0e304ef061c        none                null                local

在kind的"node"内部则默认使用了ptp模型，该模型的通信机制比较简单，容器和"node"直接通过一对veth进行通信。


# cat /etc/cni/net.d/10-kindnet.conflist

{

"cniVersion": "0.3.1",

"name": "kindnet",

"plugins": [

{

"type": "ptp",

"ipMasq": false,

"ipam": {

"type": "host-local",

"dataDir": "/run/cni-ipam-state",

"routes": [

{

"dst": "0.0.0.0/0"

}

],

"ranges": [

[

{

"subnet": "10.244.0.0/24"

}

]

]

}

,

"mtu": 1500



},

{

"type": "portmap",

"capabilities": {

"portMappings": true

}

}

]

}

"node"内的路由如下：10.244.0.2~0.4的地址就是"node"中的容器的eth0网卡上的地址，通过位于"node"上的veth进行通信。但"node"中的容器并不都是使用ptp网络模型，如kubernetes的apiserver，scheduler和controller-manager就是用了host模式。


# ip route

default via 172.18.0.1 dev eth0

10.244.0.2 dev vethc85f5947 scope host

10.244.0.3 dev vetheb946a41 scope host

10.244.0.4 dev veth46c994ce scope host

172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.2

创建集群

安装kubectl

安装kind命令：


curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.9.0/kind-linux-amd64

chmod +x ./kind

mv ./kind /${some-dir-in-your-PATH}/kind

创建集群，默认的集群名称为kind，可以使用参数--name指定创建的集群的名称，多集群情况下比较有用
```
kind create cluster
```
也指定node镜像来创建集群，便于离线安装
```
kind create cluster --image kindest/node:latest
```
与集群交互：

获取集群名称，可以看到下面有两个集群
```
# kind get clusters

kind

kind-2
```

切换集群。可以使用如下命令分别切换到集群kind和kind-2


# kubectl cluster-info --context kind-kind

# kubectl cluster-info --context kind-kind-2

删除集群，如使用如下命令可以删除集群kind-2
```
kind delete cluster --name kind-2
```

将镜像加载到kind的node中

kind创建的kubernetes会使用它的node上的镜像，因此需要将将镜像加载到node中才能被kubernetes使用(当然在node中也是可以直接拉取公网镜像的)，在无法拉取公网镜像的时候可以手动将镜像load到node上使用。例如，使用如下方式可以将nginx镜像加载到名为kind的集群中：


# kind load docker-image nginx --name kind

Image: "nginx" with ID "sha256:7e4d58f0e5f3b60077e9a5d96b4be1b974b5a484f54f9393000a99f3b6816e3d" not yet present on node "kind-control-plane", loading...

配置kind集群

可以在kind创建集群的时候使用配置文件进行自定义配置。例如可以使用--config指定配置文件来创建集群：


kind create cluster --config kind-example-config.yaml

配置多节点

上面部署的kubernetes中只有一个node，可以使用配置文件部署多个节点。下面使用官方提供的默认配置文件kind-config.yaml来创建集群，该集群含3个work节点：


# this config file contains all config fields with comments

# NOTE: this is not a particularly useful config file

kind: Cluster

apiVersion: kind.x-k8s.io/v1alpha4

# patch the generated kubeadm config with some extra settings

kubeadmConfigPatches:

- |

  apiVersion: kubelet.config.k8s.io/v1beta1

  kind: KubeletConfiguration

  evictionHard:

    nodefs.available: "0%"

 # patch it further using a JSON 6902 patch

kubeadmConfigPatchesJSON6902:

- group: kubeadm.k8s.io

version: v1beta2

kind: ClusterConfiguration

patch: |

    - op: add

      path: /apiServer/certSANs/-

      value: my-hostname

 # 1 control plane node and 3 workers

nodes:

# the control plane node config

- role: control-plane

# the three workers

- role: worker

- role: worker

- role: worker

创建上述集群：


# kind create cluster --name multi-node --config=kind-config.yaml

切换到该集群：


# kubectl cluster-info --context kind-multi-node

可以看到该集群下有1个控制面node，以及3个work node：


# kubectl get node

NAME                       STATUS   ROLES    AGE     VERSION

multi-node-control-plane   Ready    master   7m57s   v1.19.1

multi-node-worker          Ready    <none>   7m21s   v1.19.1

multi-node-worker2         Ready    <none>   7m21s   v1.19.1

multi-node-worker3         Ready    <none>   7m21s   v1.19.1

多控制面

一般一个生产使用的kubernetes都会使用多个控制面来保证高可用，使用kind config可以方便地创建多控制面的kubernetes集群。使用如下命令创建一个3控制面，3 work节点的集群：


# this config file contains all config fields with comments

# NOTE: this is not a particularly useful config file

kind: Cluster

apiVersion: kind.x-k8s.io/v1alpha4

# patch the generated kubeadm config with some extra settings

kubeadmConfigPatches:

- |

  apiVersion: kubelet.config.k8s.io/v1beta1

  kind: KubeletConfiguration

  evictionHard:

    nodefs.available: "0%"

 # patch it further using a JSON 6902 patch

kubeadmConfigPatchesJSON6902:

- group: kubeadm.k8s.io

version: v1beta2

kind: ClusterConfiguration

patch: |

    - op: add

      path: /apiServer/certSANs/-

      value: my-hostname

 # 1 control plane node and 3 workers

nodes:

# the control plane node config

- role: control-plane

- role: control-plane

- role: control-plane

# the three workers

- role: worker

- role: worker

- role: worker

此时可以看到有3个控制面：


# kubectl get node

NAME                  STATUS   ROLES    AGE   VERSION

kind-control-plane    Ready    master   15m   v1.19.1

kind-control-plane2   Ready    master   14m   v1.19.1

kind-control-plane3   Ready    master   13m   v1.19.1

kind-worker           Ready    <none>   12m   v1.19.1

kind-worker2          Ready    <none>   12m   v1.19.1

kind-worker3          Ready    <none>   12m   v1.19.1

指定Kubernetes的版本

可以通过指定node的镜像版本来修改kubernetes的版本。可以在官方release页面中中查找需要镜像tag，推荐tag带上sha，如

kindest/node:v1.19.1@sha256:98cf5288864662e37115e362b23e4369c8c4a408f99cbc06e58ac30ddc721600


kind: Cluster

apiVersion: kind.x-k8s.io/v1alpha4

nodes:

- role: control-plane

image: kindest/node:v1.16.4@sha256:b91a2c2317a000f3a783489dfb755064177dbc3a0b2f4147d50f04825d016f55

- role: worker

image: kindest/node:v1.16.4@sha256:b91a2c2317a000f3a783489dfb755064177dbc3a0b2f4147d50f04825d016f55

将node的端口映射到主机

可以通过如下方式将node的端口映射到主机，将容器的80端口映射到host的80端口：


kind: Cluster

apiVersion: kind.x-k8s.io/v1alpha4

nodes:

- role: control-plane

extraPortMappings:

- containerPort: 80

hostPort: 80

listenAddress: "0.0.0.0" # Optional, defaults to "0.0.0.0"

protocol: udp # Optional, defaults to tcp

kind对cluster的更新(如启用IPv6，配置nodeport等)有一个弊端，就是只能通过重新创建集群来"更新"配置。目前官方不支持对控制面的更新操作，可以参见该issue。更多配置参见官方文档。

ingress部署

可以通过KIND的extraPortMapping配置选项来将流量从主机转发到node的ingress控制器上。

可以通过kubeadm的InitConfiguration来设置自定义的node-labels，用于ingress控制器的nodeSelector。

创建集群

使用extraPortMappings 和node-labels创建一个集群。


cat <<EOF | kind create cluster --config=-

kind: Cluster

apiVersion: kind.x-k8s.io/v1alpha4

nodes:

- role: control-plane

kubeadmConfigPatches:

- |

    kind: InitConfiguration

    nodeRegistration:

      kubeletExtraArgs:

        node-labels: "ingress-ready=true"

   extraPortMappings:

- containerPort: 80

hostPort: 80

protocol: TCP

- containerPort: 443

hostPort: 443

protocol: TCP

EOF

部署ingress控制器

kind支持的ingress控制器如下：

Ambassador
Contour
Ingress NGINX

下面部署NGINX ingress。


kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/kind/deploy.yaml

在部署ingree的过程中可能会遇到无法找到secret ingress-nginx-admission的问题，出现该问题的原因可能是因为如下两个job无法正常启动造成的，参见该issue。如果是因为无法拉取外网镜像，可以先将deploy.yaml文件下载到本地，将镜像手动加载到本地主机上，然后使用上面提到的kind load docker-image命令将镜像加载到node上即可。

测试ingress

创建如下资源：kubectl apply -f usage.yaml


kind: Pod

apiVersion: v1

metadata:

name: foo-app

labels:

app: foo

spec:

containers:

- name: foo-app

image: hashicorp/http-echo:0.2.3

args:

- "-text=foo"

---

kind: Service

apiVersion: v1

metadata:

name: foo-service

spec:

selector:

app: foo

ports:

# Default port used by the image

- port: 5678

---

kind: Pod

apiVersion: v1

metadata:

name: bar-app

labels:

app: bar

spec:

containers:

- name: bar-app

image: hashicorp/http-echo:0.2.3

args:

- "-text=bar"

---

kind: Service

apiVersion: v1

metadata:

name: bar-service

spec:

selector:

app: bar

ports:

# Default port used by the image

- port: 5678

---

apiVersion: networking.k8s.io/v1beta1

kind: Ingress

metadata:

name: example-ingress

spec:

rules:

- http:

paths:

- path: /foo

backend:

serviceName: foo-service

servicePort: 5678

- path: /bar

backend:

serviceName: bar-service

servicePort: 5678

---

在远端curl该主机所在上的foo和bar服务，可以看到网络是通的，此时走的是ingress通过kind配置extraPortMappings暴露的nodeport 80端口。


C:\Users\liuch>curl 192.168.100.11/foo

foo



C:\Users\liuch>curl 192.168.100.11/bar

bar

总结：

kind是一个非常方便的kubernetes部署工具，可以快速地部署多个kubernetes集群。但也有一些实现上的瑕疵，比如，kind不支持对集群的升级，手动加载镜像的过程也比较麻烦，但总体使用上来看，瑕不掩瑜。

FAQ：

在切换集群时出现The connection to the server localhost:8080 was refused - did you specify the right host or port?，且使用如kubectl config use-context kind-kind这样的命令也无法成功切换集群。

可以在/root/.kube/config文件中查看支持的context名称(如下面使用的context为kind-kind)，然后使用kubectl config use-context kind-kind即可:


apiVersion: v1

clusters:

- cluster:

certificate-authority-data: ...

server: https://127.0.0.1:39923

name: kind-kind

contexts:

- context:

cluster: kind-kind

user: kind-kind

name: kind-kind

current-context: kind-kind

kind: Config

preferences: {}

users:

- name: kind-kind

user:

client-certificate-data: ...

kind创建的apiservice默认地址是127.0.0.1，无法远程连接。可以使用如下方式，修改apiservice的地址和端口


kind: Cluster

apiVersion: kind.x-k8s.io/v1alpha4

networking:

# WARNING: It is _strongly_ recommended that you keep this the default

# (127.0.0.1) for security reasons. However it is possible to change this.

apiServerAddress: "192.168.118.154"

# By default the API server listens on a random open port.

# You may choose a specific port but probably don't need to in most cases.

# Using a random port makes it easier to spin up multiple clusters.

apiServerPort: 6443

如果kind无法连通，可以查看是否有其他docker network影响

链接：https://www.cnblogs.com/charlieroro/p/13711589.html

（版权归原作者所有，侵删）

微信扫码关注该文公众号作者

戳这里提交新闻线索和高质量文章给我们。

来源: qq

点击查看作者最近其他文章

	# docker network ls
	NETWORK ID NAME DRIVER SCOPE
	0912809efeab bridge bridge local
	3b3ab0cdd358 host host local
	198d0909f9cf kind bridge local
	b0e304ef061c none null local

	# cat /etc/cni/net.d/10-kindnet.conflist
	{
	"cniVersion": "0.3.1",
	"name": "kindnet",
	"plugins": [
	{
	"type": "ptp",
	"ipMasq": false,
	"ipam": {
	"type": "host-local",
	"dataDir": "/run/cni-ipam-state",
	"routes": [
	{
	"dst": "0.0.0.0/0"
	}
	],
	"ranges": [
	[
	{
	"subnet": "10.244.0.0/24"
	}
	]
	]
	}
	,
	"mtu": 1500

	},
	{
	"type": "portmap",
	"capabilities": {
	"portMappings": true
	}
	}
	]
	}

	# ip route
	default via 172.18.0.1 dev eth0
	10.244.0.2 dev vethc85f5947 scope host
	10.244.0.3 dev vetheb946a41 scope host
	10.244.0.4 dev veth46c994ce scope host
	172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.2

	curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.9.0/kind-linux-amd64
	chmod +x ./kind
	mv ./kind /${some-dir-in-your-PATH}/kind

	# kubectl cluster-info --context kind-kind
	# kubectl cluster-info --context kind-kind-2

	# kind load docker-image nginx --name kind
	Image: "nginx" with ID "sha256:7e4d58f0e5f3b60077e9a5d96b4be1b974b5a484f54f9393000a99f3b6816e3d" not yet present on node "kind-control-plane", loading...

	# this config file contains all config fields with comments
	# NOTE: this is not a particularly useful config file
	kind: Cluster
	apiVersion: kind.x-k8s.io/v1alpha4
	# patch the generated kubeadm config with some extra settings
	kubeadmConfigPatches:
	- \|
	apiVersion: kubelet.config.k8s.io/v1beta1
	kind: KubeletConfiguration
	evictionHard:
	nodefs.available: "0%"
	# patch it further using a JSON 6902 patch
	kubeadmConfigPatchesJSON6902:
	- group: kubeadm.k8s.io
	version: v1beta2
	kind: ClusterConfiguration
	patch: \|
	- op: add
	path: /apiServer/certSANs/-
	value: my-hostname
	# 1 control plane node and 3 workers
	nodes:
	# the control plane node config
	- role: control-plane
	# the three workers
	- role: worker
	- role: worker
	- role: worker

	# kubectl get node
	NAME STATUS ROLES AGE VERSION
	multi-node-control-plane Ready master 7m57s v1.19.1
	multi-node-worker Ready <none> 7m21s v1.19.1
	multi-node-worker2 Ready <none> 7m21s v1.19.1
	multi-node-worker3 Ready <none> 7m21s v1.19.1

	# kubectl get node
	NAME STATUS ROLES AGE VERSION
	kind-control-plane Ready master 15m v1.19.1
	kind-control-plane2 Ready master 14m v1.19.1
	kind-control-plane3 Ready master 13m v1.19.1
	kind-worker Ready <none> 12m v1.19.1
	kind-worker2 Ready <none> 12m v1.19.1
	kind-worker3 Ready <none> 12m v1.19.1

	cat <<EOF \| kind create cluster --config=-
	kind: Cluster
	apiVersion: kind.x-k8s.io/v1alpha4
	nodes:
	- role: control-plane
	kubeadmConfigPatches:
	- \|
	kind: InitConfiguration
	nodeRegistration:
	kubeletExtraArgs:
	node-labels: "ingress-ready=true"
	extraPortMappings:
	- containerPort: 80
	hostPort: 80
	protocol: TCP
	- containerPort: 443
	hostPort: 443
	protocol: TCP
	EOF

	kind: Pod
	apiVersion: v1
	metadata:
	name: foo-app
	labels:
	app: foo
	spec:
	containers:
	- name: foo-app
	image: hashicorp/http-echo:0.2.3
	args:
	- "-text=foo"
	---
	kind: Service
	apiVersion: v1
	metadata:
	name: foo-service
	spec:
	selector:
	app: foo
	ports:
	# Default port used by the image
	- port: 5678
	---
	kind: Pod
	apiVersion: v1
	metadata:
	name: bar-app
	labels:
	app: bar
	spec:
	containers:
	- name: bar-app
	image: hashicorp/http-echo:0.2.3
	args:
	- "-text=bar"
	---
	kind: Service
	apiVersion: v1
	metadata:
	name: bar-service
	spec:
	selector:
	app: bar
	ports:
	# Default port used by the image
	- port: 5678
	---
	apiVersion: networking.k8s.io/v1beta1
	kind: Ingress
	metadata:
	name: example-ingress
	spec:
	rules:
	- http:
	paths:
	- path: /foo
	backend:
	serviceName: foo-service
	servicePort: 5678
	- path: /bar
	backend:
	serviceName: bar-service
	servicePort: 5678
	---

	C:\Users\liuch>curl 192.168.100.11/foo
	foo

	C:\Users\liuch>curl 192.168.100.11/bar
	bar

	apiVersion: v1
	clusters:
	- cluster:
	certificate-authority-data: ...
	server: https://127.0.0.1:39923
	name: kind-kind
	contexts:
	- context:
	cluster: kind-kind
	user: kind-kind
	name: kind-kind
	current-context: kind-kind
	kind: Config
	preferences: {}
	users:
	- name: kind-kind
	user:
	client-certificate-data: ...

	kind: Cluster
	apiVersion: kind.x-k8s.io/v1alpha4
	networking:
	# WARNING: It is _strongly_ recommended that you keep this the default
	# (127.0.0.1) for security reasons. However it is possible to change this.
	apiServerAddress: "192.168.118.154"
	# By default the API server listens on a random open port.
	# You may choose a specific port but probably don't need to in most cases.
	# Using a random port makes it easier to spin up multiple clusters.
	apiServerPort: 6443