[toc]
kubernetes 操作记录五
配置网络插件flannel
在Kubernetes集群中要解决四种通信的问题;
Kubernetes网络通信:
(1) 容器间通信: 同一个Pod内的多个容器间的通信，lo
(2) Pod通信: Pod IP <—> Pod IP
(3) Pod与 Service通信: PodIP <—> ClusterIP,不在同一网段，通过iptables规则实现通信
(4) Service 与集群外部客户端的通信;
CNI: (Container Network Interface)
flannel,calico,canel,kube-router …
# kubectl get configmap kube-proxy -n kube-system -o yaml
mode: "" # 改为ipvs 就可以了
解决方案:
虚拟网桥: 纯软件的方式，实现一个虚拟网卡接到网桥上去;
多路复用: MacVlAN 配置多个Mac物理地址，使得一个物理网卡，可以承载多个容器去使用;
硬件交换: SR-IOV 单根IO虚拟化;
kubelet, /etc/cin/net.d/
# cat /etc/cni/net.d/10-flannel.conflist
{
"name": "cbr0",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
flannel:
支持多种后端
VxLAN
host-gw: Host Gateway
# kubectl get daemonset -n kube-system
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-flannel-ds 3 3 3 3 3 beta.kubernetes.io/arch=amd64 31d
kube-flannel-ds-amd64 4 4 4 4 4 beta.kubernetes.io/arch=amd64 31d
kube-flannel-ds-arm 0 0 0 0 0 beta.kubernetes.io/arch=arm 31d
kube-flannel-ds-arm64 0 0 0 0 0 beta.kubernetes.io/arch=arm64 31d
kube-flannel-ds-ppc64le 0 0 0 0 0 beta.kubernetes.io/arch=ppc64le 31d
kube-flannel-ds-s390x 0 0 0 0 0 beta.kubernetes.io/arch=s390x 31d
kube-proxy 4 4 4 4 4 <none> 31d
# kubectl get pods -n kube-system -o wide | grep "flannel"
kube-flannel-ds-amd64-45rhc 1/1 Running 2 31d 10.1.87.80 bj-zb-vm-ops-test5 <none> <none>
kube-flannel-ds-amd64-4cs6r 1/1 Running 0 31d 10.1.87.83 node03 <none> <none>
kube-flannel-ds-amd64-bst7g 1/1 Running 0 31d 10.1.87.81 node01 <none> <none>
kube-flannel-ds-amd64-gqvz2 1/1 Running 0 31d 10.1.87.82 node02 <none> <none>
kube-flannel-ds-brzp7 2/2 Running 0 31d 10.1.87.83 node03 <none> <none>
kube-flannel-ds-khpr5 2/2 Running 2 31d 10.1.87.82 node02 <none> <none>
kube-flannel-ds-q6z65 2/2 Running 0 31d 10.1.87.81 node01
kube-flannel-cfg 用来配置以上flannel Pod是如何运行的;
flannel的配置参数:
Network: flannel使用的CIDR格式的网络地址，用于为Pod的配置网络功能;
10.244.0.0/16 ->
master: 10.244.0.0/24
node01:10.244.1.0/24
…
node255: 10.244.255.0./24
SubnetLen: 把Network切分子网供各节点使用时，使用多长的掩码进行切分，默认为24位;
SubnetMin: 10.244.10.0/24 指定子网使用起始，从哪里开始;
SubnetMax: 10.244.10.0/24 指定子网使用最大限制;
Backend: 各Pod通信时使用什么方式进行通信: vxlan, host-gw , udp
测试
# kubectl get pods -o wide
# kubectl get pods -o wide | grep "myapp-deploy"
myapp-deploy-675558bfc5-947g6 1/1 Running 0 4d1h 10.244.1.173 node01 <none> <none>
myapp-deploy-675558bfc5-9xfdm 1/1 Running 0 4d6h 10.244.3.176 node03 <none> <none>
myapp-deploy-675558bfc5-qhl4w 1/1 Running 0 5h41m 10.244.2.195 node02 <none> <none>
接入一个容器ping另一容器的ip
# kubectl exec -it myapp-deploy-675558bfc5-947g6 -- /bin/sh
/ # ping 10.244.2.195
PING 10.244.2.195 (10.244.2.195): 56 data bytes
64 bytes from 10.244.2.195: seq=0 ttl=62 time=0.871 ms
64 bytes from 10.244.2.195: seq=1 ttl=62 time=0.925 ms
64 bytes from 10.244.2.195: seq=2 ttl=62 time=0.886 ms
64 bytes from 10.244.2.195: seq=3 ttl=62 time=1.171 ms
在容器所在的物理服务器上抓包,
# tcpdump -i cni0 -nn icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on cni0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:52:45.457148 IP 10.244.1.173 > 10.244.2.195: ICMP echo request, id 3328, seq 0, length 64
16:52:45.457743 IP 10.244.2.195 > 10.244.1.173: ICMP echo reply, id 3328, seq 0, length 64
16:52:46.457445 IP 10.244.1.173 > 10.244.2.195: ICMP echo request, id 3328, seq 1, length 64
16:52:46.457948 IP 10.244.2.195 > 10.244.1.173: ICMP echo reply, id 3328, seq 1, length 64
16:52:47.457722 IP 10.244.1.173 > 10.244.2.195: ICMP echo request, id 3328, seq 2, length 64
16:52:47.458313 IP 10.244.2.195 > 10.244.1.173: ICMP echo reply, id 3328, seq 2, length 64
16:52:48.458036 IP 10.244.1.173 > 10.244.2.195: ICMP echo request, id 3328, seq 3, length 64
# tcpdump -i flannel.1 -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on flannel.1, link-type EN10MB (Ethernet), capture size 262144 bytes
17:01:13.757750 IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 2, seq 37, length 64
17:01:13.758396 IP 10.244.2.195 > 10.244.1.0: ICMP echo reply, id 2, seq 37, length 64
17:01:14.758058 IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 2, seq 38, length 64
17:01:14.758791 IP 10.244.2.195 > 10.244.1.0: ICMP echo reply, id 2, seq 38, length 64
17:01:15.758376 IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 2, seq 39, length 64
17:01:15.759072 IP 10.244.2.195 > 10.244.1.0: ICMP echo reply, id 2, seq 39, length 64
17:01:16.758661 IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 2, seq 40, length 64
# tcpdump -i eth0 -nn host 10.1.87.82
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:03:44.145010 IP 10.1.87.81.44247 > 10.1.87.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 3, seq 0, length 64
17:03:44.145551 IP 10.1.87.82.8942 > 10.1.87.81.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.2.195 > 10.244.1.0: ICMP echo reply, id 3, seq 0, length 64
17:03:45.145300 IP 10.1.87.81.44247 > 10.1.87.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 3, seq 1, length 64
17:03:45.145854 IP 10.1.87.82.8942 > 10.1.87.81.8472: OTV, flags [I] (0x08), overlay 0, instance 1
Flannel VxLAN的Direct routing模式配置
# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# vim kube-flannel.yml
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan",
"Directrouting": true # 添加字段
}
}
# kubectl delete -f kube-flannel.yml
# kubectl apply -f kube-flannel.yml
# ip route show
default via 10.1.87.1 dev eth0
10.1.87.0/24 dev eth0 proto kernel scope link src 10.1.87.80
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1
10.244.1.0/24 via 10.1.87.81 dev eth0
10.244.2.0/24 via 10.1.87.82 dev eth0
10.244.3.0/24 via 10.1.87.83 dev eth0
注: 在生产环境中不可以这么做，会影响到Pod的网络环境。生产环境一般在使用前会考虑并初始化好网络环境，也就是说先修改 flannel 才开始使用创建Pod;
calico 安装使用
Canal/flannel Hosted Install 官方文档
# kubectl apply -f https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/canal/rbac.yaml
# kubectl apply -f https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/canal/canal.yaml
创建名称空间
podSelector: {} 为空时，代表所有Pod
# kubectl create namespace dev
namespace/dev created
# kubectl create namespace prod
namespace/prod created
# vim ingress-def.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all-ingress
spec:
podSelector: {}
policyTypes:
- Ingress
# kubectl apply -f ingress-def.yaml -n dev
# kubectl get netpol -n dev
NAME POD-SELECTOR AGE
deny-all-ingress <none> 93s
# vim pod-a.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod1
spec:
containers:
- name: myapp
image: ikubernetes/myapp:v1
# kubectl apply -f pod-a.yaml -n dev
# kubectl get pods -n dev -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod1 1/1 Running 0 54s 10.244.3.2 node03 <none> <none>
这是请求10.244.3.2 发现是请求不通的
# curl 10.244.3.2
curl: (7) Failed connect to 10.244.3.2:80; Connection timed out
而把 pod-a.yaml 创建在prod 名称空间，此时prod名称空间是没有定义的
# kubectl apply -f pod-a.yaml -n prod
pod/pod1 created
# kubectl get pods -n prod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod1 1/1 Running 0 17s 10.244.3.3 node03 <none> <none>node
请求10.244.3.3名称空间的可以请求通
# curl 10.244.3.3
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
当允许dev名称空间的Pod 都可以访问时，只需加 ingress: - {} 就可以了
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all-ingress
spec:
podSelector: {}
ingress:
- {}
policyTypes:
- Ingress
# kubectl apply -f ingress-def.yaml -n dev
# curl 10.244.3.2
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
定义允许访问一组Pod,使用标签来实现
# kubectl label pods pod1 app=myapp -n dev
pod/pod1 labeled
还原ingress-def 为默认，拒绝所有dev 名称空间所有Pod的访问
# vim ingress-def.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all-ingress
spec:
podSelector: {}
policyTypes:
- Ingress
# kubectl apply -f ingress-def.yaml -n dev
# curl 10.244.3.2
curl: (7) Failed connect to 10.244.3.2:80; Connection timed out
配置测略，允许某网段允许访问指定策略
# vim allow-netpol-demo.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-myapp-ingress
spec:
podSelector:
matchLabels:
app: myapp
ingress:
- from:
- ipBlock:
cidr: 10.244.0.0/16
except:
- 10.244.1.2/32
ports:
- protocol: TCP
port: 80
- protocol: TCP
port: 443
# kubectl apply -f allow-netpol-demo.yaml -n dev
# kubectl get netpol -n dev
NAME POD-SELECTOR AGE
allow-myapp-ingress app=myapp 2m
deny-all-ingress <none> 34m
# curl 10.244.3.2:80
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
管控出站流量的方式
# vim egress-def.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all-egress
spec:
podSelector: {}
policyTypes:
- Egress
# kubectl apply -f egress-def.yaml -n prod
networkpolicy.networking.k8s.io/deny-all-egress created
# kubectl apply -f pod-a.yaml -n prod # 将之前的pod-a.yaml 创建到prod名称空间
pod/pod1 unchanged
# kubectl get pods -n prod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod1 1/1 Running 0 34m 10.244.3.3 node03 <none> <none>node
# kubectl exec pod1 -it -n prod -- /bin/sh
/ # ping 10.1.87.81
PING 10.1.87.81 (10.1.87.81): 56 data bytes
^C
--- 10.1.87.81 ping statistics ---
4 packets transmitted, 0 packets received, 100% packet loss
/ #
# vim egress-def.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all-egress
spec:
podSelector: {}
egress:
- {}
policyTypes:
- Egress
# kubectl apply -f egress-def.yaml -n prod
# kubectl exec pod1 -it -n prod -- /bin/sh
/ # ping 10.1.87.81
PING 10.1.87.81 (10.1.87.81): 56 data bytes
64 bytes from 10.1.87.81: seq=0 ttl=63 time=0.660 ms
64 bytes from 10.1.87.81: seq=1 ttl=63 time=0.564 ms
^C
--- 10.1.87.81 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.564/0.612/0.660 ms
可以看到放行所有Pod出站规则,网络是通的;
网络策略:
名称空间:
拒绝所有出站，入站;
放行所有出站目标本名称空间内的所有Pod;
 - 芊芊细语&pics=/images/favicon/k8s-logo.png&summary=){kind=logo}
 - 芊芊细语&pics=/images/favicon/k8s-logo.png&summary=){kind=logo}
 - 芊芊细语&pics=/images/favicon/k8s-logo.png&summary=){kind=logo}