[toc]
kubernetes 操作记录五
配置网络插件flannel
在Kubernetes集群中要解决四种通信的问题;
Kubernetes网络通信:
(1) 容器间通信: 同一个Pod内的多个容器间的通信，lo
(2) Pod通信: Pod IP <—> Pod IP
(3) Pod与 Service通信: PodIP <—> ClusterIP,不在同一网段，通过iptables规则实现通信
(4) Service 与集群外部客户端的通信;
CNI: (Container Network Interface)
flannel,calico,canel,kube-router …
# kubectl get configmap kube-proxy -n kube-system -o yaml mode: "" # 改为ipvs 就可以了
解决方案:
虚拟网桥: 纯软件的方式，实现一个虚拟网卡接到网桥上去;
多路复用: MacVlAN 配置多个Mac物理地址，使得一个物理网卡，可以承载多个容器去使用;
硬件交换: SR-IOV 单根IO虚拟化;
kubelet, /etc/cin/net.d/
# cat /etc/cni/net.d/10-flannel.conflist { "name": "cbr0", "plugins": [ { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", "capabilities": { "portMappings": true } } ] }
flannel:
支持多种后端
VxLAN
host-gw: Host Gateway
# kubectl get daemonset -n kube-system NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE kube-flannel-ds 3 3 3 3 3 beta.kubernetes.io/arch=amd64 31d kube-flannel-ds-amd64 4 4 4 4 4 beta.kubernetes.io/arch=amd64 31d kube-flannel-ds-arm 0 0 0 0 0 beta.kubernetes.io/arch=arm 31d kube-flannel-ds-arm64 0 0 0 0 0 beta.kubernetes.io/arch=arm64 31d kube-flannel-ds-ppc64le 0 0 0 0 0 beta.kubernetes.io/arch=ppc64le 31d kube-flannel-ds-s390x 0 0 0 0 0 beta.kubernetes.io/arch=s390x 31d kube-proxy 4 4 4 4 4 <none> 31d
# kubectl get pods -n kube-system -o wide | grep "flannel" kube-flannel-ds-amd64-45rhc 1/1 Running 2 31d 10.1.87.80 bj-zb-vm-ops-test5 <none> <none> kube-flannel-ds-amd64-4cs6r 1/1 Running 0 31d 10.1.87.83 node03 <none> <none> kube-flannel-ds-amd64-bst7g 1/1 Running 0 31d 10.1.87.81 node01 <none> <none> kube-flannel-ds-amd64-gqvz2 1/1 Running 0 31d 10.1.87.82 node02 <none> <none> kube-flannel-ds-brzp7 2/2 Running 0 31d 10.1.87.83 node03 <none> <none> kube-flannel-ds-khpr5 2/2 Running 2 31d 10.1.87.82 node02 <none> <none> kube-flannel-ds-q6z65 2/2 Running 0 31d 10.1.87.81 node01
kube-flannel-cfg 用来配置以上flannel Pod是如何运行的;
flannel的配置参数:
Network: flannel使用的CIDR格式的网络地址，用于为Pod的配置网络功能;
10.244.0.0/16 ->
master: 10.244.0.0/24
node01:10.244.1.0/24
…
node255: 10.244.255.0./24
SubnetLen: 把Network切分子网供各节点使用时，使用多长的掩码进行切分，默认为24位;
SubnetMin: 10.244.10.0/24 指定子网使用起始，从哪里开始;
SubnetMax: 10.244.10.0/24 指定子网使用最大限制;
Backend: 各Pod通信时使用什么方式进行通信: vxlan, host-gw , udp
测试
# kubectl get pods -o wide # kubectl get pods -o wide | grep "myapp-deploy" myapp-deploy-675558bfc5-947g6 1/1 Running 0 4d1h 10.244.1.173 node01 <none> <none> myapp-deploy-675558bfc5-9xfdm 1/1 Running 0 4d6h 10.244.3.176 node03 <none> <none> myapp-deploy-675558bfc5-qhl4w 1/1 Running 0 5h41m 10.244.2.195 node02 <none> <none>
接入一个容器ping另一容器的ip
# kubectl exec -it myapp-deploy-675558bfc5-947g6 -- /bin/sh / # ping 10.244.2.195 PING 10.244.2.195 (10.244.2.195): 56 data bytes 64 bytes from 10.244.2.195: seq=0 ttl=62 time=0.871 ms 64 bytes from 10.244.2.195: seq=1 ttl=62 time=0.925 ms 64 bytes from 10.244.2.195: seq=2 ttl=62 time=0.886 ms 64 bytes from 10.244.2.195: seq=3 ttl=62 time=1.171 ms
在容器所在的物理服务器上抓包,
# tcpdump -i cni0 -nn icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on cni0, link-type EN10MB (Ethernet), capture size 262144 bytes 16:52:45.457148 IP 10.244.1.173 > 10.244.2.195: ICMP echo request, id 3328, seq 0, length 64 16:52:45.457743 IP 10.244.2.195 > 10.244.1.173: ICMP echo reply, id 3328, seq 0, length 64 16:52:46.457445 IP 10.244.1.173 > 10.244.2.195: ICMP echo request, id 3328, seq 1, length 64 16:52:46.457948 IP 10.244.2.195 > 10.244.1.173: ICMP echo reply, id 3328, seq 1, length 64 16:52:47.457722 IP 10.244.1.173 > 10.244.2.195: ICMP echo request, id 3328, seq 2, length 64 16:52:47.458313 IP 10.244.2.195 > 10.244.1.173: ICMP echo reply, id 3328, seq 2, length 64 16:52:48.458036 IP 10.244.1.173 > 10.244.2.195: ICMP echo request, id 3328, seq 3, length 64
# tcpdump -i flannel.1 -nn tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on flannel.1, link-type EN10MB (Ethernet), capture size 262144 bytes 17:01:13.757750 IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 2, seq 37, length 64 17:01:13.758396 IP 10.244.2.195 > 10.244.1.0: ICMP echo reply, id 2, seq 37, length 64 17:01:14.758058 IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 2, seq 38, length 64 17:01:14.758791 IP 10.244.2.195 > 10.244.1.0: ICMP echo reply, id 2, seq 38, length 64 17:01:15.758376 IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 2, seq 39, length 64 17:01:15.759072 IP 10.244.2.195 > 10.244.1.0: ICMP echo reply, id 2, seq 39, length 64 17:01:16.758661 IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 2, seq 40, length 64
# tcpdump -i eth0 -nn host 10.1.87.82 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 17:03:44.145010 IP 10.1.87.81.44247 > 10.1.87.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 3, seq 0, length 64 17:03:44.145551 IP 10.1.87.82.8942 > 10.1.87.81.8472: OTV, flags [I] (0x08), overlay 0, instance 1 IP 10.244.2.195 > 10.244.1.0: ICMP echo reply, id 3, seq 0, length 64 17:03:45.145300 IP 10.1.87.81.44247 > 10.1.87.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 3, seq 1, length 64 17:03:45.145854 IP 10.1.87.82.8942 > 10.1.87.81.8472: OTV, flags [I] (0x08), overlay 0, instance 1
Flannel VxLAN的Direct routing模式配置
# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml # vim kube-flannel.yml net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "vxlan", "Directrouting": true # 添加字段 } } # kubectl delete -f kube-flannel.yml # kubectl apply -f kube-flannel.yml # ip route show default via 10.1.87.1 dev eth0 10.1.87.0/24 dev eth0 proto kernel scope link src 10.1.87.80 10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1 10.244.1.0/24 via 10.1.87.81 dev eth0 10.244.2.0/24 via 10.1.87.82 dev eth0 10.244.3.0/24 via 10.1.87.83 dev eth0
注: 在生产环境中不可以这么做，会影响到Pod的网络环境。生产环境一般在使用前会考虑并初始化好网络环境，也就是说先修改 flannel 才开始使用创建Pod;
calico 安装使用
Canal/flannel Hosted Install 官方文档
# kubectl apply -f https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/canal/rbac.yaml # kubectl apply -f https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/canal/canal.yaml
创建名称空间
podSelector: {} 为空时，代表所有Pod
# kubectl create namespace dev namespace/dev created # kubectl create namespace prod namespace/prod created # vim ingress-def.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: deny-all-ingress spec: podSelector: {} policyTypes: - Ingress # kubectl apply -f ingress-def.yaml -n dev # kubectl get netpol -n dev NAME POD-SELECTOR AGE deny-all-ingress <none> 93s
# vim pod-a.yaml apiVersion: v1 kind: Pod metadata: name: pod1 spec: containers: - name: myapp image: ikubernetes/myapp:v1 # kubectl apply -f pod-a.yaml -n dev # kubectl get pods -n dev -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod1 1/1 Running 0 54s 10.244.3.2 node03 <none> <none>
这是请求10.244.3.2 发现是请求不通的
# curl 10.244.3.2 curl: (7) Failed connect to 10.244.3.2:80; Connection timed out
而把 pod-a.yaml 创建在prod 名称空间，此时prod名称空间是没有定义的
# kubectl apply -f pod-a.yaml -n prod pod/pod1 created # kubectl get pods -n prod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod1 1/1 Running 0 17s 10.244.3.3 node03 <none> <none>node
请求10.244.3.3名称空间的可以请求通
# curl 10.244.3.3 Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
当允许dev名称空间的Pod 都可以访问时，只需加 ingress: - {} 就可以了
apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: deny-all-ingress spec: podSelector: {} ingress: - {} policyTypes: - Ingress # kubectl apply -f ingress-def.yaml -n dev # curl 10.244.3.2 Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
定义允许访问一组Pod,使用标签来实现
# kubectl label pods pod1 app=myapp -n dev pod/pod1 labeled
还原ingress-def 为默认，拒绝所有dev 名称空间所有Pod的访问
# vim ingress-def.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: deny-all-ingress spec: podSelector: {} policyTypes: - Ingress # kubectl apply -f ingress-def.yaml -n dev # curl 10.244.3.2 curl: (7) Failed connect to 10.244.3.2:80; Connection timed out
配置测略，允许某网段允许访问指定策略
# vim allow-netpol-demo.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-myapp-ingress spec: podSelector: matchLabels: app: myapp ingress: - from: - ipBlock: cidr: 10.244.0.0/16 except: - 10.244.1.2/32 ports: - protocol: TCP port: 80 - protocol: TCP port: 443 # kubectl apply -f allow-netpol-demo.yaml -n dev # kubectl get netpol -n dev NAME POD-SELECTOR AGE allow-myapp-ingress app=myapp 2m deny-all-ingress <none> 34m # curl 10.244.3.2:80 Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
管控出站流量的方式
# vim egress-def.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: deny-all-egress spec: podSelector: {} policyTypes: - Egress # kubectl apply -f egress-def.yaml -n prod networkpolicy.networking.k8s.io/deny-all-egress created # kubectl apply -f pod-a.yaml -n prod # 将之前的pod-a.yaml 创建到prod名称空间 pod/pod1 unchanged # kubectl get pods -n prod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod1 1/1 Running 0 34m 10.244.3.3 node03 <none> <none>node # kubectl exec pod1 -it -n prod -- /bin/sh / # ping 10.1.87.81 PING 10.1.87.81 (10.1.87.81): 56 data bytes ^C --- 10.1.87.81 ping statistics --- 4 packets transmitted, 0 packets received, 100% packet loss / #
# vim egress-def.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: deny-all-egress spec: podSelector: {} egress: - {} policyTypes: - Egress # kubectl apply -f egress-def.yaml -n prod # kubectl exec pod1 -it -n prod -- /bin/sh / # ping 10.1.87.81 PING 10.1.87.81 (10.1.87.81): 56 data bytes 64 bytes from 10.1.87.81: seq=0 ttl=63 time=0.660 ms 64 bytes from 10.1.87.81: seq=1 ttl=63 time=0.564 ms ^C --- 10.1.87.81 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.564/0.612/0.660 ms
可以看到放行所有Pod出站规则,网络是通的;
网络策略:
名称空间:
拒绝所有出站，入站;
放行所有出站目标本名称空间内的所有Pod;
 - 芊芊细语&pics=/images/favicon/k8s-logo.png&summary=){kind=logo}
 - 芊芊细语&pics=/images/favicon/k8s-logo.png&summary=){kind=logo}
 - 芊芊细语&pics=/images/favicon/k8s-logo.png&summary=){kind=logo}