[toc]
kubernetes 操作记录二
Kubernetes Pod 控制器
Replica Set(RS) kubernetes 新一代的Pod controller
# vim rs-demo.yaml
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: myapp
namespace: default
spec:
replicas: 2
selector:
matchLabels:
app: myapp
release: canary
template:
metadata:
name: myapp-pod
labels:
app: myapp
release: canary
environment: qa
spec:
containers:
- name: myapp-container
image: ikubernetes/myapp:v1
ports:
- name: http
containerPort: 80
# kubectl get rs
NAME DESIRED CURRENT READY AGE
myapp 2 2 2 6m51s
# kubectl get pods
NAME READY STATUS RESTARTS AGE
myapp-5xbwq 1/1 Running 0 7m17s
myapp-jdzph 1/1 Running 0 7m17s
# curl 10.244.3.157
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
# kubectl delete pods myapp-5xbwq
# kubectl get pods
NAME READY STATUS RESTARTS AGE
myapp-jdzph 1/1 Running 0 10m
myapp-xg9mm 1/1 Running 0 16s
# kubectl label pods pod-demo release=canary
# kubectl get pods --show-labels
NAME READY STATUS RESTARTS AGE LABELS
myapp-jdzph 1/1 Running 0 31m app=myapp,environment=qa,release=canary
精确满足用户期望
apply -f 声明式更新，即可以创建，也可以更新
# vim deploy-demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-deploy
namespace: default
spec:
replicas: 2
selector:
matchLabels:
app: myapp
release: canary
template:
metadata:
labels:
app: myapp
release: canary
spec:
containers:
- name: myapp
image: ikubernetes/myapp:v1
ports:
- name: http
containerPort: 80
# kubectl apply -f deploy-demo.yaml
# kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
myapp-deploy 2/2 2 2 115s
# kubectl get pods
NAME READY STATUS RESTARTS AGE
myapp-deploy-67b6dfcd8-6q725 1/1 Running 0 4m39s
myapp-deploy-67b6dfcd8-mshw8 1/1 Running 0 4m39s
# vim deploy-demo.yaml
将 replicas: 2 改为 replicas: 3
# kubectl apply -f deploy-demo.yaml
# kubectl get pods
NAME READY STATUS RESTARTS AGE
myapp-deploy-67b6dfcd8-6q725 1/1 Running 0 6m34s
myapp-deploy-67b6dfcd8-gltjz 1/1 Running 0 58s
myapp-deploy-67b6dfcd8-mshw8 1/1 Running 0 6m34s
实现流动更新
# vim deploy-demo.yaml
image: ikubernetes/myapp:v1 改为 image: ikubernetes/myapp:v2
# kubcetl apply -f deploy-demo.yaml
# kubectl get pods -l app=myapp -w
NAME READY STATUS RESTARTS AGE
myapp-deploy-67b6dfcd8-6q725 1/1 Running 0 3h23m
myapp-deploy-67b6dfcd8-gltjz 1/1 Running 0 3h18m
myapp-deploy-67b6dfcd8-mshw8 1/1 Running 0 3h23m
myapp-deploy-675558bfc5-89nc7 0/1 Pending 0 0s
myapp-deploy-675558bfc5-89nc7 0/1 Pending 0 0s
myapp-deploy-675558bfc5-89nc7 0/1 ContainerCreating 0 0s
myapp-deploy-675558bfc5-89nc7 1/1 Running 0 2s
myapp-deploy-67b6dfcd8-gltjz 1/1 Terminating 0 3h18m
myapp-deploy-675558bfc5-7zht2 0/1 Pending 0 0s
myapp-deploy-675558bfc5-7zht2 0/1 Pending 0 0s
myapp-deploy-675558bfc5-7zht2 0/1 ContainerCreating 0 0s
myapp-deploy-67b6dfcd8-gltjz 0/1 Terminating 0 3h18m
myapp-deploy-675558bfc5-7zht2 1/1 Running 0 3s
myapp-deploy-67b6dfcd8-6q725 1/1 Terminating 0 3h24m
myapp-deploy-675558bfc5-xjq9z 0/1 Pending 0 0s
myapp-deploy-675558bfc5-xjq9z 0/1 Pending 0 0s
myapp-deploy-675558bfc5-xjq9z 0/1 ContainerCreating 0 0s
myapp-deploy-67b6dfcd8-gltjz 0/1 Terminating 0 3h18m
myapp-deploy-67b6dfcd8-gltjz 0/1 Terminating 0 3h18m
myapp-deploy-67b6dfcd8-6q725 0/1 Terminating 0 3h24m
myapp-deploy-675558bfc5-xjq9z 1/1 Running 0 2s
myapp-deploy-67b6dfcd8-mshw8 1/1 Terminating 0 3h24m
myapp-deploy-67b6dfcd8-6q725 0/1 Terminating 0 3h24m
myapp-deploy-67b6dfcd8-6q725 0/1 Terminating 0 3h24m
myapp-deploy-67b6dfcd8-mshw8 0/1 Terminating 0 3h24m
myapp-deploy-67b6dfcd8-mshw8 0/1 Terminating 0 3h24m
myapp-deploy-67b6dfcd8-mshw8 0/1 Terminating 0 3h24m
查看目前所有的replica set
# kubectl get rs -o wide
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
myapp-deploy-675558bfc5 3 3 3 3m19s myapp ikubernetes/myapp:v2 app=myapp,pod-template-hash=675558bfc5,release=canary
myapp-deploy-67b6dfcd8 0 0 0 3h27m myapp ikubernetes/myapp:v1 app=myapp,pod-template-hash=67b6dfcd8,release=canary
查看历史滚动信息
# kubectl rollout history deployment myapp-deploy
deployment.extensions/myapp-deploy
REVISION CHANGE-CAUSE
1 <none>
2 <none>
定义一个补丁
# kubectl patch deployment myapp-deploy -p '{"spec":{"replicas":5}}'
# kubectl get pods
NAME READY STATUS RESTARTS AGE
myapp-deploy-675558bfc5-7zht2 1/1 Running 0 11m
myapp-deploy-675558bfc5-89nc7 1/1 Running 0 11m
myapp-deploy-675558bfc5-9nkk4 1/1 Running 0 33s
myapp-deploy-675558bfc5-rbs4d 1/1 Running 0 33s
myapp-deploy-675558bfc5-xjq9z 1/1 Running 0 11m
# kubectl patch deployment myapp-deploy -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'
deployment.extensions/myapp-deploy patched
# kubectl describe deployment myapp-deploy
RollingUpdateStrategy: 0 max unavailable, 1 max surge
金丝雀更新发布
# kubectl set image deployment myapp-deploy myapp=ikubernetes/myapp:v3 && kubectl rollout pause deployment myapp-deploy # 更新并且暂停更新
deployment.extensions/myapp-deploy image updated
deployment.extensions/myapp-deploy paused
# kubectl get pods -l app=myapp -w
NAME READY STATUS RESTARTS AGE
myapp-deploy-675558bfc5-7zht2 1/1 Running 0 27m
myapp-deploy-675558bfc5-89nc7 1/1 Running 0 27m
myapp-deploy-675558bfc5-9nkk4 1/1 Running 0 17m
myapp-deploy-675558bfc5-rbs4d 1/1 Running 0 17m
myapp-deploy-675558bfc5-xjq9z 1/1 Running 0 27m
myapp-deploy-7f577979c8-92tj8 0/1 Pending 0 0s
myapp-deploy-7f577979c8-92tj8 0/1 Pending 0 0s
myapp-deploy-7f577979c8-92tj8 0/1 ContainerCreating 0 0s
myapp-deploy-7f577979c8-92tj8 1/1 Running 0 14s
使用以下命令进行监控更新
# kubectl rollout status deployment myapp-deploy
Waiting for deployment "myapp-deploy" rollout to finish: 1 out of 5 new replicas have been updated...
若发现没有问题后继续更新
# kubectl rollout resume deployment myapp-deploy
deployment.extensions/myapp-deploy resumed
# kubectl rollout status deployment myapp-deploy
Waiting for deployment "myapp-deploy" rollout to finish: 1 out of 5 new replicas have been updated...
Waiting for deployment spec update to be observed...
Waiting for deployment spec update to be observed...
Waiting for deployment "myapp-deploy" rollout to finish: 1 out of 5 new replicas have been updated...
Waiting for deployment "myapp-deploy" rollout to finish: 1 out of 5 new replicas have been updated...
Waiting for deployment "myapp-deploy" rollout to finish: 2 out of 5 new replicas have been updated...
Waiting for deployment "myapp-deploy" rollout to finish: 2 out of 5 new replicas have been updated...
Waiting for deployment "myapp-deploy" rollout to finish: 2 out of 5 new replicas have been updated...
Waiting for deployment "myapp-deploy" rollout to finish: 3 out of 5 new replicas have been updated...
Waiting for deployment "myapp-deploy" rollout to finish: 3 out of 5 new replicas have been updated...
Waiting for deployment "myapp-deploy" rollout to finish: 3 out of 5 new replicas have been updated...
Waiting for deployment "myapp-deploy" rollout to finish: 4 out of 5 new replicas have been updated...
Waiting for deployment "myapp-deploy" rollout to finish: 4 out of 5 new replicas have been updated...
Waiting for deployment "myapp-deploy" rollout to finish: 4 out of 5 new replicas have been updated...
Waiting for deployment "myapp-deploy" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "myapp-deploy" rollout to finish: 1 old replicas are pending termination...
deployment "myapp-deploy" successfully rolled out
若更新时发现问题，我们可以执行回滚
查看可以回滚的版本
# kubectl rollout history deployment myapp-deploy
deployment.extensions/myapp-deploy
REVISION CHANGE-CAUSE
1 <none>
2 <none>
3 <none>
# kubectl rollout undo deployment myapp-deploy --to-revision=1
deployment.extensions/myapp-deploy rolled back
# kubectl rollout history deployment myapp-deploy
deployment.extensions/myapp-deploy
REVISION CHANGE-CAUSE
2 <none>
3 <none>
4 <none>
这时第一版已经回滚成第一版了
# kubectl get rs -o wide
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
myapp-deploy-675558bfc5 0 0 0 42m myapp ikubernetes/myapp:v2 app=myapp,pod-template-hash=675558bfc5,release=canary
myapp-deploy-67b6dfcd8 5 5 5 4h6m myapp ikubernetes/myapp:v1 app=myapp,pod-template-hash=67b6dfcd8,release=canary
DaemonSet
DaemonSet 在整个集群的每个节点上只运行指定Pod的一个副本，用于系统级的管理功能
比如filebeat，收集日志的服务
# vim ds-demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: redis
role: logstor
template:
metadata:
labels:
app: redis
role: logstor
spec:
containers:
- name: redis
image: redis:4.0-alpine
ports:
- name: redis
containerPort: 6379
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat-ds
namespace: default
spec:
selector:
matchLabels:
app: filebeat
release: stable
template:
metadata:
labels:
app: filebeat
release: stable
spec:
containers:
- name: filebeat
image: ikubernetes/filebeat:5.6.5-alpine
env:
- name: REDIS_HOST
value: redis.default.svc.cluster.local
- name: REDIS_LOG_LEVEL
value: info
# kubectl apply -f ds-demo.yaml
# kubectl expose deployment redis --port=6379
service/redis exposed
# kubectl set image daemonsets filebeat-ds filebeat=ikubernetes/filebeat:5.6.6-alpine
daemonset.extensions/filebeat-ds image updated
Kubernetes Service
ExternalName, ClusterIP, NodePort, and LoadBalancer
# vim redis-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: default
spec:
selector:
app: redis
role: logstor
clusterIP: 10.96.88.88
type: ClusterIP
ports:
- port: 6339
targetPort: 6379
资源记录
SVC_NAME.NS_NAME.DOMAIN.LTD.
svc.cluster.local
redis.default.svc.cluster.local
# vim myapp-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: myapp
namespace: default
spec:
selector:
app: myapp
release: canary
clusterIP: 10.99.99.99
type: NodePort
ports:
- port: 80
targetPort: 80
nodePort: 30080
# kubectl get svc myapp
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
myapp NodePort 10.99.99.99 <none> 80:30080/TCP 4m50s
ssjinyao ➤ while true; do curl 10.1.87.82:30080/hostname.html ; sleep 1 ; done
myapp-deploy-675558bfc5-5c8hn
myapp-deploy-675558bfc5-qtlqf
myapp-deploy-675558bfc5-qtlqf
myapp-deploy-675558bfc5-5c8hn
myapp-deploy-675558bfc5-qgps7
myapp-deploy-675558bfc5-5c8hn
myapp-deploy-675558bfc5-qgps7
myapp-deploy-675558bfc5-qgps7
myapp-deploy-675558bfc5-qtlqf
# 来自同上客户端的请求发往同一Pod
# kubectl patch svc myapp -p '{"spec":{"sessionAffinity":"ClientIP"}}'
service/myapp patched
# kubectl describe svc myapp
Name: myapp
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"myapp","namespace":"default"},"spec":{"clusterIP":"10.99.99.99","...
Selector: app=myapp,release=canary
Type: NodePort
IP: 10.99.99.99
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 30080/TCP
Endpoints: 10.244.1.150:80,10.244.2.166:80,10.244.3.167:80
Session Affinity: ClientIP
External Traffic Policy: Cluster
Events: <none>
ssjinyao ➤ while true; do curl 10.1.87.82:30080/hostname.html ; sleep 1 ; done
myapp-deploy-675558bfc5-qgps7
myapp-deploy-675558bfc5-qgps7
myapp-deploy-675558bfc5-qgps7
myapp-deploy-675558bfc5-qgps7
无头服务
# vim myapp-svc-headless.yaml
apiVersion: v1
kind: Service
metadata:
name: myapp-svc
namespace: default
spec:
selector:
app: myapp
release: canary
clusterIP: None
ports:
- port: 80
targetPort: 80
# kubectl apply -f myapp-svc-headless.yaml
service/myapp-svc created
# dig -t A myapp-svc.default.svc.cluster.local. @10.96.0.10
; <<>> DiG 9.9.4-RedHat-9.9.4-73.el7_6 <<>> -t A myapp-svc.default.svc.cluster.local. @10.96.0.10
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38590
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;myapp-svc.default.svc.cluster.local. IN A
;; ANSWER SECTION:
myapp-svc.default.svc.cluster.local. 5 IN A 10.244.3.167
myapp-svc.default.svc.cluster.local. 5 IN A 10.244.1.150
myapp-svc.default.svc.cluster.local. 5 IN A 10.244.2.166
;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Thu May 16 10:59:03 CST 2019
;; MSG SIZE rcvd: 217
# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 20d
# kubectl get pods -o wide -l app=myapp
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
myapp-deploy-675558bfc5-5c8hn 1/1 Running 0 42h 10.244.1.150 bj-gzf-vm-ops-test01 <none> <none>
myapp-deploy-675558bfc5-qgps7 1/1 Running 0 42h 10.244.2.166 bj-gzf-vm-ops-test02 <none> <none>
myapp-deploy-675558bfc5-qtlqf 1/1 Running 0 42h 10.244.3.167 bj-gzf-vm-ops-test03 <none> <none>
ingress 及 ingress controller
# mkdir ingress-nginx
for file in configmap.yaml mandatory.yaml namespace.yaml rbac.yaml with-rbac.yaml ; do wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/$file ; done
# kubectl apply -f ./
# cat deploy-demo.yaml
apiVersion: v1
kind: Service
metadata:
name: myapp
namespace: default
spec:
selector:
app: myapp
release: canary
ports:
- name: httpd
targetPort: 80
port: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-deploy
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: myapp
release: canary
template:
metadata:
labels:
app: myapp
release: canary
spec:
containers:
- name: myapp
image: ikubernetes/myapp:v2
ports:
- name: http
containerPort: 80
# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml
# pwd
/root/ingress-nginx
# ls
configmap.yaml mandatory.yaml namespace.yaml rbac.yaml service-nodeport.yaml with-rbac.yaml
# vim service-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
nodePort: 30080
- name: https
port: 443
targetPort: 443
protocol: TCP
nodePort: 30443
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
# kubectl apply -f ./
# kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx NodePort 10.96.59.169 <none> 80:30080/TCP,443:30443/TCP 2m35s
现在可以尝试访问
查看访问apiVersion
# kubectl explain ingress
KIND: Ingress
VERSION: extensions/v1beta1
# kubectl explain ingress.spec.rules.http.paths
# vim ingress-myapp.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-myapp
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: myapp.ssjinyao.com
http:
paths:
- path:
backend:
serviceName: myapp
servicePort: 80
# kubectl describe ingress ingress-myapp
Name: ingress-myapp
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
myapp.ssjinyao.com
myapp:80 (10.244.1.150:80,10.244.2.166:80,10.244.3.167:80)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"ingress-myapp","namespace":"default"},"spec":{"rules":[{"host":"myapp.ssjinyao.com","http":{"paths":[{"backend":{"serviceName":"myapp","servicePort":80},"path":null}]}}]}}
kubernetes.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 18s nginx-ingress-controller Ingress default/ingress-myapp
# kubectl exec -n ingress-nginx nginx-ingress-controller-5694ccb578-xkgkx -it -- /bin/sh
这时再访问
tomcat 七层负载创建
# vim tomcat-deploy.yaml
apiVersion: v1
kind: Service
metadata:
name: tomcat
namespace: default
spec:
selector:
app: tomcat
release: canary
ports:
- name: httpd
targetPort: 8080
port: 8080
- name: ajp
targetPort: 8009
port: 8009
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-deploy
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: tomcat
release: canary
template:
metadata:
labels:
app: tomcat
release: canary
spec:
containers:
- name: tomcat
image: tomcat:8.5.32-jre8-alpine
ports:
- name: http
containerPort: 8080
- name: ajp
containerPort: 8009
# kubectl apply -f tomcat-deploy.yaml
# vim ingress-tomcat.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-tomcat
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: tomcat.ssjinyao.com
http:
paths:
- path:
backend:
serviceName: tomcat
servicePort: 8080
# kubectl apply -f ingress-tomcat.yaml
secret 创建与使用
# kubectl create secret tls tomcat-ingress-secret --cert=2214184_tomcat.ssjinyao.com.pem --key=2214184_tomcat.ssjinyao.com.key
secret/tomcat-ingress-secret created
# kubectl describe secret tomcat-ingress-secret
Name: tomcat-ingress-secret
Namespace: default
Labels: <none>
Annotations: <none>
Type: kubernetes.io/tls
Data
====
tls.crt: 3659 bytes
tls.key: 1675 bytes
# vim ingress-tomcat-tls.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-tomcat-tls
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
tls:
- hosts:
- tomcat.ssjinyao.com
secretName: tomcat-ingress-secret
rules:
- host: tomcat.ssjinyao.com
http:
paths:
- path:
backend:
serviceName: tomcat
servicePort: 8080
# kubectl apply -f ingress-tomcat-tls.yaml
# kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
ingress-tomcat tomcat.ssjinyao.com 80 39m
ingress-tomcat-tls tomcat.ssjinyao.com 80, 443 93s
# kubectl describe ingress ingress-tomcat-tls
Name: ingress-tomcat-tls
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
TLS:
tomcat-ingress-secret terminates tomcat.ssjinyao.com
Rules:
Host Path Backends
---- ---- --------
tomcat.ssjinyao.com
tomcat:8080 (10.244.1.153:8080,10.244.2.171:8080,10.244.3.170:8080)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"ingress-tomcat-tls","namespace":"default"},"spec":{"rules":[{"host":"tomcat.ssjinyao.com","http":{"paths":[{"backend":{"serviceName":"tomcat","servicePort":8080},"path":null}]}}],"tls":[{"hosts":["tomcat.ssjinyao.com"],"secretName":"tomcat-ingress-secret"}]}}
kubernetes.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 2m29s nginx-ingress-controller Ingress default/ingress-tomcat-tls
 - 芊芊细语&pics=/images/favicon/k8s-logo.png&summary=){kind=logo}
 - 芊芊细语&pics=/images/favicon/k8s-logo.png&summary=){kind=logo}
 - 芊芊细语&pics=/images/favicon/k8s-logo.png&summary=){kind=logo}