抱歉，您的浏览器无法访问本站
本页面需要浏览器支持（启用）JavaScript
了解详情 >

工具

chat

badge

友链

自习之 Linux HA Cluster 2

中间件应用高可用

发布于：2018年1月3日

字数：2.3k 字

时长：12 分钟

[toc]

温故而知新

Messaaging Layer:
    heartbaet v1,v2,v3 
    corosync v1, v2(votequorum)
    OpenAIS
CRM:
    pacemaker 需要一个配置接口
        配置接口:crmsh (由SUSE研发的),由pssh管理工具来实现
                 pcs (agent c/s ),pcsd来实现
                 conga(ricci运行在各结点上的进程/luci发送指令到各节点)组成
 
                 group,constraint(基于约束)
 
                 rgmanager(cman)
                     resouce group(资源组):
                         failover domain
RA:
    LSB: /etc/rc.d/init.d
    systemd: /etc/systemd/system/multi-user.wants
        服务开机处于enable状态;
    OCF:[provider]
        heartbeat
        pacemaker 
        linbit 基于内核跨节点的块设备
    service
    stonith
    高可用集群的可用方案:
        heartbeat v1
        heartbeat v2
        heartbeat v3 + pacemaker X 
        corosync + pacemaker 
        cman + rgmanager 
        corosync + cman + pacemaker
        keepalived

Linux-HA-Cluster 2

Heartbeat集群之间传递心跳的方法
1、使用串型线缆
2、使用以太网通信
3、Unicast 单播，使用udpu，一般在不支持多播的网络中使用的
4、Mutlicast 多播/组播, 使用udp
5、Broadcast 广播，占用过高的网络资源

1、组播地址:

用于标识一个IP组播域:IANA把D类地址留给组播使用:224.0.0.0-239.255.255.255 
永久组播地址:224.0.0.0-224.0.0.255
临时组播地址:224.0.1.0-238.255.255.255  建议使用临时组播地址
本地组播地址:239.0.0.0-239.255.255.255

2、配置以组播方式进行高可用集群通信 

# 在Node1 和 Node2中执行
# yum -y install corosync pacemaker
# rpm -ql corosync #查看corosync安装完之后所生成的配置文件;
/etc/corosync
/etc/corosync/corosync.conf.example
/etc/corosync/corosync.conf.example.udpu
/etc/corosync/corosync.xml.example
/etc/corosync/uidgid.d
/etc/dbus-1/system.d/corosync-signals.conf
/etc/logrotate.d/corosync
/etc/sysconfig/corosync
/etc/sysconfig/corosync-notifyd
/usr/bin/corosync-blackbox
/usr/bin/corosync-xmlproc
/usr/lib/systemd/system/corosync-notifyd.service
/usr/lib/systemd/system/corosync.service
/usr/sbin/corosync
/usr/sbin/corosync-cfgtool
/usr/sbin/corosync-cmapctl
/usr/sbin/corosync-cpgtool
/usr/sbin/corosync-keygen
/usr/sbin/corosync-notifyd
/usr/sbin/corosync-quorumtool
/usr/share/corosync
/usr/share/corosync/corosync
/usr/share/corosync/corosync-notifyd
/usr/share/corosync/xml2conf.xsl
/usr/share/doc/corosync-2.4.0
/usr/share/doc/corosync-2.4.0/LICENSE
/usr/share/doc/corosync-2.4.0/SECURITY
/usr/share/man/man5/corosync.conf.5.gz
/usr/share/man/man5/corosync.xml.5.gz
/usr/share/man/man5/votequorum.5.gz
/usr/share/man/man8/cmap_keys.8.gz
/usr/share/man/man8/corosync-blackbox.8.gz
/usr/share/man/man8/corosync-cfgtool.8.gz
/usr/share/man/man8/corosync-cmapctl.8.gz
/usr/share/man/man8/corosync-cpgtool.8.gz
/usr/share/man/man8/corosync-keygen.8.gz
/usr/share/man/man8/corosync-notifyd.8.gz
/usr/share/man/man8/corosync-quorumtool.8.gz
/usr/share/man/man8/corosync-xmlproc.8.gz
/usr/share/man/man8/corosync.8.gz
/usr/share/man/man8/corosync_overview.8.gz
/usr/share/snmp/mibs/COROSYNC-MIB.txt
/var/lib/corosync
/var/log/cluster
# rpm -ql pacemaker #查看pacemaker所生成的配置文件;
/etc/sysconfig/pacemaker
/usr/lib/ocf/resource.d/.isolation
/usr/lib/ocf/resource.d/.isolation/docker-wrapper
/usr/lib/ocf/resource.d/pacemaker/controld
/usr/lib/ocf/resource.d/pacemaker/remote
/usr/lib/systemd/system/pacemaker.service
/usr/libexec/pacemaker/attrd
/usr/libexec/pacemaker/cib
/usr/libexec/pacemaker/cibmon
/usr/libexec/pacemaker/crmd
/usr/libexec/pacemaker/lrmd
/usr/libexec/pacemaker/lrmd_internal_ctl
/usr/libexec/pacemaker/pengine
/usr/libexec/pacemaker/stonith-test
/usr/libexec/pacemaker/stonithd
/usr/sbin/crm_attribute
/usr/sbin/crm_master
/usr/sbin/crm_node
/usr/sbin/pacemakerd
/usr/sbin/stonith_admin
/usr/share/doc/pacemaker-1.1.16
/usr/share/doc/pacemaker-1.1.16/COPYING
/usr/share/doc/pacemaker-1.1.16/ChangeLog
/usr/share/licenses/pacemaker-1.1.16
/usr/share/licenses/pacemaker-1.1.16/GPLv2
/usr/share/man/man7/crmd.7.gz
/usr/share/man/man7/ocf_pacemaker_controld.7.gz
/usr/share/man/man7/ocf_pacemaker_remote.7.gz
/usr/share/man/man7/pengine.7.gz
/usr/share/man/man7/stonithd.7.gz
/usr/share/man/man8/crm_attribute.8.gz
/usr/share/man/man8/crm_master.8.gz
/usr/share/man/man8/crm_node.8.gz
/usr/share/man/man8/pacemakerd.8.gz
/usr/share/man/man8/stonith_admin.8.gz
/usr/share/pacemaker/alerts
/usr/share/pacemaker/alerts/alert_file.sh.sample
/usr/share/pacemaker/alerts/alert_smtp.sh.sample
/usr/share/pacemaker/alerts/alert_snmp.sh.sample
/var/lib/pacemaker/cib
/var/lib/pacemaker/pengine

2、配置corosync 

# cd /etc/corosync/
# cp corosync.conf.example 
# cp corosync.conf.example corosync.conf
# 某些环境中可能不支持组播。这时应该配置Corosync使用单播;  
# 下面是使用单播的Corosync 配置文件的一部分;
 
totem {
         #...
         interface {
                 ringnumber: 0
                 bindnetaddr: 10.180.22.0
                 broadcast: yes (1) 
                 mcastport: 5405
         }
         interface {
                 ringnumber: 1 
                 bindnetaddr: 10.180.22.0
                 brodcast: yes
                 mcastport: 5405
         }
         transport: udpu (2)
}
 
nodelist{ (3)
        node {
                    ring0_addr: 10.180.22.166
                    ring1_addr:10.180.55.1
                    nodedid: 1
        }
        node { 
                    ring0_addr: 10.180.22.167
                    ring1_addr: 10.180.55.2
                    nodeid: 2
         }
# 如果将 broadcast 设置为yes,集群心跳将通过广播实现。设置该参数时,不能设置mcastaddr;
# transport 配置项决定集群通信方式，要完全禁用组播，应该配置单播传输参数 udpu;
# 这要求将所有的节点服务器信息写入nodelist;
# 也就是需要在配置HA 集群之前确定节点组成。默认配置是udp。 通信方式类型还支持udpu和iba;
# 在nodelist 之下可以为栽一节点设置只与该节点相关的信息，这些设置项只能包含在node之中;
# 即只能对属于集群的节点服务器进行设置，而且只应包括那些与默认设置不同的参数;
# 每台服务器都必需配置 ring0_addr; 
 
# 实验中配置信息如下
# Please read the corosync.conf.5 manual page
totem {
    version: 2
 
    # crypto_cipher and crypto_hash: Used for mutual node authentication.
    # If you choose to enable this, then do remember to create a shared
    # secret with "corosync-keygen".
    # enabling crypto_cipher, requires also enabling of crypto_hash.
    crypto_cipher: aes128
    crypto_hash: sha1
    secauth: on
    # interface: define at least one interface to communicate
    # over. If you define more than one interface stanza, you must
    # also set rrp_mode.
    interface {
                # Rings must be consecutively numbered, starting at 0.
        ringnumber: 0
        # This is normally the *network* address of the
        # interface to bind to. This ensures that you can use
        # identical instances of this configuration file
        # across all your cluster nodes, without having to
        # modify this option.
        bindnetaddr: 10.180.0.0
        # However, if you have multiple physical network
        # interfaces configured for the same subnet, then the
        # network address alone is not sufficient to identify
        # the interface Corosync should bind to. In that case,
        # configure the *host* address of the interface
        # instead:
        # bindnetaddr: 192.168.1.1
        # When selecting a multicast address, consider RFC
        # 2365 (which, among other things, specifies that
        # 239.255.x.x addresses are left to the discretion of
        # the network administrator). Do not reuse multicast
        # addresses across multiple Corosync clusters sharing
        # the same network.
        mcastaddr: 239.185.1.31
        # Corosync uses the port you specify here for UDP
        # messaging, and also the immediately preceding
        # port. Thus if you set this to 5405, Corosync sends
        # messages over UDP ports 5405 and 5404.
        mcastport: 5405
        # Time-to-live for cluster communication packets. The
        # number of hops (routers) that this ring will allow
        # itself to pass. Note that multicast routing must be
        # specifically enabled on most network routers.
        ttl: 1
    }
}
 
nodelist{ 
        node {
            ring0_addr: 10.180.22.166
            nodedid: 1
        }
        node { 
            ring0_addr: 10.180.22.167
            nodeid: 2
       }
       node {
               ring0_addr: 10.180.22.168
               nodeid: 3
       }
     }
 
logging {
    # Log the source file and line where messages are being
    # generated. When in doubt, leave off. Potentially useful for
    # debugging.
    fileline: off
    # Log to standard error. When in doubt, set to no. Useful when
    # running in the foreground (when invoking "corosync -f")
    to_stderr: no
    # Log to a log file. When set to "no", the "logfile" option
    # must not be set.
    to_logfile: yes
    logfile: /var/log/cluster/corosync.log
    # Log to the system log daemon. When in doubt, set to yes.
    to_syslog: no
    # Log debug messages (very verbose). When in doubt, leave off.
    debug: off
    # Log messages with time stamps. When in doubt, set to on
    # (unless you are only logging to syslog, where double
    # timestamps can be annoying).
    timestamp: on
    logger_subsys {
        subsys: QUORUM
        debug: off
    }
}
 
quorum {
    # Enable and configure quorum subsystem (default: off)
    # see also corosync.conf.5 and votequorum.5
    provider: corosync_votequorum
}
 
# 生成密钥文件  
# corosync-keygen
# scp -p authkey coronsync.conf root@node2.ssjinyao.com:/etc/corosync
 
# corosync 启动集群投票系统默认必需需要三人结点;
# systemctl start corosync.serivce #这个时候启动是失败的;
# systemctl status corosync.serivce #查看corosync的状态信息;
 
# 此时需要开启另一个结点
# yum -y install corosync pacemaker 
# Node1 中执行
# scp -p  /etc/coronsync.conf /etc/authkey root@node2.ssjinyao.com:/etc/corosync

3、验证结点 

# corosync-cfgtool -s  # 验证结点
# corosync-cmapctl  #查看结点间的信息;
# grep -v '^[[:space:]]*#' /etc/corosync.conf  # 保存实用的配置示例

4、编辑pacemaker

# vim /etc/sysconfig/pacemaker
PCMK_logfile= /var/log/pacemaker.log
# 各个节点间需要启动pacemaker
# systemctl start pacemaker.service
# crm_mon  # 进行验证

5、 crm 工具的使用简明 

# crm_node -n  # 查看当前节点信息
# crm_node -l  # 例出集群所有的节点信息
# crm_verify -L -V # 但看报错信息
# crm shell 的安装与使用，若无rpm包，可在网上找相关的rpm包
# yum -y install crmsh pssh python-pssh

HA Web Service

    vip: 10.180.xx.xxx, ocf:hearbeat:IPaddr 
    httpd: systemd 
    nfs shared storage: ocf:heartbeat:Filesystem
Ha Cluster 工作模型:
    A/P: 两节点集群; active/apsslve;
        without-quorum-policy=(stop|ignore|suicide|freeze)
    A/A:
# node0 
# mkdir -pv  /www/htdocs 
# echo "<h1> Test Page on NFS Server</h1>" > /www/htdocs/index.html
# vim /etc/exports 
/www/htdocs 10.180.xx.xxx/16(rw)
# iptables -L -n  # 查看是否有防火墙规则
# checkconfig nfs on 
# systemctl start nfs
# node 1 
# showmount -e 10.180.xx.xxx
# mount -t nfs 10.180.xx.xxx:/www/htdocs /var/www/html # 挂载网络存储NFS 
# systemctl start httpd 
# systemctl enable httpd
# 此时便可以查看能否可以能否访问到以上的测试信息
# node 2 
# systemctl start httpd 
# systemctl enabl httpd
# crm configure 
crm(live)configure# show
crm(live)# configure property 
crm(live)# configure property stonith-enabled=false
crm(live)# configrue verify 
crm(live)# show 
crm(live)# configure commit
crm(live)# configure 
crm(live)configure# primitive webip ocf:heartbeat:IPaddr2 params ip="10.180.xx.xxx" op monitor interval=30s timeout=20s
crm(live)# configure show 
crm(live)# configure commit
crm(live)# configure edit  # 可以打开vim编辑配置文件 
crm(live)# configure verify
crm(live)# configure primitive webstore ocf:heartbat:Filesystem params device="10.180.xx.xxx:/www/htdocs" directory="/var/www/html" fstype="nfs" op start timeout=60s op stop timeout=60s op monitor interval=20s timeout=40s 
crm(live)# configure verify
crm(live)configure# colocation webserver_with_webstore_and_webip inf: webserver 
( webip webstore )
crm(live)# show xmls # 查看xml格式的配置
crm(live)configure# order webstore_afer_webip Mandatory: webip webstore
crm(live)configure# order webserver_after_webstore Manadatory: webstore webserver 
crm(live)configure# verify 
crm(live)configure# commit 
crm(live)configure# location webservice_perf_node1 webip 100: node1.ssjinao.com
crm(live)configure# verify 
crm(live)configure# commit
crom(live)configure# property default-resource-sticklines = 50
# systemctl stop httpd.service 
# systemctl enable httpd.service 
# crm_verify -V -L
# crm node standby  
# crm onde online

更新于：2023年1月16日

sre

heartbeat

字数：2.3k 字

时长：12 分钟

评论