zookeeper 쿼럼을 과반수 이상 작동해야하는 이유

kafka 기술문서

zookeeper 쿼럼을 과반수 이상 작동해야하는 이유

saay-hi 2024. 6. 24. 13:36

[ 기동 불가한 경우 ]

[ 기동 가능한 경우 ]

Zookeeper 쿼럼을 과반수 이상 작동 해야하는 이유

1 ) Zookeeper 1대만 기동할 경우 : zookeeper2,3 번갈아서 connection 시도

2 ) Zookeeper 2대 기동할 경우 : zookeeper3를 connection 하려고하지만, 연결이 안 되자 더 이상 connection 시도하지 않음. 또한 이 때 leader – follower 선출, zookeeper2가 following 상태로 전환

3 ) Zookeeper 3대 모두 기동할 경우 :기존 leader 변동 없이 zookeeper3가 following 상태로 전환

zookeeper 먼저 기동하고 과반수 이상 작동해야, broker 정상 기동

1 ) 단, cluster 로 묶인 consumer group일 경우 broker 0,1,2가 모두 기동해야 consumer group 조회 가능

기본 구성

Zookeeper_servers

server.1=192.168.56.105:2887:3887
server.2=192.168.56.105:2888:3888
server.3=192.168.56.105:2889:3889

Broker 0,1,2

Broker 0 : 192.168.56.105:9092
Broekr 1 : 192.168.56.105:9093
Broker 2 : 192.168.56.105:9094

참고)

1. zookeeper 과반수 이상 기동해야 정상 기동 - 쿼럼

ookeeper1만 기동했을때 server log 상태

[2024-02-27 09:30:53,912] INFO Reading configuration from: ../config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2024-02-27 09:30:53,920] WARN ../config/zookeeper.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2024-02-27 09:30:53,924] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.
생략...

java.net.ConnectException: 연결이 거부됨 (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:607)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:384)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:458)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

zookeeper2 이어서 기동 시작

[2024-02-27 09:33:50,796] INFO Reading configuration from: ../config/zookeeper2.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2024-02-27 09:33:50,798] WARN ../config/zookeeper2.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2024-02-27 09:33:50,804] INFO clientPortAddress is 0.0.0.0:2182 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2024-02-27 09:33:50,804] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2024-02-27 09:33:50,804] INFO observerMasterPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2024-02-27 09:33:50,804] INFO metricsProvider.className is org.apache.zookeeper.metrics.impl.DefaultMetricsProvider (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2024-02-27 09:33:50,819] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper)'

java.net.ConnectException: 연결이 거부됨 (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:607)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:384)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:458)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
[2024-02-27 09:33:51,177] INFO Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x1, n.peerEpoch:0x18, n.zxid:0x18000000a6, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2024-02-27 09:33:51,179] INFO Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x18, n.zxid:0x18000000a6, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2024-02-27 09:33:51,181] INFO Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x18, n.zxid:0x18000000a6, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2024-02-27 09:33:51,183] INFO Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x18, n.zxid:0x18000000a6, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2024-02-27 09:33:51,183] WARN Cannot open channel to 3 at election address /192.168.56.105:3889 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.net.ConnectException: 연결이 거부됨 (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:607)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:384)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:458)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
[2024-02-27 09:33:51,387] INFO Peer state changed: leading (org.apache.zookeeper.server.quorum.QuorumPeer)
[2024-02-27 09:33:51,388] INFO Peer state changed: following (org.apache.zookeeper.server.quorum.QuorumPeer)
[2024-02-27 09:33:51,388] INFO LEADING (org.apache.zookeeper.server.quorum.QuorumPeer)
[2024-02-27 09:33:51,389] INFO FOLLOWING (org.apache.zookeeper.server.quorum.QuorumPeer)
[2024-02-27 09:33:51,395] INFO zookeeper.leader.maxConcurrentSnapSyncs = 10 (org.apache.zookeeper.server.quorum.LearnerMaster)
[2024-02-27 09:33:51,395] INFO zookeeper.leader.maxConcurrentDiffSyncs = 100 (org.apache.zookeeper.server.quorum.LearnerMaster)
[2024-02-27 09:33:51,396] INFO TCP NoDelay set to: true (org.apache.zookeeper.server.quorum.Leader)
[2024-02-27 09:33:51,396] INFO zookeeper.leader.ackLoggingFrequency = 1000 (org.apache.zookeeper.server.quorum.Leader)
[2024-02-27 09:33:51,396] INFO zookeeper.leader.maxTimeToWaitForEpoch = -1ms (org.apache.zookeeper.server.quorum.Leader)
[2024-02-27 09:33:51,398] INFO leaderConnectDelayDuringRetryMs: 100 (org.apache.zookeeper.server.quorum.Learner)
[2024-02-27 09:33:51,398] INFO TCP NoDelay set to: true (org.apache.zookeeper.server.quorum.Learner)
[2024-02-27 09:33:51,398] INFO zookeeper.learner.asyncSending = false (org.apache.zookeeper.server.quorum.Learner)
[2024-02-27 09:33:51,398] INFO zookeeper.learner.closeSocketAsync = false (org.apache.zookeeper.server.quorum.Learner)
[2024-02-27 09:33:51,401] INFO Weighed connection throttling is disabled (org.apache.zookeeper.server.BlueThrottle)
[2024-02-27 09:33:51,402] INFO Weighed connection throttling is disabled (org.apache.zookeeper.server.BlueThrottle)
[2024-02-27 09:33:51,402] INFO minSessionTimeout set to 6000 ms (org.apache.zookeeper.server.ZooKeeperServer)
[2024-02-27 09:33:51,402] INFO maxSessionTimeout set to 60000 ms (org.apache.zookeeper.server.ZooKeeperServer)
[2024-02-27 09:33:51,403] INFO minSessionTimeout set to 6000 ms (org.apache.zookeeper.server.ZooKeeperServer)
[2024-02-27 09:33:51,404] INFO maxSessionTimeout set to 60000 ms (org.apache.zookeeper.server.ZooKeeperServer)

//zookeeper3 기동

[2024-02-27 09:38:20,356] INFO Reading configuration from: ../config/zookeeper3.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2024-02-27 09:38:20,358] WARN ../config/zookeeper3.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2024-02-27 09:38:20,365] INFO clientPortAddress is 0.0.0.0:2183 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2024-02-27 09:38:20,365] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2024-02-27 09:38:20,365] INFO observerMasterPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2024-02-27 09:38:20,365] INFO metricsProvider.className is org.apache.zookeeper.metrics.impl.DefaultMetricsProvider (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2024-02-27 09:38:20,382] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2024-02-27 09:38:20,382] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2024-02-27 09:38:20,382] INFO Purge task is not scheduled.
생략...
[2024-02-27 09:38:20,829] INFO 3 is accepting connections now, my election bind port: /192.168.56.105:3889 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2024-02-27 09:38:20,831] INFO LOOKING (org.apache.zookeeper.server.quorum.QuorumPeer)
[2024-02-27 09:38:20,833] INFO New election. My id = 3, proposed zxid=0x18000000a6 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2024-02-27 09:38:20,845] INFO Notification: my state:LOOKING; n.sid:3, n.state:LOOKING, n.leader:3, n.round:0x1, n.peerEpoch:0x18, n.zxid:0x18000000a6, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2024-02-27 09:38:20,846] INFO Received connection request from /192.168.56.105:37654 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2024-02-27 09:38:20,846] INFO Received connection request from /192.168.56.105:35090 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2024-02-27 09:38:20,852] INFO Notification: my state:LEADING; n.sid:3, n.state:LOOKING, n.leader:3, n.round:0x1, n.peerEpoch:0x18, n.zxid:0x18000000a6, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2024-02-27 09:38:20,853] INFO Notification: my state:FOLLOWING; n.sid:3, n.state:LOOKING, n.leader:3, n.round:0x1, n.peerEpoch:0x18, n.zxid:0x18000000a6, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2024-02-27 09:38:20,853] INFO Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x18, n.zxid:0x18000000a6, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2024-02-27 09:38:20,854] INFO Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x18, n.zxid:0x18000000a6, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2024-02-27 09:38:20,855] INFO Notification: my state:LOOKING; n.sid:1, n.state:FOLLOWING, n.leader:2, n.round:0x1, n.peerEpoch:0x19, n.zxid:0x18000000a6, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2024-02-27 09:38:20,857] INFO Notification: my state:LOOKING; n.sid:2, n.state:LEADING, n.leader:2, n.round:0x1, n.peerEpoch:0x19, n.zxid:0x18000000a6, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection)

고민해볼 것 : 연결이 성공했다는 워딩을 뭘로 잡고 grep으로 조회할 지