zookeeper、kafka搭建部署
kafka搭建
单机部署参考文档： https://www.cnblogs.com/wonglu/p/8687488.html
群集部署参考文档：https://www.cnblogs.com/luotianshuai/p/5206662.html
第一步：安装Java环境
下面安装所使用的安装包为自行下载的离线包。
# tar xf jdk-8u211-linux-x64.tar.gz
# mv jdk1.8.0_211 /usr/local/jdk
# chown -R root:root /usr/local/jdk
# echo "export JAVA_HOME=/usr/local/jdk" >>/etc/profile
# echo 'export PATH=$JAVA_HOME/bin:$PATH' >>/etc/profile
②.验证Java环境
# java -version
java version "1.8.0_211" Java(TM) SE Runtime Environment (build 1.8.0_211-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
第二步：安装zookeeper
①.创建目录
# useradd zookeeper
# echo zookeeper |passwd --stdin zookeeper
# mkdir -p /data/zookeeper/{zkdata,zkdatalog}
# wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.5.5/apache-zookeeper-3.5.5-bin.tar.gz
# tar xf apache-zookeeper-3.5.5.tar.gz
# mv apache-zookeeper-3.5.5 /usr/local/zookeeper
# chown -R zookeeper:zookeeper /usr/local/zookeeper
# chown -R zookeeper:zookeeper /data/zookeeper
# cd /usr/local/zookeeper
# cp zoo_sample.cfg zoo.cfg
②.配置
# cat zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/data/zookeeper/zkdata
dataLogDir=/data/zookeeper/zkdatalog
# the port at which the clients will connect
clientPort=12181
# the maximum number of client connections.
# increase this if you need to handle more clients
# maxClientCnxns=60
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html
# sc_maintenance # # The number of snapshots to retain in dataDir
# autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
# autopurge.purgeInterval=1
server.1=192.168.8.40:12888:13888
server.2=192.168.8.106:12888:13888
server.3=192.168.8.177:12888:13888
将这个配置文件scp到集群的其他机器上。
③.创建myid文件
在server1机器上
# echo "1" > /data/zookeeper/zkdata/myid
在server2机器上
# echo "2" > /data/zookeeper/zkdata/myid
在server3机器上
# echo "3" > /data/zookeeper/zkdata/myid
④.启动zookeeper
进入到Zookeeper的bin目录下
# cd /usr/local/zookeeper/bin
启动服务（3台都需要操作）
# ./zkServer.sh start
检查服务器状态
# ./zkServer.sh status ./zkServer.sh status JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
配置文件 Mode: follower #他是否为领导
第三步：安装kafka
①.创建目录
# mkdir -p /data/kafka/kafkalogs
②.下载安装包
# wget https://www.apache.org/dyn/closer.cgi?path=/kafka/2.3.0/kafka_2.11-2.3.0.tgz
# tar xf kafka_2.11-2.3.0.tgz # mv kafka_2.11-2.3.0 /usr/local/kafka
③.配置
# grep -Ev "^#|^$" server.properties
broker.id=0
listeners=PLAINTEXT://192.168.8.177:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/kafkalogs
num.partitions=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
message.max.byte=5242880
default.replication.factor=2
replica.fetch.max.bytes=5242880
delete.topic.enable=true
log.cleaner.enable=true
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.8.40:12181,192.168.8.106:12181,192.168.8.177:12181 zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=3000
将配置文件scp到另外两台机器，将broker.id和listeners修改一下即可。
④.编写启动文件
# cd /usr/local/kafka/bin
# vim start.sh
#!/bin/bash
/usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
⑤.启动
# bash /usr/local/kafka/bin/start.sh
⑥.检查
# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 4202/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 3138/master
tcp6 0 0 :::39180 :::* LISTEN 22784/java
tcp6 0 0 :::111 :::* LISTEN 1/systemd
tcp6 0 0 :::8080 :::* LISTEN 22270/java
tcp6 0 0 :::12181 :::* LISTEN 22270/java
tcp6 0 0 :::22 :::* LISTEN 4202/sshd
tcp6 0 0 192.168.8.177:12888 :::* LISTEN 22270/java
tcp6 0 0 ::1:25 :::* LISTEN 3138/master
tcp6 0 0 :::43195 :::* LISTEN 22270/java
tcp6 0 0 192.168.8.177:13888 :::* LISTEN 22270/java
tcp6 0 0 192.168.8.177:9092 :::* LISTEN 22784/java