查询速度快 30 倍的 ClickHouse，凭什么替代 ELK？

yum install java-1.8.0-openjdk-devel.x86_64/etc/profile 配置环境变量

更新系统时间yum install ntpdatentpdate asia.pool.ntp.org

mkdir zookeepermkdir ./zookeeper/datamkdir ./zookeeper/logswget --no-check-certificate https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.7.1/apache-zookeeper-3.7.1-bin.tar.gztar -zvxf apache-zookeeper-3.7.1-bin.tar.gz -C /usr/zookeeper

export ZOOKEEPER_HOME=/usr/zookeeper/apache-zookeeper-3.7.1-binexport PATH=$ZOOKEEPER_HOME/bin:$PATH

进入ZooKeeper配置目录

cd $ZOOKEEPER_HOME/conf

新建配置文件vi zoo.cfg

tickTime=2000initLimit=10syncLimit=5dataDir=/usr/zookeeper/datadataLogDir=/usr/zookeeper/logsclientPort=2181server.1=zk1:2888:3888server.2=zk2:2888:3888server.3=zk3:2888:3888

在每台服务器上执行，给zookeeper创建myidecho "1" > /usr/zookeeper/data/myidecho "2" > /usr/zookeeper/data/myidecho "3" > /usr/zookeeper/data/myid

进入ZooKeeper bin目录cd $ZOOKEEPER_HOME/binsh zkServer.sh start

mkdir -p /usr/kafkachmod 777 -R /usr/kafkawget --no-check-certificate https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/3.2.0/kafka_2.12-3.2.0.tgz

tar -zvxf kafka_2.12-3.2.0.tgz -C /usr/kafka

不同的broker Id 设置不一样，比如 1,2,3broker.id=1listeners=PLAINTEXT://ip:9092socket.send.buffer.bytes=102400socket.receive.buffer.bytes=102400socket.request.max.bytes=104857600log.dir=/usr/kafka/logsnum.partitinotallow=5num.recovery.threads.per.data.dir=3offsets.topic.replication.factor=2transaction.state.log.replication.factor=3transaction.state.log.min.isr=3log.retention.hours=168log.segment.bytes=1073741824log.retention.check.interval.ms=300000zookeeper.cnotallow=zk1:2181,zk2:2181,zk3:2181zookeeper.connection.timeout.ms=30000group.initial.rebalance.delay.ms=0

后台常驻进程启动kafkanohup /usr/kafka/kafka_2.12-3.2.0/bin/kafka-server-start.sh /usr/kafka/kafka_2.12-3.2.0/config/server.properties >/usr/kafka/logs/kafka.log >&1 &

/usr/kafka/kafka_2.12-3.2.0/bin/kafka-server-stop.sh

$KAFKA_HOME/bin/kafka-topics.sh --list --bootstrap-server  ip:9092

$KAFKA_HOME/bin/kafka-console-consumer.sh --bootstrap-server ip:9092 --topic test --from-beginning

$KAFKA_HOME/bin/kafka-topics.sh  --create --bootstrap-server  ip:9092  --replication-factor 2 --partitions 3 --topic xxx_data

sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

Create a file with a .repo extension (for example, elastic.repo) in your /etc/yum.repos.d/ directory and add the following lines:在/etc/yum.repos.d/ 目录下创建elastic.repo

[elastic-8.x]name=Elastic repository for 8.x packagesbaseurl=https://artifacts.elastic.co/packages/8.x/yumgpgcheck=1gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearchenabled=1autorefresh=1type=rpm-md

yum install filebeatsystemctl enable filebeatchkconfig --add filebeat

文件目录：/etc/filebeat/filebeat.yml

filebeat.inputs:- type: log  enabled: true  paths:    - /root/logs/xxx/inner/*.log  json:  如果不设置该索性，所有的数据都存储在message里面，这样设置以后数据会平铺。       keys_under_root: true output.kafka:  hosts: ["kafka1:9092", "kafka2:9092", "kafka3:9092"]  topic: 'xxx_data_clickhouse'  partition.round_robin:            reachable_only: false            required_acks: 1            compression: gzipprocessors: 剔除filebeat 无效的字段数据    - drop_fields:          fields: ["input", "agent", "ecs", "log", "metadata", "timestamp"]        ignore_missing: false

nohup ./filebeat -e -c /etc/filebeat/filebeat.yml > /user/filebeat/filebeat.log & 输出到filebeat.log文件中，方便排查

检查当前CPU是否支持SSE 4.2，如果不支持，需要通过源代码编译构建grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported"返回 "SSE 4.2 supported" 表示支持，返回 "SSE 4.2 not supported" 表示不支持

创建数据保存目录，将它创建到大容量磁盘挂载的路径mkdir -p /data/clickhouse修改/etc/hosts文件，添加clickhouse节点举例：10.190.85.92 bigdata-clickhouse-0110.190.85.93 bigdata-clickhouse-02

服务器性能参数设置：cpu频率调节，将CPU频率固定工作在其支持的最高运行频率上，而不动态调节，性能最好echo 'performance' | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

内存调节，不要禁用 overcommitecho 0 | tee /proc/sys/vm/overcommit_memory

始终禁用透明大页(transparent huge pages)。它会干扰内存分配器，从而导致显着的性能下降echo 'never' | tee /sys/kernel/mm/transparent_hugepage/enabled

首先，需要添加官方存储库：yum install yum-utilsrpm --import <https://repo.clickhouse.tech/CLICKHOUSE-KEY.GPG>yum-config-manager --add-repo <https://repo.clickhouse.tech/rpm/stable/x86_64>

查看clickhouse可安装的版本：yum list | grep clickhouse运行安装命令：yum -y install clickhouse-server clickhouse-client

修改/etc/clickhouse-server/config.xml配置文件，修改日志级别为information，默认是trace<level>information</level>执行日志所在目录：

正常日志/var/log/clickhouse-server/clickhouse-server.log异常错误日志/var/log/clickhouse-server/clickhouse-server.err.log

查看安装的clickhouse版本：clickhouse-server --versionclickhouse-client --password

sudo clickhouse stopsudo clickhouse tartsudo clickhouse start

max_memory_usage：单个服务器上运行查询的最大内存；max_memory_usage_for_user：单个服务器上运行用户查询的最大内存；max_memory_usage_for_all_queries：单个服务器上运行所有查询的最大内存；max_rows_to_read：运行查询时可从表中读取的最大行数；max_result_rows：限制结果中的行数；max_bytes_to_read：运行查询时可以从表中读取的最大字节数（未压缩数据）。