凌晨1点，ES集群红到发黑

在线日志平台凌晨告警：cluster status: red，5个节点里有2个掉线，查询接口超时率100%。登录服务器看到的是这样的日志：

[2024-01-15T01:23:11,812][WARN ][o.e.c.c.ClusterFormationFailureHelper] master not discovered yet, have discovered [{node-3}{...}{xyz}{10.0.0.13}], retrying...
[2024-01-15T01:23:14,015][ERROR][o.e.x.m.e.l.LocalExecutionCircuitBreaker] [node-1] circuit_breaking_exception: [parent] Data too large, data for [] would be [1.2gb], which is larger than the limit of [1.0gb]
[2024-01-15T01:23:15,902][ERROR][o.e.m.j.JvmGcMonitorService] [node-2] [gc][young][1734612][3642] overhead, spent [1.8s] collecting in the last [1.6s]

这套集群的配置是：3台裸金属，每台16核64GB内存，全部角色 master+data+ingest，JVM堆各给了31GB。业务流量一上来，数据节点Full GC，主节点跟着遭殃，最后整个集群陷入选举循环。

我用两个方案对比测试，最终用分层架构把集群救回来了。下面是完整过程。

方案对比：混用 vs 分层

方案A：三节点全角色混用（原配置）

• 3节点，每台 node.roles: [ master, data, ingest ]
• 堆内存31GB（16核64G物理机）
• 数据盘：本地SATA SSD
• 集群状态：稳定运行2个月，高峰期必崩

方案B：专用主节点 + 数据节点 + 协调节点（推荐）

• 3台 master-only：4核8GB，堆4GB
• 3台 data：8核32GB，堆16GB
• 2台 coordinating：8核16GB，堆8GB
• 数据盘：本地NVMe SSD
• 集群状态：压测30分钟无异常，切换节点无感知

对比项	方案A（3节点混用）	方案B（分层架构）
master节点JVM Old GC耗时	平均320ms/次	平均40ms/次
写入吞吐量	约1.6万 docs/s	约3.2万 docs/s
查询P99	820ms	135ms
数据节点故障恢复	33s（触发重新选主）	11s（主节点不受影响）
横向扩展影响	加节点会拖慢master	只加data/coordinating即可

为什么会差这么多？混用架构下，master和数据职责争抢CPU、内存、GC线程。写入高峰期数据节点在merge segment、刷translog，此时master节点要响应其他节点的心跳、维护集群状态。数据节点CPU一打满，master线程就被饿死，其他节点收不到master的响应，触发新一轮选举。越写越崩，越崩越写。

部署实现：ES 8.11.3 分层集群

环境说明：CentOS 7.9，内核3.10.0-1160，ES 8.11.3（自带JDK 17），Python 3.9.18。

第一步：系统参数调整（所有节点执行）

#!/bin/bash
# system-tuning.sh 所有8台节点执行

# 1. 关闭swap
swapoff -a
sed -i '/swap/s/^/#/' /etc/fstab

# 2. 内核参数
tee -a /etc/sysctl.d/99-elasticsearch.conf <<'EOF'
vm.max_map_count=262144
net.core.somaxconn=65535
net.ipv4.tcp_max_syn_backlog=65535
net.ipv4.tcp_keepalive_time=600
net.ipv4.tcp_keepalive_intvl=60
net.ipv4.tcp_keepalive_probes=9
vm.swappiness=0
EOF
sysctl -p /etc/sysctl.d/99-elasticsearch.conf

# 3. 文件描述符与内存锁定
tee -a /etc/security/limits.conf <<'EOF'
elasticsearch soft nofile 1048576
elasticsearch hard nofile 1048576
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
EOF

# 4. 覆盖systemd的LimitMEMLOCK（CentOS7必须做，否则memory_lock永远失败）
mkdir -p /etc/systemd/system/elasticsearch.service.d
cat > /etc/systemd/system/elasticsearch.service.d/override.conf <<'EOF'
[Service]
LimitMEMLOCK=infinity
LimitNOFILE=1048576
EOF
systemctl daemon-reload

不生效的话用 systemctl show elasticsearch -p LimitMEMLOCK 验证。我见过十个人里有八个漏掉第4步，然后被 bootstrap.memory_lock: true 卡住。

第二步：生成节点证书（ES 8.x默认开启TLS）

# 在master-1上执行
cd /usr/share/elasticsearch

# 创建CA
bin/elasticsearch-certutil ca \
  --out /etc/elasticsearch/certs/elastic-stack-ca.p12 \
  --pass YourCAPassword

# 为每个节点生成PEM证书（IP SAN必须写全！）
# 生成master-1
bin/elasticsearch-certutil cert \
  --ca /etc/elasticsearch/certs/elastic-stack-ca.p12 \
  --ca-pass YourCAPassword \
  --pem \
  --name master-1 \
  --dns master-1,localhost \
  --ip 10.0.0.11,127.0.0.1 \
  --out /etc/elasticsearch/certs/master-1.zip

# 生成master-2
bin/elasticsearch-certutil cert \
  --ca /etc/elasticsearch/certs/elastic-stack-ca.p12 \
  --ca-pass YourCAPassword \
  --pem \
  --name master-2 \
  --dns master-2,localhost \
  --ip 10.0.0.12,127.0.0.1 \
  --out /etc/elasticsearch/certs/master-2.zip

# 生成data-1，IP换成10.0.0.21，名字换成data-1
# 每个节点都要单独生成，并把zip解压到对应节点的 /etc/elasticsearch/certs/
unzip /etc/elasticsearch/certs/master-1.zip -d /etc/elasticsearch/certs/
# 解压后得到 ca.crt, master-1.crt, master-1.key
# 把ca.crt复制到所有节点

第三步：编写elasticsearch.yml

master-1节点的配置，/etc/elasticsearch/elasticsearch.yml：

cluster.name: es-prod
node.name: master-1
node.roles:
  - master
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 10.0.0.11
http.port: 9200
transport.port: 9300
discovery.seed_hosts:
  - 10.0.0.11:9300
  - 10.0.0.12:9300
  - 10.0.0.13:9300
  - 10.0.0.21:9300
  - 10.0.0.22:9300
  - 10.0.0.23:9300
  - 10.0.0.31:9300
  - 10.0.0.32:9300
cluster.initial_master_nodes:
  - master-1
  - master-2
  - master-3
bootstrap.memory_lock: true

# 8.x安全配置，TLS必须写全
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/master-1.crt
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/master-1.key
xpack.security.transport.ssl.certificate_authorities: [ /etc/elasticsearch/certs/ca.crt ]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/master-1.crt
xpack.security.http.ssl.key: /etc/elasticsearch/certs/master-1.key
xpack.security.http.ssl.certificate_authorities: [ /etc/elasticsearch/certs/ca.crt ]

master-2、master-3只改 node.name 和证书文件名，IP对应改。注意：cluster.initial_master_nodes 只在首次启动时用，集群形成后可以删掉，留着也不影响。

data节点配置（以data-1为例），/etc/elasticsearch/elasticsearch.yml：

cluster.name: es-prod
node.name: data-1
node.roles:
  - data
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 10.0.0.21
http.port: 9200
transport.port: 9300
discovery.seed_hosts:
  - 10.0.0.11:9300
  - 10.0.0.12:9300
  - 10.0.0.13:9300
  - 10.0.0.21:9300
  - 10.0.0.22:9300
  - 10.0.0.23:9300
  - 10.0.0.31:9300
  - 10.0.0.32:9300
bootstrap.memory_lock: true

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/data-1.crt
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/data-1.key
xpack.security.transport.ssl.certificate_authorities: [ /etc/elasticsearch/certs/ca.crt ]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/data-1.crt
xpack.security.http.ssl.key: /etc/elasticsearch/certs/data-1.key
xpack.security.http.ssl.certificate_authorities: [ /etc/elasticsearch/certs/ca.crt ]

协调节点把 node.roles 设为空数组：

node.roles: []

第四步：JVM堆内存设置

用独立配置文件，别直接改 jvm.options，升级时会被覆盖。在 /etc/elasticsearch/jvm.options.d/heap.options 里写：

-Xms16g
-Xmx16g

分配规则：master节点堆给4GB，data节点给物理内存一半，协调节点给8GB。所有节点堆不要超过31GB，超了对象指针压缩失效。

第五步：启动集群并设置密码

# 先启动三个master节点，再启动data和coordinating
systemctl enable --now elasticsearch

# 等2分钟后，在任意节点执行密码配置
cd /usr/share/elasticsearch
bin/elasticsearch-setup-passwords interactive

# 检查集群状态
curl -sk https://10.0.0.31:9200/_cluster/health?pretty \
  -u elastic:YourPassword
# 期望返回 "status": "green"，节点数 8

核心调优项

索引模板：写入性能的关键

先建模板 template.json：

{
  "index_patterns": ["perftest*"],
  "template": {
    "settings": {
      "number_of_shards": 6,
      "number_of_replicas": 1,
      "refresh_interval": "30s",
      "translog.durability": "async",
      "translog.sync_interval": "5s",
      "index.routing.allocation.total_shards_per_node": 2
    },
    "mappings": {
      "properties": {
        "timestamp": { "type": "date" },
        "user_id": { "type": "integer" },
        "action": { "type": "keyword" },
        "price": { "type": "float" },
        "title": { "type": "text" }
      }
    }
  }
}

curl -sk -u elastic:YourPassword \
  -XPUT "https://10.0.0.31:9200/_index_template/perftest_template" \
  -H "Content-Type: application/json" \
  -d @template.json

三个调优点解释：

• refresh_interval: 30s：默认1秒刷新一次segment，写入量大时每秒生成一个segment，merge压力大。改30s，查询能接受30s延迟的话，写入吞吐能涨50%以上。
• translog.durability: async：默认每次请求都fsync到磁盘，改为异步后每5秒刷一次。性能提升明显，代价是异常宕机会丢最多5秒数据。日志类场景可以接受，金融交易类别用。
• total_shards_per_node: 2：防止分片在节点间rebalance时打满磁盘IO。分片总数规划：单分片按20-40GB数据估算。本次测试6个分片，3个数据节点，每节点2个主分片+副本。

线程池队列调优

写入峰值时看到大量 429 EsRejectedExecutionException，去改 elasticsearch.yml：

thread_pool:
  write:
    queue_size: 20000

默认queue_size是1000，压测时单节点并发写入很快就堆满，直接拒绝新请求。调到20000后队列能扛住短时尖峰，但队列太长内存压力会变大，根据实际吞吐慢慢加。

熔断器调优

原崩盘日志里的 CircuitBreakingException 是parent熔断器触发了，说明JVM堆里已有的数据超过了总堆的95%。在 elasticsearch.yml 中调低熔断阈值，让它在更早的环节拒绝请求，而不是等堆快满了才动手：

indices.breaker.total.limit: 90%
indices.breaker.fielddata.limit: 40%
indices.breaker.request.limit: 30%

这是保护机制，不是性能参数。但它能避免节点因OOM整个掉出集群。原始默认95%太激进，堆到95%后GC根本来不及。

压测与效果数据

用Python写了一个bulk写入压测脚本，10个并发线程，每批1000条，一共200批。运行环境：Python 3.9.18 + requests 2.31.0。

# write_benchmark.py
import json
import time
import random
import requests
from concurrent.futures import ThreadPoolExecutor, as_completed

ES_HOST = "https://10.0.0.31:9200"
ES_USER = "elastic"
ES_PASS = "YourPassword"
CA_CERT = "/etc/elasticsearch/certs/ca.crt"
INDEX = "perftest"

BATCH_SIZE = 1000
CONCURRENCY = 10
TOTAL_BATCHES = 200

def send_batch(batch_index):
    docs = []
    base = batch_index * BATCH_SIZE
    for i in range(BATCH_SIZE):
        docs.append(f'{{"index":{{"_index":"{INDEX}","_id":{base + i}}}}}')
        docs.append(json.dumps({
            "timestamp": int(time.time() * 1000),
            "user_id": random.randint(1, 100000),
            "action": random.choice(["view", "click", "buy"]),
            "price": round(random.uniform(1, 500), 2),
            "title": f"product-{random.randint(1, 10000)}"
        }))
    payload = "\n".join(docs) + "\n"
    start = time.perf_counter()
    r = requests.post(
        f"{ES_HOST}/_bulk",
        data=payload,
        auth=(ES_USER, ES_PASS),
        verify=CA_CERT,
        headers={"Content-Type": "application/x-ndjson"},
        timeout=120
    )
    r.raise_for_status()
    resp = r.json()
    elapsed = time.perf_counter() - start
    return {
        "batch": batch_index,
        "took": resp.get("took", 0),
        "elapsed": elapsed,
        "errors": resp.get("errors", False)
    }

def main():
    start = time.perf_counter()
    results = []
    with ThreadPoolExecutor(max_workers=CONCURRENCY) as pool:
        futures = [pool.submit(send_batch, i) for i in range(TOTAL_BATCHES)]
        for f in as_completed(futures):
            results.append(f.result())

    total = TOTAL_BATCHES * BATCH_SIZE
    cost = time.perf_counter() - start
    error_batches = sum(1 for x in results if x["errors"])
    avg_elapsed = sum(x["elapsed"] for x in results) / len(results)
    print(f"总文档数: {total}")
    print(f"总耗时: {cost:.2f}s")
    print(f"吞吐量: {total / cost:.0f} docs/s")
    print(f"每次batch平均耗时: {avg_elapsed*1000:.1f}ms")
    print(f"bulk API平均耗时(took): {sum(x['took'] for x in results)/len(results)}ms")
    print(f"错误批次: {error_batches}")

if __name__ == "__main__":
    main()

方案B的压测输出：

总文档数: 200000
总耗时: 6.14s
吞吐量: 32573 docs/s
每次batch平均耗时: 294.7ms
bulk API平均耗时(took): 187ms
错误批次: 0

同样的脚本打到方案A旧集群上：

总文档数: 200000
总耗时: 12.2s
吞吐量: 16393 docs/s
每次batch平均耗时: 682.1ms
bulk API平均耗时(took): 421ms
错误批次: 0

再跑查询压测，20个并发线程，发range、term、match三类查询各200次：

# query_benchmark.py
import time
import requests
from concurrent.futures import ThreadPoolExecutor

ES_HOST = "https://10.0.0.31:9200"
ES_USER = "elastic"
ES_PASS = "YourPassword"
CA_CERT = "/etc/elasticsearch/certs/ca.crt"

def run_query(q):
    start = time.perf_counter()
    r = requests.get(
        f"{ES_HOST}/perftest/_search",
        auth=(ES_USER, ES_PASS),
        verify=CA_CERT,
        json={"query": q},
        timeout=60
    )
    r.raise_for_status()
    return (time.perf_counter() - start) * 1000

def main():
    queries = [
        {"range": {"timestamp": {"gte": "now-1h", "lt": "now"}}},
        {"term": {"action": "buy"}},
        {"match": {"title": "product 壳"}}
    ]
    for i, q in enumerate(queries):
        latencies = []
        with ThreadPoolExecutor(max_workers=20) as pool:
            futures = [pool.submit(run_query, q) for _ in range(200)]
            for f in futures:
                latencies.append(f.result())
        latencies.sort()
        p99 = latencies[int(len(latencies) * 0.99) - 1]
        avg = sum(latencies) / len(latencies)
        print(f"query{i+1}: avg={avg:.1f}ms p99={p99:.1f}ms")

if __name__ == "__main__":
    main()

方案B输出：

query1: avg=42.3ms p99=135.2ms
query2: avg=18.7ms p99=78.1ms
query3: avg=65.4ms p99=210.5ms

方案A输出：

query1: avg=231.5ms p99=820.3ms
query2: avg=160.2ms p99=584.9ms
query3: avg=352.8ms p99=1260.7ms

GC数据是压测后从 _nodes/stats/jvm 接口拉了一天：

JVM指标	方案A	方案B
Young GC平均耗时	25ms	12ms
Full GC次数	8次/天	0次/天
Old gen平均占用率	75%	40%

避坑指南

这轮部署踩了6个坑，每一个都花了几小时甚至一整天。

坑1：堆内存31GB的上限不是玄学

方案A用31GB是因为机器64GB，后来试过给一台测试节点堆设40GB。结果GC次数少了，但每次GC耗时反而长了，吞吐下降15%。原因是HotSpot JVM在超过32GB时会关闭普通对象指针压缩（-XX:-UseCompressedOops），内存寻址开销变大。堆越大不一定越快，超过32GB一定更慢。

坑2：CentOS 7上memory_lock永远失败

配好 bootstrap.memory_lock: true 后启动报错：

memory locking requested for elasticsearch process but memory is not locked

文件描述符和memlock明明在 /etc/security/limits.conf 里写了，就是不生效。查了半天发现systemd会覆盖limits.conf。必须加 /etc/systemd/system/elasticsearch.service.d/override.conf，然后 systemctl daemon-reload，重新 systemctl restart elasticsearch。不加这个，锁内存永远起不来。

坑3：证书IP SAN漏了导致节点间TLS握手失败

第一次生成证书没加 --ip，节点间transport层一直报 SSLHandshakeException。ES 8.x的TLS校验的是证书里的IP SAN，不是配置文件里的 network.host。生成证书时必须把每台节点的内网IP都写进对应证书的 --ip 参数里。

坑4：node.roles不能混着写data和data_hot

ES 8.x里 data 是 data_content + data_hot + data_warm + data_cold + data_frozen 的合集。我一开始想指定热数据角色，写了 node.roles: [ data, data_hot ]，启动直接报错："data" and one of data_content/data_hot/data_warm... are not allowed combined。普通场景只写 data 就好。

坑5：分片数拍脑袋设30，rebalance把集群打爆

方案A的索引建了30个分片，某天加了一台data节点，分片开始大规模rebalance，磁盘IO瞬间拉满，集群从green变yellow。分片数不是越大越好，按官方建议单分片20-40GB，我们单索引200GB，数据节点3台，最终定为6个主分片。再用 total_shards_per_node 限制单节点分片数，防止rebalance失控。

坑6：节点时钟不同步，master选举像抽风

三台master节点时钟差了2秒，集群频繁出现 discovery validation failed，某台一直选不上master。用NTP同步后问题消失。ES的选举对节点间时间戳敏感，别忽略这个基础项。

结论

这套分层架构上线后跑了4个月，高峰期写入量是原来崩盘时的1.5倍，集群状态稳定在green。方案A不是不能跑，但业务量一上来，master和data混用就是定时炸弹。如果你现在用的还是全角色混用，趁早拆开。过程不复杂，改配置、加三台小机器的事，换来的是master节点永远不被数据写入拖垮，查询和写入互不干扰。

所有配置文件都在上面，按顺序执行就能复现。有问题的评论区见。