一、真实场景：一个超时问题查了三天

2024年Q2，我们一个核心API接口偶发超时（P99从200ms飙到3s）。日志里只有"timeout"，没有调用链。我们翻遍了Nginx、PHP-FPM、MySQL慢查询，最后发现是Redis集群中一个节点网络抖动，导致某个缓存查询阻塞了3秒。

这个排查过程用了3天。如果当时有链路追踪，10分钟就能定位。痛定思痛，我们决定上分布式链路追踪。

二、方案对比：Jaeger vs Zipkin

选型时我们对比了Jaeger 1.53.0和Zipkin 3.4.0，基于以下维度：

三、部署与接入：完整代码实现

3.1 Jaeger部署（Docker Compose）

# docker-compose-jaeger.yml
version: '3.8'
services:
  jaeger-collector:
    image: jaegertracing/jaeger-collector:1.53.0
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
      - ES_SERVER_URLS=http://elasticsearch:9200
      - COLLECTOR_OTLP_ENABLED=true
    ports:
      - "14269:14269"  # admin
      - "14268:14268"  # HTTP
      - "4317:4317"    # gRPC OTLP
      - "4318:4318"    # HTTP OTLP
    depends_on:
      - elasticsearch

  jaeger-query:
    image: jaegertracing/jaeger-query:1.53.0
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
      - ES_SERVER_URLS=http://elasticsearch:9200
    ports:
      - "16686:16686"  # UI
    depends_on:
      - jaeger-collector

  jaeger-agent:
    image: jaegertracing/jaeger-agent:1.53.0
    environment:
      - REPORTER_GRPC_HOST_PORT=jaeger-collector:14250
    ports:
      - "5775:5775/udp"  # Zipkin thrift
      - "6831:6831/udp"  # Jaeger thrift
      - "6832:6832/udp"  # Jaeger binary
      - "5778:5778"      # config
    depends_on:
      - jaeger-collector

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
    ports:
      - "9200:9200"
    volumes:
      - es_data:/usr/share/elasticsearch/data

volumes:
  es_data:

3.2 Zipkin部署（Docker Compose）

# docker-compose-zipkin.yml
version: '3.8'
services:
  zipkin:
    image: openzipkin/zipkin:3.4.0
    environment:
      - STORAGE_TYPE=elasticsearch
      - ES_HOSTS=http://elasticsearch:9200
      - JAVA_OPTS=-Xms512m -Xmx512m
    ports:
      - "9411:9411"  # UI + API
    depends_on:
      - elasticsearch

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
    ports:
      - "9200:9200"
    volumes:
      - es_data:/usr/share/elasticsearch/data

volumes:
  es_data:

3.3 PHP SDK接入（OpenTelemetry + Jaeger）

// composer require open-telemetry/opentelemetry
// composer require open-telemetry/exporter-otlp
// PHP 8.3 + Laravel 11

use OpenTelemetry\API\Globals;
use OpenTelemetry\API\Trace\SpanKind;
use OpenTelemetry\SDK\Trace\TracerProvider;
use OpenTelemetry\SDK\Trace\SpanProcessor\BatchSpanProcessor;
use OpenTelemetry\SDK\Trace\Exporter\Otlp\OtlpHttpExporter;
use OpenTelemetry\SDK\Common\Export\StreamTransport;
use OpenTelemetry\SDK\Common\Time\Clock;
use OpenTelemetry\SDK\Trace\Sampler\AlwaysOnSampler;

// 初始化TracerProvider
$exporter = new OtlpHttpExporter(
    new StreamTransport('http://jaeger-collector:4318/v1/traces', 'POST'),
    Clock::getDefault()
);

$spanProcessor = new BatchSpanProcessor($exporter, Clock::getDefault(), 2048);
$tracerProvider = new TracerProvider($spanProcessor, new AlwaysOnSampler());

// 设置全局Tracer
Globals::setTracerProvider($tracerProvider);

// 在业务代码中使用
$tracer = Globals::tracer();
$span = $tracer->spanBuilder('order.create')
    ->setSpanKind(SpanKind::KIND_SERVER)
    ->startSpan();

try {
    // 业务逻辑
    $span->setAttribute('order_id', 12345);
    $span->setAttribute('user_id', 67890);
    
    // 创建子span
    $childSpan = $tracer->spanBuilder('redis.get')
        ->setParent($span)
        ->startSpan();
    // Redis操作
    $childSpan->end();
    
    // 数据库查询
    $dbSpan = $tracer->spanBuilder('mysql.query')
        ->setParent($span)
        ->startSpan();
    // SQL执行
    $dbSpan->end();
    
} catch (\Throwable $e) {
    $span->recordException($e);
    $span->setStatus(\OpenTelemetry\API\Trace\StatusCode::STATUS_ERROR, $e->getMessage());
    throw $e;
} finally {
    $span->end();
    $tracerProvider->forceFlush();
}

3.4 Zipkin PHP SDK接入

// composer require zipkin/zipkin
// composer require guzzlehttp/guzzle

use Zipkin\TracingBuilder;
use Zipkin\Samplers\BinarySampler;
use Zipkin\Reporters\Http;
use Zipkin\Endpoint;

// 初始化Tracing
$endpoint = Endpoint::create('order-service', '127.0.0.1', 8080);
$reporter = new Http(['endpoint_url' => 'http://zipkin:9411/api/v2/spans']);
$sampler = BinarySampler::createAsAlwaysSample();

$tracing = TracingBuilder::create()
    ->havingLocalEndpoint($endpoint)
    ->havingSampler($sampler)
    ->havingReporter($reporter)
    ->build();

$tracer = $tracing->getTracer();

// 创建Span
$span = $tracer->newTrace();
$span->setName('order.create');
$span->start();

try {
    // 业务逻辑
    $span->tag('order_id', '12345');
    $span->tag('user_id', '67890');
    
    // 子span
    $childSpan = $tracer->newChild($span->getContext());
    $childSpan->setName('redis.get');
    $childSpan->start();
    // Redis操作
    $childSpan->finish();
    
} catch (\Throwable $e) {
    $span->tag('error', $e->getMessage());
    throw $e;
} finally {
    $span->finish();
    $tracer->flush();
}

3.5 采样策略配置（Jaeger远程采样）

// 通过Jaeger Collector API配置采样策略
// POST http://jaeger-collector:14269/api/sampling
{
  "service": "order-service",
  "strategy_type": "probabilistic",
  "probabilistic_sampling": {
    "sampling_rate": 0.1
  }
}

// 或者使用速率限制采样
{
  "service": "payment-service",
  "strategy_type": "rate_limiting",
  "rate_limiting_sampling": {
    "max_traces_per_second": 10
  }
}

四、效果数据：性能开销对比

测试环境：PHP 8.3.2, Laravel 11.0.5, MySQL 8.0.35, Redis 7.2.4, 4核8G云服务器。使用wrk压测，每个请求包含1次Redis查询+1次MySQL查询。

结论：Jaeger性能开销更小（OTLP协议更高效），Zipkin存储更省（数据压缩更好）。

五、避坑指南（5个真实踩坑记录）

坑1：Span ID冲突导致链路断裂

现象：同一个trace的span显示在不同trace里。原因：PHP的uniqid()在高并发下生成重复ID。解决：使用OpenTelemetry自带的UUID生成器。

// 错误做法
$spanId = uniqid(); // 高并发下重复

// 正确做法
use OpenTelemetry\SDK\Trace\SpanContext;
$spanId = SpanContext::generateSpanId(); // 128位随机ID

坑2：BatchSpanProcessor缓冲区溢出

现象：高并发下span丢失。原因：默认缓冲区2048，QPS>2000时溢出。解决：增大缓冲区并启用独立进程上报。

// 增大缓冲区
$spanProcessor = new BatchSpanProcessor(
    $exporter, 
    Clock::getDefault(), 
    8192,  // 缓冲区大小
    5000   // 超时时间(ms)
);

坑3：Zipkin HTTP上报阻塞业务

现象：Zipkin上报失败导致业务接口超时。原因：同步HTTP上报，网络抖动时阻塞。解决：使用异步上报或消息队列。

// 使用异步HTTP客户端
use GuzzleHttp\Client;
use GuzzleHttp\Promise;

$client = new Client(['timeout' => 2]);
$promises = [];
foreach ($spans as $span) {
    $promises[] = $client->postAsync('http://zipkin:9411/api/v2/spans', [
        'json' => $span
    ]);
}
Promise\settle($promises)->wait();

坑4：Elasticsearch索引膨胀

现象：Jaeger每天产生大量索引，磁盘很快占满。原因：Jaeger默认按天创建索引，未设置生命周期。解决：配置ILM策略。

// Elasticsearch ILM策略
PUT _ilm/policy/jaeger-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "1d"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

坑5：采样率设置不当导致成本失控

现象：100%采样导致存储成本飙升10倍。原因：生产环境误用AlwaysOnSampler。解决：生产环境使用概率采样，建议0.1%-1%。

// 生产环境采样配置
use OpenTelemetry\SDK\Trace\Sampler\TraceIdRatioBasedSampler;

$sampler = new TraceIdRatioBasedSampler(0.01); // 1%采样
$tracerProvider = new TracerProvider($spanProcessor, $sampler);

六、选型建议

• 选Jaeger：对性能敏感、需要远程采样配置、团队熟悉Go生态
• 选Zipkin：部署简单、存储成本敏感、团队熟悉Java生态
• 都别选：日均请求<10万，用ELK+自定义日志即可

我们最终选了Jaeger，因为性能开销小10%，且远程采样配置方便。但如果你只有3-5个服务，Zipkin的简单部署更合适。