1. 為什么需要多服務日志集中管理在微服務架構中一個典型的生產環(huán)境往往包含多個后端服務和前端應用。以我最近部署的一個電商平臺為例整套系統(tǒng)包含3個Node.js后端服務用戶服務/訂單服務/商品服務2個Vue前端應用主站/管理后臺1個Nginx網(wǎng)關Redis和MySQL各1個實例當所有服務都通過Docker Compose部署后最頭疼的問題就是日志分散在各個容器中。上周排查一個支付失敗問題時我不得不同時docker logs order-service查看訂單服務日志docker logs payment-service查看支付服務日志docker logs nginx查看網(wǎng)關日志再到Kibana中篩選前端錯誤日志這種碎片化的排查方式效率極低特別是當多個服務間存在調用鏈時。更糟的是docker logs默認只保留最新日志歷史數(shù)據(jù)隨時可能丟失。2. 基礎版方案直接掛載日志卷2.1 目錄結構設計我采用的日志目錄結構如下/project /services /backend - docker-compose.yml /frontend - docker-compose.yml /logs /backend /user-service /order-service /product-service /frontend /main-site /admin-console /infra /nginx /mysql2.2 關鍵Compose配置以訂單服務為例的配置片段version: 3.8 services: order-service: image: registry.example.com/order:v1.2 volumes: - /project/logs/backend/order-service:/var/log/app logging: driver: json-file options: max-size: 10m max-file: 32.3 實際踩坑記錄權限問題首次啟動時出現(xiàn)Permission denied錯誤解決方法mkdir -p /project/logs/backend/order-service chmod -R 777 /project/logs # 生產環(huán)境應配置精確權限日志輪轉失效發(fā)現(xiàn)日志文件超過10MB后未切割原因是必須同時設置max-size和max-file某些應用如Nginx需要在其配置中也開啟日志輪轉磁盤空間預警建議添加監(jiān)控腳本#!/bin/bash LOG_DIR/project/logs THRESHOLD90 current_usage$(df -h $LOG_DIR | awk NR2 {print $5} | tr -d %) if [ $current_usage -gt $THRESHOLD ]; then echo WARNING: Log directory usage $current_usage% | mail -s Log Alert adminexample.com fi3. 進階版方案EFK全棧日志系統(tǒng)3.1 架構設計graph TD A[Frontend] --|Log| B(Fluentd) C[Backend] --|Log| B D[Database] --|Log| B B -- E[Elasticsearch] E -- F[Kibana]3.2 關鍵組件配置3.2.1 Fluentd配置fluentd.conf核心部分source type forward port 24224 /source match docker.** type elasticsearch host elasticsearch port 9200 logstash_format true logstash_prefix docker flush_interval 5s /match3.2.2 Elasticsearch調優(yōu)docker-compose.efk.yml內存設置services: elasticsearch: image: elasticsearch:7.17.0 environment: - ES_JAVA_OPTS-Xms2g -Xmx2g # 根據(jù)服務器內存調整 - discovery.typesingle-node ulimits: memlock: soft: -1 hard: -13.3 性能優(yōu)化技巧日志過濾在Fluentd中丟棄DEBUG級別日志filter docker.** type grep exclude key log pattern /DEBUG/ /exclude /filter索引策略每天自動創(chuàng)建新索引match docker.** type elasticsearch logstash_date_format %Y%m%d /match查詢加速為常用字段添加索引PUT /docker-*/_mapping { properties: { container_name: { type: keyword }, service: { type: keyword } } }4. 混合部署實戰(zhàn)案例4.1 場景需求開發(fā)環(huán)境使用基礎版日志卷生產環(huán)境使用EFK全棧方案兩種環(huán)境配置無縫切換4.2 條件化Compose配置x-logging: default-logging driver: ${LOG_DRIVER:-json-file} options: max-size: 10m max-file: 3 x-efk-logging: efk-logging driver: fluentd options: fluentd-address: fluentd:24224 tag: docker.{{.Name}} services: webapp: image: myapp:${TAG:-latest} logging: : *${LOG_CONFIG:-default-logging}啟動命令示例# 開發(fā)環(huán)境 LOG_DRIVERjson-file docker-compose up # 生產環(huán)境 LOG_CONFIGefk-logging TAGprod docker-compose -f docker-compose.yml -f docker-compose.efk.yml up4.3 監(jiān)控指標設計Grafana儀表盤建議包含日志量趨勢按服務/級別錯誤日志TOP 10日志存儲增長預測關鍵服務響應時間關聯(lián)分析5. 異常處理手冊5.1 Fluentd常見問題癥狀日志未出現(xiàn)在Kibana中 排查步驟檢查Fluentd容器狀態(tài)docker-compose logs fluentd | grep -i error驗證Elasticsearch連接curl -XGET http://elasticsearch:9200/_cat/indices?v查看緩沖隊列docker exec fluentd ls -lh /var/log/fluentd/buffer5.2 日志丟失預防啟用本地備份雙寫策略match docker.** type copy store type elasticsearch ... /store store type file path /var/log/fluentd/backup /store /match設置磁盤水位警戒線docker run -it --rm --privileged --pidhost debian nsenter -t 1 -m -u -n -i sh -c echo 90 /proc/sys/vm/dirty_ratio6. 安全加固方案6.1 訪問控制Kibana基礎認證services: kibana: environment: - ELASTICSEARCH_USERNAMEadmin - ELASTICSEARCH_PASSWORD${ES_PASSWORD}Elasticsearch TLS加密# 生成證書 docker run --rm elasticsearch:7.17.0 \ bin/elasticsearch-certutil cert -out /tmp/elastic-certificates.p12 -pass # 掛載到容器 volumes: - ./certs/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p126.2 日志脫敏Fluentd過濾器示例filter docker.** type record_transformer enable_ruby true record message ${record[message].gsub(/\b\d{4}-\d{4}-\d{4}-\d{4}\b/, [CREDIT_CARD])} /record /filter7. 成本優(yōu)化實踐7.1 冷熱數(shù)據(jù)分離PUT _ilm/policy/logs_policy { policy: { phases: { hot: { actions: { rollover: { max_size: 50GB, max_age: 7d } } }, warm: { min_age: 7d, actions: { allocate: { number_of_replicas: 1 } } } } } }7.2 日志采樣配置對DEBUG日志進行50%采樣filter docker.** type sample rate 50 sample_key level sample_value DEBUG /filter 微服務架構下的日志集中管理實踐與EFK方案
1. 為什么需要多服務日志集中管理在微服務架構中一個典型的生產環(huán)境往往包含多個后端服務和前端應用。以我最近部署的一個電商平臺為例整套系統(tǒng)包含3個Node.js后端服務用戶服務/訂單服務/商品服務2個Vue前端應用主站/管理后臺1個Nginx網(wǎng)關Redis和MySQL各1個實例當所有服務都通過Docker Compose部署后最頭疼的問題就是日志分散在各個容器中。上周排查一個支付失敗問題時我不得不同時docker logs order-service查看訂單服務日志docker logs payment-service查看支付服務日志docker logs nginx查看網(wǎng)關日志再到Kibana中篩選前端錯誤日志這種碎片化的排查方式效率極低特別是當多個服務間存在調用鏈時。更糟的是docker logs默認只保留最新日志歷史數(shù)據(jù)隨時可能丟失。2. 基礎版方案直接掛載日志卷2.1 目錄結構設計我采用的日志目錄結構如下/project /services /backend - docker-compose.yml /frontend - docker-compose.yml /logs /backend /user-service /order-service /product-service /frontend /main-site /admin-console /infra /nginx /mysql2.2 關鍵Compose配置以訂單服務為例的配置片段version: 3.8 services: order-service: image: registry.example.com/order:v1.2 volumes: - /project/logs/backend/order-service:/var/log/app logging: driver: json-file options: max-size: 10m max-file: 32.3 實際踩坑記錄權限問題首次啟動時出現(xiàn)Permission denied錯誤解決方法mkdir -p /project/logs/backend/order-service chmod -R 777 /project/logs # 生產環(huán)境應配置精確權限日志輪轉失效發(fā)現(xiàn)日志文件超過10MB后未切割原因是必須同時設置max-size和max-file某些應用如Nginx需要在其配置中也開啟日志輪轉磁盤空間預警建議添加監(jiān)控腳本#!/bin/bash LOG_DIR/project/logs THRESHOLD90 current_usage$(df -h $LOG_DIR | awk NR2 {print $5} | tr -d %) if [ $current_usage -gt $THRESHOLD ]; then echo WARNING: Log directory usage $current_usage% | mail -s Log Alert adminexample.com fi3. 進階版方案EFK全棧日志系統(tǒng)3.1 架構設計graph TD A[Frontend] --|Log| B(Fluentd) C[Backend] --|Log| B D[Database] --|Log| B B -- E[Elasticsearch] E -- F[Kibana]3.2 關鍵組件配置3.2.1 Fluentd配置fluentd.conf核心部分source type forward port 24224 /source match docker.** type elasticsearch host elasticsearch port 9200 logstash_format true logstash_prefix docker flush_interval 5s /match3.2.2 Elasticsearch調優(yōu)docker-compose.efk.yml內存設置services: elasticsearch: image: elasticsearch:7.17.0 environment: - ES_JAVA_OPTS-Xms2g -Xmx2g # 根據(jù)服務器內存調整 - discovery.typesingle-node ulimits: memlock: soft: -1 hard: -13.3 性能優(yōu)化技巧日志過濾在Fluentd中丟棄DEBUG級別日志filter docker.** type grep exclude key log pattern /DEBUG/ /exclude /filter索引策略每天自動創(chuàng)建新索引match docker.** type elasticsearch logstash_date_format %Y%m%d /match查詢加速為常用字段添加索引PUT /docker-*/_mapping { properties: { container_name: { type: keyword }, service: { type: keyword } } }4. 混合部署實戰(zhàn)案例4.1 場景需求開發(fā)環(huán)境使用基礎版日志卷生產環(huán)境使用EFK全棧方案兩種環(huán)境配置無縫切換4.2 條件化Compose配置x-logging: default-logging driver: ${LOG_DRIVER:-json-file} options: max-size: 10m max-file: 3 x-efk-logging: efk-logging driver: fluentd options: fluentd-address: fluentd:24224 tag: docker.{{.Name}} services: webapp: image: myapp:${TAG:-latest} logging: : *${LOG_CONFIG:-default-logging}啟動命令示例# 開發(fā)環(huán)境 LOG_DRIVERjson-file docker-compose up # 生產環(huán)境 LOG_CONFIGefk-logging TAGprod docker-compose -f docker-compose.yml -f docker-compose.efk.yml up4.3 監(jiān)控指標設計Grafana儀表盤建議包含日志量趨勢按服務/級別錯誤日志TOP 10日志存儲增長預測關鍵服務響應時間關聯(lián)分析5. 異常處理手冊5.1 Fluentd常見問題癥狀日志未出現(xiàn)在Kibana中 排查步驟檢查Fluentd容器狀態(tài)docker-compose logs fluentd | grep -i error驗證Elasticsearch連接curl -XGET http://elasticsearch:9200/_cat/indices?v查看緩沖隊列docker exec fluentd ls -lh /var/log/fluentd/buffer5.2 日志丟失預防啟用本地備份雙寫策略match docker.** type copy store type elasticsearch ... /store store type file path /var/log/fluentd/backup /store /match設置磁盤水位警戒線docker run -it --rm --privileged --pidhost debian nsenter -t 1 -m -u -n -i sh -c echo 90 /proc/sys/vm/dirty_ratio6. 安全加固方案6.1 訪問控制Kibana基礎認證services: kibana: environment: - ELASTICSEARCH_USERNAMEadmin - ELASTICSEARCH_PASSWORD${ES_PASSWORD}Elasticsearch TLS加密# 生成證書 docker run --rm elasticsearch:7.17.0 \ bin/elasticsearch-certutil cert -out /tmp/elastic-certificates.p12 -pass # 掛載到容器 volumes: - ./certs/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p126.2 日志脫敏Fluentd過濾器示例filter docker.** type record_transformer enable_ruby true record message ${record[message].gsub(/\b\d{4}-\d{4}-\d{4}-\d{4}\b/, [CREDIT_CARD])} /record /filter7. 成本優(yōu)化實踐7.1 冷熱數(shù)據(jù)分離PUT _ilm/policy/logs_policy { policy: { phases: { hot: { actions: { rollover: { max_size: 50GB, max_age: 7d } } }, warm: { min_age: 7d, actions: { allocate: { number_of_replicas: 1 } } } } } }7.2 日志采樣配置對DEBUG日志進行50%采樣filter docker.** type sample rate 50 sample_key level sample_value DEBUG /filter