中间件——日志平台

1、项目背景

上一家离开的公司，使用的技术是 springcloud 的技术，日志是分散在各处的。当时的技术经理，让我搭建一套属于公司自己的日志系统，我做了相应了调研。
有了以下的几个方案。

阿里日志系统，自带，可集成
开源的框架 Plumelog
传统的 ELK

2、方案分析

分析以上的框架和成本，最终决定用 plumelog 日志做就够用

每天线上的日志没有那么多，仅仅在商品大卖的时候日志多了些
ELK 需要再次学习的成本还是有的，而且需要的成本比较大
阿里的最方便，基本集成就可以了，但是有相关的费用

3、关键事项

为达成上述目标，需要完成哪些关键事项。

学习 PlumeLog 开源框架：
搭建 ES 服务
搭建对应的 springboot 服务

4、Plumelog 的日志架构图

5、搭建 PlumeLog 的服务

首先搭建对应的 docker 和 docker-compose 的运行环境
- 参考本知识库中的：Docker&Compose——安装
上传脚本到文件里面
- sh 一键安装脚本：add-plumelog.sh

chmod -R 777 init-data.sh
./init-data.sh
docker-compose up -d

等待运行完毕，校验安装的结果

这里面的地址记得自己换

6、搭建客户端

可以使用已经运行好的服务：https://gitee.com/lovepaul/open-holiday.git
也可以新建对应的 springboot 的微服务，如下代码三件套
1. pom 文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.7.0</version>
    <relativePath/> <!-- lookup parent from repository -->
  </parent>
  <groupId>com.yjc</groupId>
  <artifactId>open-holiday</artifactId>
  <version>1.0-SNAPSHOT</version>
  <name>open-holiday</name>
  <description>Demo project for Spring Boot</description>
  <properties>
    <java.version>1.8</java.version>
  </properties>
  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
    </dependency>



    <dependency>
      <groupId>de.codecentric</groupId>
      <artifactId>spring-boot-admin-starter-client</artifactId>
      <version>2.1.6</version>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>


    <dependency>
      <groupId>org.projectlombok</groupId>
      <artifactId>lombok</artifactId>
      <optional>true</optional>
      <version>1.18.0</version>
    </dependency>

    <dependency>
      <groupId>org.codehaus.janino</groupId>
      <artifactId>janino</artifactId>
      <version>3.0.6</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.codehaus.janino/commons-compiler -->
    <dependency>
      <groupId>org.codehaus.janino</groupId>
      <artifactId>commons-compiler</artifactId>
      <version>3.0.8</version>
    </dependency>


    <dependency>
      <groupId>cn.hutool</groupId>
      <artifactId>hutool-all</artifactId>
      <version>5.6.6</version>
    </dependency>

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-test</artifactId>
      <scope>test</scope>
    </dependency>

    <!--分布式日志收集plumelog-->
    <dependency>
      <groupId>com.plumelog</groupId>
      <artifactId>plumelog-logback</artifactId>
      <version>3.5.3</version>
    </dependency>
    <dependency>
      <groupId>com.plumelog</groupId>
      <artifactId>plumelog-trace</artifactId>
      <version>3.5.3</version>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-aop</artifactId>
      <version>2.1.11.RELEASE</version>
      <scope>provided</scope>
    </dependency>
    <dependency>
      <groupId>cn.hutool</groupId>
      <artifactId>hutool-core</artifactId>
      <version>5.5.8</version>
    </dependency>


  </dependencies>



  <build>
    <finalName>${project.artifactId}</finalName>
    <plugins>
      <plugin>
        <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
        </plugins>
        </build>

        </project>

properties

server.port=8083
spring.application.name=open-holiday
spring.redis.host=124.223.101.204
spring.redis.password=521521
plumelog.appName=${spring.application.name}
plumelog.redisHost=${spring.redis.host}
plumelog.redisAuth=${spring.redis.password}
spring.profiles.active=dev

logback-spring.xml 文件

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false">
    <conversionRule conversionWord="clr" converterClass="org.springframework.boot.logging.logback.ColorConverter"/>
    <conversionRule conversionWord="wex"
                    converterClass="org.springframework.boot.logging.logback.WhitespaceThrowableProxyConverter"/>
    <conversionRule conversionWord="wEx"
                    converterClass="org.springframework.boot.logging.logback.ExtendedWhitespaceThrowableProxyConverter"/>

    <!-- 增加如下的TLog MDC Listener -->


    <!-- 彩色日志格式 -->
    <property name="CONSOLE_LOG_PATTERN"
              value="${CONSOLE_LOG_PATTERN:-%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint}  %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}}"/>
    <!--输出到控制台-->
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <filter class="com.plumelog.logback.util.FilterSyncLogger">
            <level>info</level>
            <filterPackage>com.plumelog.trace.aspect.AbstractAspect</filterPackage>
        </filter>
        <encoder >
            <Pattern>${CONSOLE_LOG_PATTERN}</Pattern>
            <!-- 设置字符集 -->
            <charset>UTF-8</charset>
        </encoder>
    </appender>
    <!-- 输出到文件 -->
    <appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <FileNamePattern>logs/${plumelog.appName}.log.%d{yyyy-MM-dd}.log</FileNamePattern>
            <MaxHistory>3</MaxHistory>
        </rollingPolicy>
        <encoder >
            <Pattern>${CONSOLE_LOG_PATTERN}</Pattern>
            <!-- 设置字符集 -->
            <charset>UTF-8</charset>
        </encoder>
    </appender>
    <!-- 环境配置 -->
    <springProperty scope="context" name="plumelog.appName" source="plumelog.appName"/>
    <springProperty scope="context" name="plumelog.redisHost" source="plumelog.redisHost"/>
    <springProperty scope="context" name="plumelog.redisPort" source="plumelog.redisPort"/>
    <springProperty scope="context" name="plumelog.redisAuth" source="plumelog.redisAuth"/>
    <springProperty scope="context" name="plumelog.redisDb" source="plumelog.redisDb"/>
    <springProperty scope="context" name="plumelog.env" source="spring.profiles.active"/>
    <!-- 输出plumelog -->
    <appender name="plumelog" class="com.plumelog.logback.appender.RedisAppender">
        <appName>${plumelog.appName}</appName>
        <redisHost>${plumelog.redisHost}</redisHost>
        <redisAuth>${plumelog.redisAuth}</redisAuth>
        <redisDb>${plumelog.redisDb}</redisDb>
        <env>${plumelog.env}</env>
    </appender>
    <!-- 配置日志输出，只输出info，只保留控制台和plumelog输出-->
    <!-- 正常开发环境本地，只输出到控制台，测试环境只输出到plumelog,生产环境输出到本地文件plumelog,因为有plumelog加持本地文件就保留3天即可-->
    <!-- 这些都可以根据环境配置不同加载不同的ref->-->
    <root level="info">
        <!--输出到控制台-->
        <appender-ref ref="CONSOLE"/>
        <!-- 输出到文件 -->
        <appender-ref ref="file"/>
        <!-- 输出plumelog -->
        <appender-ref ref="plumelog"/>
    </root>

</configuration>

几个配置文件，主要是为了配置链路追踪的

package com.yjc.openholiday.config;


import com.plumelog.trace.aspect.AbstractAspect;
import org.aspectj.lang.JoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.springframework.stereotype.Component;


/**
* 方面配置
*
* @author admin
* @date 2023/03/04
*/
@Aspect
    @Component
    public class AspectConfig extends AbstractAspect {

        /**
* 围绕
*
* @param joinPoint 连接点
* @return {@link Object}
* @throws Throwable 可丢弃
*/// 注意这里要替换自己的包地址
        @Around("within(com.yjc..*))")
        public Object around(JoinPoint joinPoint) throws Throwable {
            return aroundExecute(joinPoint);
        }
    }

package com.yjc.openholiday.config;


import com.plumelog.core.TraceId;
import org.springframework.stereotype.Component;
import org.springframework.web.servlet.HandlerInterceptor;
import org.springframework.web.servlet.ModelAndView;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.util.UUID;


/**
* 拦截器
*
* @author admin
* @date 2023/03/04
*/
@Component
    public class Interceptor implements HandlerInterceptor {
        /**
* 预处理
*
* @param request  要求
* @param response 回答
* @param handler  处理器
* @return boolean
*/
        @Override
        public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) {
            //设置TraceID值，不埋此点链路ID就没有
            TraceId.logTraceID.set(UUID.randomUUID().toString().replaceAll("-", ""));
            return true;
        }

        /**
* 柱状把手
*
* @param request      要求
* @param response     回答
* @param handler      处理器
* @param modelAndView 模型和视图
*/
        @Override
        public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) {
        }

        /**
* 完工后
*
* @param request  要求
* @param response 回答
* @param handler  处理器
* @param ex       前任
*/
        @Override
        public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) {
        }
    }

package com.yjc.openholiday.config;


import org.springframework.context.annotation.Configuration;
import org.springframework.web.servlet.config.annotation.InterceptorRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;


/**
* 拦截器配置
*
* @author admin
* @date 2023/03/04
*/
@Configuration
    public class InterceptorConfig implements WebMvcConfigurer {
        /**
* 添加拦截器
*
* @param registry 注册表
*/
        @Override
        public void addInterceptors(InterceptorRegistry registry) {
            // 自定义拦截器，添加拦截路径和排除拦截路径
            registry.addInterceptor(new Interceptor()).addPathPatterns("/**");
        }
    }

package com.yjc.openholiday;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.ComponentScan;

@SpringBootApplication
    // 注意这个扫描包是开启链路追踪的
    @ComponentScan({"com.plumelog","com.yjc.openholiday"})
    public class OpenHolidayApplication {

        public static void main(String[] args) {
            SpringApplication.run(OpenHolidayApplication.class, args);
        }

    }

编写对应的 controller 的接口

package com.yjc.openholiday;

import cn.hutool.core.date.DateUtil;
import cn.hutool.core.util.IdUtil;
import cn.hutool.core.util.ObjectUtil;
import cn.hutool.core.util.RandomUtil;
import cn.hutool.core.util.StrUtil;
import cn.hutool.http.HttpRequest;
import cn.hutool.http.HttpResponse;
import cn.hutool.http.HttpUtil;
import cn.hutool.json.JSONObject;
import cn.hutool.json.JSONUtil;
import lombok.extern.slf4j.Slf4j;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.Date;
import java.util.Set;

@Slf4j
    @RestController
    public class HolidayController {


        @GetMapping("/test")
        public Integer getCurrentDataIsHoliday() {
            String currentYear = DateUtil.format(new Date(), "yyyy");
            String currentDate = DateUtil.format(new Date(), "MM-dd");

            log.info("getCurrentDataIsHoliday-{}----{}", currentYear, currentDate);
            String url = "http://timor.tech/api/holiday/year/" + currentYear;
            HttpRequest get = HttpUtil.createGet(url);
            HttpResponse execute = get.execute();
            String s = execute.body();
            if (StrUtil.isNotEmpty(s)) {
                JSONObject jsonObject = JSONUtil.parseObj(s);
                Object holiday = jsonObject.get("holiday");
                JSONObject holidayJsonObj = JSONUtil.parseObj(holiday);
                Set<String> allHolidayDate = holidayJsonObj.keySet();
                log.info("getCurrentDataIsHoliday-{}", allHolidayDate);
                if (ObjectUtil.isNotEmpty(allHolidayDate) && allHolidayDate.contains(currentDate)) {
                    return 0;
                } else {
                    return 1;
                }
            }
            return 1;
        }

        @GetMapping("/random")
        public String random() {
            log.info(RandomUtil.randomInt() + "");
            return "1.0_" + IdUtil.fastSimpleUUID();
        }

        @GetMapping("/errorlog")
        public String errorlog() {
            log.info("我测一下");
            int a = 1 / 0;
            return "1.0_" + IdUtil.fastSimpleUUID();
        }


        @GetMapping("/test2")
        public Integer getCurrentDataIsHoliday2() {
            String currentYear = DateUtil.format(new Date(), "yyyy");
            String currentDate = DateUtil.format(new Date(), "MM-dd");

            log.info("getCurrentDataIsHoliday-{}----{}", currentYear, currentDate);
            String url = "http://timor.tech/api/holiday/year/" + currentYear;
            HttpRequest get = HttpUtil.createGet(url);
            HttpResponse execute = get.execute();
            String s = execute.body();
            if (StrUtil.isNotEmpty(s)) {
                JSONObject jsonObject = JSONUtil.parseObj(s);
                Object holiday = jsonObject.get("holiday");
                JSONObject holidayJsonObj = JSONUtil.parseObj(holiday);
                Set<String> allHolidayDate = holidayJsonObj.keySet();
                log.info("getCurrentDataIsHoliday-{}", allHolidayDate);
                if (ObjectUtil.isNotEmpty(allHolidayDate) && allHolidayDate.contains(currentDate)) {
                    return 0;
                        } else {
                        return 1;
                        }
                        }
                        return 1;
                        }


                        }

7、检验结果

中间件——日志平台

http://example.com/2023/03/03/qleyofzi4m3x6y6n/

作者

杨靖成

发布于

2023年3月3日

许可协议

GC-案例上一篇

Docker——Nacos（单机）下一篇