利用 Filebeat 收集日志（中）

1、案例: 利用 Filebeat 收集日志到 ELasticsearch

默认生成的索引名称为 filebeat-<版本>-<时间>*

1.1、修改配置

root@web01:~# vim /etc/filebeat/filebeat.yml

filebeat.inputs:
- type: log

  id: my-apache-id

  enabled: true

  paths:
    - /var/log/apache2/access.log

output.elasticsearch:
  hosts: ["192.168.1.101:9200"]

1.2、启动服务

root@web01:~# systemctl enable --now filebeat.service 

1.3、head 插件查看索引

1.4、通过 Kibana 查看收集的日志信息

2、案例: 自定义索引名称收集所有系统日志到 ELasticsearch

2.1、修改配置

范例：收集系统日志

root@web01:~# vim /etc/rsyslog.conf
.....
*.* /var/log/system.log

root@web01:~# systemctl restart rsyslog.service

root@web01:~# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
 enabled: true                       #开启日志
 paths:
 - /var/log/syslog.log               #指定收集的日志文件
 include_lines: ['failed', 'nginx']  #只过滤指定包含关健字的日志
  #exclude_lines: ['Debug']           #排除包含关健字的日志

  #paths: /var/log/*
  #exclude_files: ['.gz$']           #排除文件名包含关健字的日志文件
output.elasticsearch:
 hosts: ["192.168.1.101:9200"]        #指定ES集群服务器地址和端口
 index: "test-%{[agent.version]}-%{+yyyy.MM.dd}" #自定义索引名称

setup.ilm.enabled: false #关闭索引生命周期ilm功能，默认开启时索引名称只能为filebeat-*
setup.template.name: "test" #定义模板名称,要自定义索引名称,必须指定此项,否则无法启动
setup.template.pattern: "test-*" #定义模板的匹配索引名称,要自定义索引名称,必须指定此项,否则无法启动
#默认情况下 Filebeat 写入到 ES 的索引分片为1，如果需要修订分片，可以通过如下实现
#注意:如果模板已经存在,需要利用插件先删除模板和索引才能生效
setup.template.settings:
 index.number_of_shards: 3
 index.number_of_replicas: 0

范例：收集Nginx的访问日志

root@web01:~# vim /etc/filebeat/filebeat.yml

filebeat.inputs:
- type: log
 enabled: true                             #开启日志
 paths:
  - /var/log/nginx/access.log               #指定收集的日志文件
output.elasticsearch:
 hosts: ["192.168.1.101:9200"]        #指定ES集群服务器地址和端口
 index: "nginx-access-%{[agent.version]}-%{+yyyy.MM.dd}" #自定义索引名称
setup.ilm.enabled: false
setup.template.name: "nginx" #定义模板名称,要自定义索引名称,必须指定此项,否则无法启动
setup.template.pattern: "nginx-*" #定义模板的匹配索引名称,要自定义索引名称,必须指定此项,否则无法启动

2.2、插件查看索引

2.3、通过kibana查看收集的日志信息

创建索引模式，然后查看

3、案例: 利用 Filebeat 收集 Nginx Json 格式日志到Elasticsearch

生产环境中我们经常需要获取Web访问用户的信息，比如：来源的IP是哪个地域，网站的PV、UV、状态码、访问时间等等;所以需要收集的Nginx访问日志

3.1、安装 nginx 配置访问日志使用 Json格式

# 修改nginx访问日志为Json格式
root@web01:~# vim /etc/nginx/nginx.conf

        log_format access_json '{"@timestamp":"$time_iso8601",'
            '"host":"$server_addr",'
            '"clientip":"$remote_addr",'
            '"size":$body_bytes_sent,'
            '"responsetime":$request_time,'
            '"upstreamtime":"$upstream_response_time",'
            '"upstreamhost":"$upstream_addr",'
            '"http_host":"$host",'
            '"uri":"$uri",'
            '"domain":"$host",'
            '"xff":"$http_x_forwarded_for",'
            '"referer":"$http_referer",'
            '"tcp_xff":"$proxy_protocol_addr",'
            '"http_user_agent":"$http_user_agent",'
            '"status":"$status"}';
        access_log /var/log/nginx/access_json.log access_json;
        error_log /var/log/nginx/error.log;

# 默认开启nginx的错误日志,但如果是ubuntu,还需要修改下面行才能记录错误日志
root@web01:~# vim /etc/nginx/sites-available/default
        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                # try_files $uri $uri/ =404;  # 将此行注释

        }

root@web01:~# tail -f /var/log/nginx/access_json.log
192.168.1.1 - - [03/Jan/2023:11:18:19 +0000] "GET /favicon.ico HTTP/1.1" 404 197 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36 SE 2.X MetaSr 1.0"
192.168.1.105 - - [03/Jan/2023:11:18:25 +0000] "GET / HTTP/1.1" 200 12 "-" "Elastic-Heartbeat/7.17.8 (linux; amd64; 692b4aac606e457bd2f5ef092d2d23c2fa950828; 2022-12-03 00:38:18 +0000 UTC)"
192.168.1.105 - - [03/Jan/2023:11:18:35 +0000] "GET / HTTP/1.1" 200 12 "-" "Elastic-Heartbeat/7.17.8 (linux; amd64; 692b4aac606e457bd2f5ef092d2d23c2fa950828; 2022-12-03 00:38:18 +0000 UTC)"
192.168.1.105 - - [03/Jan/2023:11:18:45 +0000] "GET / HTTP/1.1" 200 12 "-" "Elastic-Heartbeat/7.17.8 (linux; amd64; 692b4aac606e457bd2f5ef092d2d23c2fa950828; 2022-12-03 00:38:18 +0000 UTC)"
192.168.1.105 - - [03/Jan/2023:11:18:55 +0000] "GET / HTTP/1.1" 200 12 "-" "Elastic-Heartbeat/7.17.8 (linux; amd64; 692b4aac606e457bd2f5ef092d2d23c2fa950828; 2022-12-03 00:38:18 +0000 UTC)"
192.168.1.105 - - [03/Jan/2023:11:19:05 +0000] "GET / HTTP/1.1" 200 12 "-" "Elastic-Heartbeat/7.17.8 (linux; amd64; 692b4aac606e457bd2f5ef092d2d23c2fa950828; 2022-12-03 00:38:18 +0000 UTC)"
192.168.1.105 - - [03/Jan/2023:11:19:15 +0000] "GET / HTTP/1.1" 200 12 "-" "Elastic-Heartbeat/7.17.8 (linux; amd64; 692b4aac606e457bd2f5ef092d2d23c2fa950828; 2022-12-03 00:38:18 +0000 UTC)"
192.168.1.105 - - [03/Jan/2023:11:19:25 +0000] "GET / HTTP/1.1" 200 12 "-" "Elastic-Heartbeat/7.17.8 (linux; amd64; 692b4aac606e457bd2f5ef092d2d23c2fa950828; 2022-12-03 00:38:18 +0000 UTC)"
{"@timestamp":"2023-01-03T11:19:35+00:00","host":"192.168.1.105","clientip":"192.168.1.105","size":12,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.1.105","uri":"/index.html","domain":"192.168.1.105","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"Elastic-Heartbeat/7.17.8 (linux; amd64; 692b4aac606e457bd2f5ef092d2d23c2fa950828; 2022-12-03 00:38:18 +0000 UTC)","status":"200"}
{"@timestamp":"2023-01-03T11:19:45+00:00","host":"192.168.1.105","clientip":"192.168.1.105","size":12,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.1.105","uri":"/index.html","domain":"192.168.1.105","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"Elastic-Heartbeat/7.17.8 (linux; amd64; 692b4aac606e457bd2f5ef092d2d23c2fa950828; 2022-12-03 00:38:18 +0000 UTC)","status":"200"}
{"@timestamp":"2023-01-03T11:19:55+00:00","host":"192.168.1.105","clientip":"192.168.1.105","size":12,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.1.105","uri":"/index.html","domain":"192.168.1.105","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"Elastic-Heartbeat/7.17.8 (linux; amd64; 692b4aac606e457bd2f5ef092d2d23c2fa950828; 2022-12-03 00:38:18 +0000 UTC)","status":"200"}


root@web01:~# tail /var/log/nginx/error.log
2023/01/03 10:52:53 [emerg] 21596#21596: bind() to [::]:80 failed (98: Address already in use)
2023/01/03 10:52:53 [emerg] 21596#21596: bind() to 0.0.0.0:80 failed (98: Address already in use)
2023/01/03 10:52:53 [emerg] 21596#21596: bind() to [::]:80 failed (98: Address already in use)
2023/01/03 10:52:53 [emerg] 21596#21596: bind() to 0.0.0.0:80 failed (98: Address already in use)
2023/01/03 10:52:53 [emerg] 21596#21596: bind() to [::]:80 failed (98: Address already in use)
2023/01/03 10:52:53 [emerg] 21596#21596: bind() to 0.0.0.0:80 failed (98: Address already in use)
2023/01/03 10:52:53 [emerg] 21596#21596: bind() to [::]:80 failed (98: Address already in use)
2023/01/03 10:52:53 [emerg] 21596#21596: still could not bind()
2023/01/03 11:14:06 [emerg] 23468#23468: unknown log format "access_json" in /etc/nginx/nginx.conf:41
2023/01/03 11:18:19 [error] 23629#23629: *17 open() "/var/www/html/favicon.ico" failed (2: No such file or directory), client: 192.168.1.1, server: _, request: "GET /favicon.ico HTTP/1.1", host: "192.168.1.105"

3.2、修改 Filebeat 配置文件

root@web01:~# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access_json.log
  json.keys_under_root: true #默认false会将全部数据存储至message字段，改为true则会以
Json格式存储
  json.overwrite_keys: true  #设为true,覆盖默认的message字段，使用自定义json格式中的key
  tags: ["nginx-access"]           #指定tag,用于分类

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  tags: ["nginx-error"]

output.elasticsearch:
  hosts: ["192.168.1.101:9200","192.168.1.102:9200","192.168.1.103:9200"]
  indices:
    - index: "nginx-access-%{[agent.version]}-%{+yyy.MM.dd}"
      when.contains:
      tags: "nginx-access"   #如果记志中有access的tag,就记录到nginx-access的索引中
    - index: "nginx-error-%{[agent.version]}-%{+yyy.MM.dd}"
      when.contains:
      tags: "nginx-error"   #如果记志中有error的tag,就记录到nginx-error的索引中

setup.ilm.enabled: false #关闭索引生命周期ilm功能，默认开启时索引名称只能为filebeat-*
setup.template.name: "nginx" #定义模板名称,要自定义索引名称,必须指定此项,否则无法启动
setup.template.pattern: "nginx-*" #定义模板的匹配索引名称,要自定义索引名称,必须指定此项,否则无法启动

root@web01:~# systemctl restart filebeat.service 

3.3、插件查看索引

3.4、在 Kibana 验证日志数据

创建索引模式