之前通过logstash的插件做过mysql与ElasticSearch的数据同步.个人认为在logstash可以大致分为2个阶段.即Input(输入),Output(输出).当然logstash还有许多的插件如Filter数据转换.也可以说是有3个阶段.在使用过程中我觉得有点像新的编程语言.

Event:是logstash中内部数据形式.当原始数据在Input中会通过codec被转为Event,而到了Output到时候Event会被转换成需要格式的数据.

小技巧

  可以发现在启动logstash的时候每改一次配置都需要重启.而重启又非常的慢.可以使用热加载的方式来启动.创建一个启动测试的配置start.conf

input{
  http{
    port => 你的端口号
  }
  stdin{
    codec => line
  }
}
output{
  stdout{
    codec => rubydebug
  }
}

使用postman工具

  在终端显示

{
    "@timestamp" => 2019-07-25T10:57:52.260Z,
       "headers" => {
        "accept_encoding" => "gzip, deflate",
             "connection" => "keep-alive",
        "http_user_agent" => "PostmanRuntime/7.6.0",
           "http_version" => "HTTP/1.1",
         "content_length" => "13",
              "http_host" => "127.0.0.1:8401",
           "content_type" => "text/plain",
           "request_path" => "/",
          "cache_control" => "no-cache",
            "http_accept" => "*/*",
          "postman_token" => "3b86cb4c-8618-4274-985e-6eeb0a6f6965",
         "request_method" => "POST"
    },
       "message" => "hello,qvbilam",
      "@version" => "1",
          "host" => "127.0.0.1"
}

Input

  数据的输入,可以通过配置来定义数据的类型

数据类型配置

数据类型演示

# 首先为了方便讲,logstash加入全局
vim ~/.zshrc
# 添加
alias logstash="/usr/local/Cellar/logstash/bin/logstash"
# 保存,立即生效
source ~/.zshrc
# 创建input配置文件演示
vim input.conf
# 添加如下
input {
    stdin {
            # plain是什么都不做的转码处理
        codec => "plain"
        # 自定义
        tags => ["isArray"]
        # 自动移
        type => "isString"
        # 添加name字段
        add_field => {"name" => "qvbilam"}
    }
}
output{
    stdout{
            # 按照json的格式输出
        codec => "json"
    }
}
# 保存,执行测试
echo 'qvbilam' | logstash -f input.conf
qvbilam
# 显示结果
Successfully started Logstash API endpoint {:port=>9600}
{"name":"qvbilam","@version":"1","host":"erhuadamowangdeMacBook-Pro.local","@timestamp":"2019-07-10T08:35:30.399Z","tags":["isArray"],"message":"qvbilam","type":"isString"}

  为了方便看结果.我讲返回内容转换成json格式

{
    "name":"qvbilam",
    "@version":"1",
    "host":"erhuadamowangdeMacBook-Pro.local",
    "@timestamp":"2019-07-10T08:35:30.399Z",
    "tags":[
        "isArray"
    ],
    "message":"qvbilam",
    "type":"isString"
}

File

  input中的插件,是基于ruby语言开发的,可以指定目录.假如监听的目录中出现了新的日志文件,也是可以进行读取的.对于监听的文件会时刻坚持是否有新的内容产生.对于意外重启logstash.对于之前的日志监听也是不会重复读取的(默认配置).并且如果文件重命名,也不会影响到logstash对文件的读取.

File配置

Glob匹配

测试

# 创建测试文件
mkdir log
cd log
vim 1.log
# 添加如下
hello
qvbilam
# 保存
cd ../
vim plugin.conf
# 添加
input{
    file{
        path => "/Users/qvbilam/Sites/test/logstash/log/*.log"
        type => "test"
        start_position => "end"
        stat_interval => 1
    }
}

output{
    stdout{
        codec => rubydebug 
    }
}
# 保存,执行
logstash -f plugin.conf
# 返回结果
{
       "message" => "hello",
      "@version" => "1",
          "host" => "erhuadamowangdeMacBook-Pro.local",
          "path" => "/Users/qvbilam/Sites/test/logstash/log/1.log",
    "@timestamp" => 2019-07-10T13:25:58.642Z
}
{
       "message" => "qvbilam",
      "@version" => "1",
          "host" => "erhuadamowangdeMacBook-Pro.local",
          "path" => "/Users/qvbilam/Sites/test/logstash/log/1.log",
    "@timestamp" => 2019-07-10T13:25:58.673Z
}

重启测试

# 发现并没有读取之前的文件,追加内容到log
echo '\nangel' >> 1.log
cat 1.log
# log文件
hello
qvbilam

angel
# 查看logstash结果
{
          "path" => "/Users/qvbilam/Sites/test/logstash/log/1.log",
    "@timestamp" => 2019-07-10T13:30:35.178Z,
          "type" => "test",
      "@version" => "1",
          "host" => "erhuadamowangdeMacBook-Pro.local",
       "message" => ""
}
{
          "path" => "/Users/qvbilam/Sites/test/logstash/log/1.log",
    "@timestamp" => 2019-07-10T13:30:35.201Z,
          "type" => "test",
      "@version" => "1",
          "host" => "erhuadamowangdeMacBook-Pro.local",
       "message" => "angel"
}

Output

  将输入的数据进行处理后输出.例个小demo吧:)

输出到Email

  常用参数说明

  实例

input {
    stdin {
        codec => "json"
    }
}
output{
    stdout{
        codec => "json"
    }
email {
        to => 'xxxxxxxx@qq.com'  
        from => 'qvbilam@163.com'
        subject => 'Alert - %{title}'
        body => "Tags: %{tags}\\nI\\Content:\\n%{message}"
        address => 'smtp.163.com'
        username => 'qvbilam@163.com'
        password => '******'
        port => 25
    }
}

# 启动服务键入
{"title":"Hello","tags":"今天星期二","message":"还疼吗?小老弟."}
# 查看已发送邮件
Tags: 今天星期二\
I\\Content:\
还疼吗?小老弟.

  其实有好多,例如输出到elasticsearch的在后面的文章有介绍,对于输入输出的实例打算单独列出来.就不在这篇文章做过多的演示了.