logstash Nginx日志解析

OpenCDN中tengine access日志格式

log_format  access  '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" '   '"$http_user_agent" "$http_x_forwarded_for" ' '"$upstream_cache_status"' &@&$host ;

logstash日志解析配置

input {
 udp {
     type => "linux-syslog"
     port => 515
  }
}

filter {
grok {
        type => "linux-syslog"
        pattern => "%{IPORHOST:source_ip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent} %{QS:x_forword} %{QS:upstream_cache_status}&@&(%{HOST:domain}|-)"
}
}

output {
   stdout {
        debug => true
        debug_format => json
  }
   elasticsearch {
      embedded => true
   }
}

下面是debug模式出来的json结果

@timestamp: "2013-10-22 14:22:38 UTC"
@version: "1"
message: "Oct 22 22:21:26 www14443ui.sakura.ne.jp OpenCDN Beta[1214]: 114.179.18.37 - - [22/Oct/2013:22:21:26 +0800] "GET /data/attachment/common/f4/common_94_medal.gif HTTP/1.1" 200 11552 "http://www.firefoxbug.com/thread-58598-1-1.html" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1" "-" "HIT"&@&www.firefoxbug.net"
body_bytes_sent: "11552"
domain: "www.firefoxbug.net"
host: "133.242.187.217"
http_referer: ""http://www.firefoxbug.com/thread-58598-1-1.html""
http_user_agent: ""Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1""
remote_user: "-"
request: ""GET /data/attachment/common/f4/common_94_medal.gif HTTP/1.1""
source_ip: "114.179.18.37"
status: "200"
tags: Array[1]
time_local: "22/Oct/2013:22:21:26 +0800"
type: "linux-syslog"
upstream_cache_status: ""HIT""
x_forword: ""-""

下面看截图

转载请注明：爱开源 » logstash Nginx日志解析

转载请注明：爱开源 » logstash Nginx日志解析