Skip to main content

experience sharing for real-time log indexing in Elasticsearch

Elasticsearch automatically makes type mapping with log data, but in the most of cases, the mapping data is not correct, so we should change them. this is my experience of that.

1) run elasticsearch

2) run logstash with configration

elk:/data1/elasticsearch/logstash-1.4.0> vi logstash-teton_runtime.conf&

input{
    file{
        codec => json
        path => ["/data1/elasticsearch/temp/*.log"]
        start_position => "end"
    }
}

output{
    elasticsearch {
        cluster => "locketCast"
        node_name => "logstash-teton_runtime"
        host => "xxx.xxx.xxx.xxx"
        index => "teton_runtime"
    }
}

elk:/data1/elasticsearch/logstash-1.4.0> bin/logstash -f mixpanel.conf

3) make log file in /data1/elasticsearch/temp/*.log

{"androidId":"91e8b0c3d89","facebookId":"11250","type":0,"numberOfMyFollowers":0,"externalBalance":0,"userType":"F","uniqueId":"dSTSbIg","id":326599,"profileImagePath":"http://graph.facebook.com/1125547890/picture?type=large&width=200&height=200","tos":true,"activeUser":1,"Action":"unlock","$gender":"unknown","event":"Impression_test","age":36,"name":"abcdef","Campaign_id":"tec_cus_9321","created_at":1400294301000,"gender":"unknown","udob":578600000,"active_timestamp":1404520448000,"longitude":"0.00000000","$age":"04/30/1970","os":"4.1.2","user_group":"2","status":"YET_TO_REQUEST","zipcode":"12345","cash_amount":0,"manu":"LGE","email":"marshall.s@yahoo.com","appVersion":"1.5.3","dob":"04/30/1988","latitude":"0.00000000"}

=> issue 1) : "type":0 => "type" can not be available, it should be changed.
error logs :
{"acknowledged":true}mac:/data1/elasticsearch/logstash-1.4.0> Failed to flush outgoing items {:outgoing_count=>1, :exception=>#<NameError: no method 'type' for arguments (org.jruby.RubyFixnum) on Java::OrgElasticsearchActionIndex::IndexRequest>, :backtrace=>["/data1/elasticsearch/logstash-1.4.0/lib/logstash/outputs/elasticsearch/protocol.rb:225:in
~~~
<NameError: no method 'type' for arguments (org.jruby.RubyFixnum) on Java::OrgElasticsearchActionIndex::IndexRequest>, :backtrace=>["/data1/elasticsearch/logstash-1.4.0/lib/logstash/outputs/elasticsearch/protocol.rb:225:in `build_request'", "/data1/elasticsearch/logstash-1.4.0/lib/logstash/outputs/elasticsearch/protocol.rb:205:in `bulk'", "org/jruby/RubyArray.java:1613:in `each'", "/data1/elasticsearch/logstash-1.4.0/lib/logstash/outputs/elasticsearch/protocol.rb:204:in `bulk'",

=> issue 2) : all of entries are not indexed, they're saved as a hole of string entry "message"

Solution )
step 1) check out the current type mapping

elk:/data1/elasticsearch> curl -XGET 'http://localhost:9200/teton_runtime/logs/_mapping'
{"teton_runtime":{"mappings":{"logs":{"properties":{"@timestamp":{"type":"date","format":"dateOptionalTime"},"@version":{"type":"string"},"androidId":{"type":"string"},"externalBalance":{"type":"long"},"facebookId":{"type":"string"},"host":{"type":"string"},"id":{"type":"long"},"message":{"type":"string"},"numberOfMyFollowers":{"type":"long"},"path":{"type":"string"},"profileImagePath":{"type":"string","store":true},"uniqueId":{"type":"string"},"userType":{"type":"string"}}}}}}mac:/data1/elasticsearch/logstash-1.4.0>

step 2) make new type mapping considering with current type mapping
curl -XPUT 'http://localhost:9200/teton_runtime/logs/_mapping' -d '
{
"logs" : {
        "properties" : {
"facebookId" : {"type" : "string"},
"androidId" : {"type" : "string"},
"numberOfMyFollowers" : {"type" : "long"},
"externalBalance" : {"type" : "long"},
"userType" : {"type" : "string"},
"uniqueId" : {"type" : "string"},
"id" : {"type" : "long"},
"profileImagePath" : {"type" : "string", "store" : true},
"tos" : {"type" : "boolean"},
"activeUser" : {"type" : "string"},
"Action" : {"type" : "string"},
"$gender" : {"type" : "string"},
"event" : {"type" : "string"},
"age" : {"type" : "integer"},
"name" : {"type" : "string"},
"Campaign_id" : {"type" : "string"},
"created_at" : {"type" : "date"},
"gender" : {"type" : "string"},
"udob" : {"type" : "date"},
"active_timestamp" : {"type" : "string"},
"longitude" : {"type" : "string"},
"$age" : {"type" : "date"},
"os" : {"type" : "string"},
"user_group" : {"type" : "string"},
"status" : {"type" : "string"},
"zipcode" : {"type" : "string"},
"cash_amount" : {"type" : "long"},
"manu" : {"type" : "string"},
"email" : {"type" : "string"},
"appVersion" : {"type" : "string"},
"dob" : {"type" : "date"},
"latitude" : {"type" : "string"}
        }
    }
}
'

step 3) delete current index
curl -XDELETE 'http://localhost:9200/teton_runtime'

step 4) append log file in /data1/elasticsearch/temp/*.log with last empty line

ex)
curl -XGET 'http://localhost:9200/impression/logs/_mapping'

curl -XDELETE 'http://localhost:9200/impression/logs/_mapping'

curl -XPUT 'http://localhost:9200/impression/logs/_mapping' -d '
{
"logs" : {
        "properties" : {
"carrier" : {"type" : "string", "index": "not_analyzed"}
        }
    }
}
'

* http://www.elasticsearch.org/blog/changing-mapping-with-zero-downtime/


Comments

Post a Comment

Popular posts from this blog

Amazon RDS Blue/Green Deployments

In order to avoid some errors I experienced when proceeding as described in the official documentation, I describe what I did in order. 1) Modify parameters of source_database * error: Blue Green Deployments requires cluster parameter group has binlog enabled. RDS Parameter groups: source-params-group binlog_format => MIXED mysql> show global variables like 'binlog_format'; 2) Insert a row after rebooting the source database, to avoid this error. * error: Correct the replication errors and then switch over. Read Replica Replication Error - IOError: 1236, reason: Got fatal error 1236 from master when reading data from binary log: 'Could not find first log file name in binary log index file' => To Fix: You need to change the data in the source database. INSERT INTO dummy_table ( `favorite_id` , `favorite_order` , `user_id` , `board_id` ) VALUES ('100001', '1', '11111', '11111'); 3) Modify the param...
Post a Comment
Read more

Fluentd for mysql in AWS

(0) preparation ulimit -n If your console shows 1024, it is insufficient. Please add following lines to your /etc/security/limits.conf file and reboot your machine. root soft nofile 65536 root hard nofile 65536 (1) install Fluentd // “Ubuntu 12.04 LTS / Precise” curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-precise.sh | sh /etc/init.d/td-agent start/stop/restart/status // test curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test /etc/init.d/td-agent stop chown: changing ownership of `/var/run/td-agent/td-agent.pid': Operation not permitted chown: changing ownership of `/var/run/td-agent': Operation not permitted  * Stopping td-agent td-agent                                                                                   ...
Post a Comment
Read more

실리콘밸리 구직 체험기2_201505 - 5. 기억나는 onsite 인터뷰

Groupon: 주워진 restful api를 이용해서 목록 UI를 구현하기, 갑작스런 javascript 문제로 버벅거렸고 javascript 안써본 사람이런 평가를 받았습니다. Frankly: log 분석을 위한 아키텍쳐를 구성해보라고 했고 apache tajo를 이용한 해법을 설명했더니 외부 솔루션 의존적이라 자체 구현 능력이 있는지 모르겠다는 평가를 받았습니다. 로그 분석은 매 프로젝트에서 언제나 하는 것 아닌가? 새로운 해법을 제시했더니 사족이 된 건지 모르겠습니다. Samsung Lab: 10개의 계단을 올라가는 경우의 수를 구하는 프로그램을 작성하라고 해서 node를 이용해서 이것 저것 지도 했는데 그냥 피보나치 수열이라고 했습니다. Phone Veeva: Object가 어떤 method들을 갖고 있는지, 자바 스크립트 inheritance 구현을 어떻게 하는지, generic 타입을 쓰는 이유가 뭔지 어떻게 쓰는지 이것저것 놓친 것이 너무 많아서 챙피했습니다. Resilient: queue 2개로 stack을 구현하라고 했는데 이렇게 작은 회사에서도 코드인터뷰는 쉽지 않았습니다. 한번은 full-stack senior 엔지니어 면접을 보면서 다음과 같은 질문들을 받았습니다. back-end와 front-end 전체 분야를 다 깊이있게 물어볼 경우 탈탈 털리는 느낌이였습니다. 특히 javascript 문제에 대한 준비가 제대로 되어 있지 못했습니다. - java, framework 일반적인 질문 - was, linux 일반적인 질문 - 멀티 쓰레드 프로그래밍 (쓰레드간 순서 통제 등) - architecture, deploy, development 방법 - javascript concept / coding (상속, closure) - SQL 쿼리 문제 - 출력된 자바 코드를 보여주고 JAVA 코드 수정하기 ㅇ full-stack 엔지니어 full-stack은 분명 ...
Post a Comment
Read more