experience sharing for real-time log indexing in Elasticsearch

Elasticsearch automatically makes type mapping with log data, but in the most of cases, the mapping data is not correct, so we should change them. this is my experience of that.

1) run elasticsearch

2) run logstash with configration

elk:/data1/elasticsearch/logstash-1.4.0> vi logstash-teton_runtime.conf&

input{
    file{
        codec => json
        path => ["/data1/elasticsearch/temp/*.log"]
        start_position => "end"
    }
}

output{
    elasticsearch {
        cluster => "locketCast"
        node_name => "logstash-teton_runtime"
        host => "xxx.xxx.xxx.xxx"
        index => "teton_runtime"
    }
}

elk:/data1/elasticsearch/logstash-1.4.0> bin/logstash -f mixpanel.conf

3) make log file in /data1/elasticsearch/temp/*.log

{"androidId":"91e8b0c3d89","facebookId":"11250","type":0,"numberOfMyFollowers":0,"externalBalance":0,"userType":"F","uniqueId":"dSTSbIg","id":326599,"profileImagePath":"http://graph.facebook.com/1125547890/picture?type=large&width=200&height=200","tos":true,"activeUser":1,"Action":"unlock","$gender":"unknown","event":"Impression_test","age":36,"name":"abcdef","Campaign_id":"tec_cus_9321","created_at":1400294301000,"gender":"unknown","udob":578600000,"active_timestamp":1404520448000,"longitude":"0.00000000","$age":"04/30/1970","os":"4.1.2","user_group":"2","status":"YET_TO_REQUEST","zipcode":"12345","cash_amount":0,"manu":"LGE","email":"marshall.s@yahoo.com","appVersion":"1.5.3","dob":"04/30/1988","latitude":"0.00000000"}

=> issue 1) : "type":0 => "type" can not be available, it should be changed.
error logs :
{"acknowledged":true}mac:/data1/elasticsearch/logstash-1.4.0> Failed to flush outgoing items {:outgoing_count=>1, :exception=>#<NameError: no method 'type' for arguments (org.jruby.RubyFixnum) on Java::OrgElasticsearchActionIndex::IndexRequest>, :backtrace=>["/data1/elasticsearch/logstash-1.4.0/lib/logstash/outputs/elasticsearch/protocol.rb:225:in
~~~
<NameError: no method 'type' for arguments (org.jruby.RubyFixnum) on Java::OrgElasticsearchActionIndex::IndexRequest>, :backtrace=>["/data1/elasticsearch/logstash-1.4.0/lib/logstash/outputs/elasticsearch/protocol.rb:225:in `build_request'", "/data1/elasticsearch/logstash-1.4.0/lib/logstash/outputs/elasticsearch/protocol.rb:205:in `bulk'", "org/jruby/RubyArray.java:1613:in `each'", "/data1/elasticsearch/logstash-1.4.0/lib/logstash/outputs/elasticsearch/protocol.rb:204:in `bulk'",

=> issue 2) : all of entries are not indexed, they're saved as a hole of string entry "message"

Solution )
step 1) check out the current type mapping

elk:/data1/elasticsearch> curl -XGET 'http://localhost:9200/teton_runtime/logs/_mapping'
{"teton_runtime":{"mappings":{"logs":{"properties":{"@timestamp":{"type":"date","format":"dateOptionalTime"},"@version":{"type":"string"},"androidId":{"type":"string"},"externalBalance":{"type":"long"},"facebookId":{"type":"string"},"host":{"type":"string"},"id":{"type":"long"},"message":{"type":"string"},"numberOfMyFollowers":{"type":"long"},"path":{"type":"string"},"profileImagePath":{"type":"string","store":true},"uniqueId":{"type":"string"},"userType":{"type":"string"}}}}}}mac:/data1/elasticsearch/logstash-1.4.0>

step 2) make new type mapping considering with current type mapping
curl -XPUT 'http://localhost:9200/teton_runtime/logs/_mapping' -d '
{
"logs" : {
        "properties" : {
"facebookId" : {"type" : "string"},
"androidId" : {"type" : "string"},
"numberOfMyFollowers" : {"type" : "long"},
"externalBalance" : {"type" : "long"},
"userType" : {"type" : "string"},
"uniqueId" : {"type" : "string"},
"id" : {"type" : "long"},
"profileImagePath" : {"type" : "string", "store" : true},
"tos" : {"type" : "boolean"},
"activeUser" : {"type" : "string"},
"Action" : {"type" : "string"},
"$gender" : {"type" : "string"},
"event" : {"type" : "string"},
"age" : {"type" : "integer"},
"name" : {"type" : "string"},
"Campaign_id" : {"type" : "string"},
"created_at" : {"type" : "date"},
"gender" : {"type" : "string"},
"udob" : {"type" : "date"},
"active_timestamp" : {"type" : "string"},
"longitude" : {"type" : "string"},
"$age" : {"type" : "date"},
"os" : {"type" : "string"},
"user_group" : {"type" : "string"},
"status" : {"type" : "string"},
"zipcode" : {"type" : "string"},
"cash_amount" : {"type" : "long"},
"manu" : {"type" : "string"},
"email" : {"type" : "string"},
"appVersion" : {"type" : "string"},
"dob" : {"type" : "date"},
"latitude" : {"type" : "string"}
        }
    }
}
'

step 3) delete current index
curl -XDELETE 'http://localhost:9200/teton_runtime'

step 4) append log file in /data1/elasticsearch/temp/*.log with last empty line

ex)
curl -XGET 'http://localhost:9200/impression/logs/_mapping'

curl -XDELETE 'http://localhost:9200/impression/logs/_mapping'

curl -XPUT 'http://localhost:9200/impression/logs/_mapping' -d '
{
"logs" : {
        "properties" : {
"carrier" : {"type" : "string", "index": "not_analyzed"}
        }
    }
}
'

* http://www.elasticsearch.org/blog/changing-mapping-with-zero-downtime/