Skip to main content

Setup for Elasticsearch


0. preparation
 - add jars to lib folder (/tz.search/lib)
 elasticsearch-1.0.0.jar
 lucene-analyzers-common-4.6.1.jar
 lucene-codecs-4.6.1.jar
 lucene-core-4.6.1.jar
 lucene-grouping-4.6.1.jar
 lucene-highlighter-4.6.1.jar
 lucene-join-4.6.1.jar
 lucene-memory-4.6.1.jar
 lucene-misc-4.6.1.jar
 lucene-queries-4.6.1.jar
 lucene-queryparser-4.6.1.jar
 lucene-sandbox-4.6.1.jar
 lucene-spatial-4.6.1.jar
 lucene-suggest-4.6.1.jar
 - add jars to classpath
 <classpathentry kind="lib" path="lib/elasticsearch-1.0.0.jar"/>
 <classpathentry kind="lib" path="lib/lucene-core-4.6.1.jar"/>
 <classpathentry kind="lib" path="lib/lucene-analyzers-common-4.6.1.jar"/>
 <classpathentry kind="lib" path="lib/lucene-codecs-4.6.1.jar"/>
 <classpathentry kind="lib" path="lib/lucene-queries-4.6.1.jar"/>
 <classpathentry kind="lib" path="lib/lucene-memory-4.6.1.jar"/>
 <classpathentry kind="lib" path="lib/lucene-highlighter-4.6.1.jar"/>
 <classpathentry kind="lib" path="lib/lucene-queryparser-4.6.1.jar"/>
 <classpathentry kind="lib" path="lib/lucene-sandbox-4.6.1.jar"/>
 <classpathentry kind="lib" path="lib/lucene-suggest-4.6.1.jar"/>
 <classpathentry kind="lib" path="lib/lucene-misc-4.6.1.jar"/>
 <classpathentry kind="lib" path="lib/lucene-join-4.6.1.jar"/>
 <classpathentry kind="lib" path="lib/lucene-grouping-4.6.1.jar"/>
 <classpathentry kind="lib" path="lib/lucene-spatial-4.6.1.jar"/>

 - install elasticsearch managing console
 ./elasticsearch/bin/plugin -install lmenezes/elasticsearch-kopf

1. modify configuration file and sources
 - modify elasticsearch nodes' configuration (elasticsearch.yml)
 cluster.name: "locketCast"
 - modify tz.search configuration (/tz.search/conf/application.conf)
 # elasticsearch
 elasticsearch.clusterName = locketCast
 elasticsearch.nodes = "localhost:9300,localhost:9301,localhost:9302"

 - add a service (/tz.search/app/services/ElasticsearchService.java)

 - add the lines under updateLogs method in a controller (/tz.search/app/controllers/UserApi.java)
 
   public static Result updateLogs(Long user_id, String androidId,
   ~~~~
 // put the log into elasticsearch
 ElasticsearchService.addIndexing("updatelogs", new SimpleDateFormat("yyyyMMdd").format(createdAt), logEvent.id, propertiesString);
 } catch (JSONException e) {
 e.printStackTrace();
 } catch (Exception e) {
 e.printStackTrace();
 }
 return ok();
 }

2. run servers

 /data1/elasticsearch/node1> ./start.sh
 /data1/elasticsearch/node2> ./start.sh
 /data1/elasticsearch/node3> ./start.sh

 /Users/mac/git/tz.search> play start

3. test source
 http://localhost:9000/api/user/logupdate?user_id=1&androidId=1&email=doohee323@gmail.com&eventName=1&propertiesString={"property":"Action", "value":"unlock"}

4. check the result
 http://localhost:9200/_plugin/kopf/#/cluster
 http://localhost:9200/_plugin/kopf/#/rest
 -> http://localhost:9200/_search?q=property:Action
 -> http://localhost:9200/updatelogs/20140423/_search
 -> http://localhost:9200/updatelogs/20140423/_count?q=property:Action

Comments

Post a Comment

Popular posts from this blog

Amazon RDS Blue/Green Deployments

In order to avoid some errors I experienced when proceeding as described in the official documentation, I describe what I did in order. 1) Modify parameters of source_database * error: Blue Green Deployments requires cluster parameter group has binlog enabled. RDS Parameter groups: source-params-group binlog_format => MIXED mysql> show global variables like 'binlog_format'; 2) Insert a row after rebooting the source database, to avoid this error. * error: Correct the replication errors and then switch over. Read Replica Replication Error - IOError: 1236, reason: Got fatal error 1236 from master when reading data from binary log: 'Could not find first log file name in binary log index file' => To Fix: You need to change the data in the source database. INSERT INTO dummy_table ( `favorite_id` , `favorite_order` , `user_id` , `board_id` ) VALUES ('100001', '1', '11111', '11111'); 3) Modify the param...
Post a Comment
Read more

Fluentd for mysql in AWS

(0) preparation ulimit -n If your console shows 1024, it is insufficient. Please add following lines to your /etc/security/limits.conf file and reboot your machine. root soft nofile 65536 root hard nofile 65536 (1) install Fluentd // “Ubuntu 12.04 LTS / Precise” curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-precise.sh | sh /etc/init.d/td-agent start/stop/restart/status // test curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test /etc/init.d/td-agent stop chown: changing ownership of `/var/run/td-agent/td-agent.pid': Operation not permitted chown: changing ownership of `/var/run/td-agent': Operation not permitted  * Stopping td-agent td-agent                                                                                   ...
Post a Comment
Read more

Install CoreOs on linode without VM

Install CoreOs on linode without VM 1. Add a Linode 2. Create a new Disk   CoreOS 3. Rescue > Reboot into Rescue Mode 4. Remote Access   Launch Lish Console 5. make an install script cat <<'EOF1' > install.sh # add needed package sudo apt-get update sudo apt-get install -y curl wget whois sudo apt-get install -y ca-certificates #sudo apt-get install gawk -y # get discovery url discoveryUrl=`curl https://discovery.etcd.io/new` # write cloud-config.yml cat <<EOF2 > cloud-config.yml #cloud-config users:   - name: core     groups:       - sudo       - docker coreos:   etcd:     name: node01     discovery: $discoveryUrl hostname: node01 EOF2 # get the coreos installation script #wget https://raw.github.com/coreos/init/master/bin/coreos-install wget https://raw.githubusercontent.com/coreos/init/master/bin/coreos-install # run installation chmod 75...
31 comments
Read more