Skip to main content

Storm-Esper From Kafka / Zmq / Graphite


This is a Storm-Esper example on vagrant.
(From Kafka / Zmq / Graphite)

https://github.com/doohee323/tzstorm

=====================================

1. build
You can run it on vagrant like this,
./build_deb.sh

and you can see the status of storm at http://192.168.82.157:8080.

2. examples
There are some examples.

1) Simple storm example
/tzstorm/src/main/java/example/tzstorm/TestTopology.java

2) storm + esper example
/tzstorm/src/main/java/example2/tzstorm/TestTopology2.java

3) multiple bolts example
/tzstorm/src/main/java/example3/tzstorm/TestTopology3.java

4) kafka - storm(multiple bolts) example
/tzstorm/src/main/java/example4/tzstorm/TestTopology4.java

5) storm + trident(unique data) + esper example
/tzstorm/src/main/java/example5/tzstorm/TestTopology5.java

6) zmq PUB/SUB + storm + trident(unique data) + esper example

7) zmq REP/REQ + storm + trident(unique data) + esper example
/tzstorm/src/main/java/example7/tzstorm/TestTopology7.java
-classpath .:/home/vagrant/tzstorm/jzmq-3.1.0.jar -Djava.library.path=/usr/local/lib -Xcheck:jni

/tzstorm/src/main/java/example7/tzstorm/zmq/ZMQClient.java
-classpath .:/home/vagrant/tzstorm/jzmq-3.1.0.jar -Djava.library.path=/usr/local/lib -Xcheck:jni

8) graphite + storm + trident(unique data) + esper example
/tzstorm/src/main/java/example8/tzstorm/TestTopology8.java

3. set VM configuration for vagrant
You can define the VM server which you want to run.
for example, if you define the VMs for example7,
- in /tzstorm/setup.conf
{
 'ip'=> {
   'tzstorm7' => "192.168.82.157"
 }
}
- in /tzstorm/setup.rc
cfg_ip_tzstorm7="192.168.82.157"

Then, vagrant will use this shell, /tzstorm/scripts/tzstorm7.sh for build VM.

4. set VM configuration for vagrant
It includes some features, (/tzstorm/scripts/tzstorm.sh)

5. etc
When you run topology, you can use VM arguments.

- in local environment like eclipse, use "-DrunType=local".
- in storm environment, use "-DrunType=storm".
- in kafka-integrate environment, use "-DrunType=kafka".

You can change logback configration file for each examples like this.
-Dlogback.configurationFile=logback6.xml

cf) zmq install on mac

1) libzmq
git clone git://github.com/zeromq/libzmq.git
cd libzmq
./autogen.sh
./configure --prefix=/usr/share/pkgconfig --without-libsodium
make
sudo make install

2) jzmq
git clone https://github.com/zeromq/jzmq.git
cd jzmq
./autogen.sh
./configure --with-zeromq=/home/vagrant/libzmq
(./configure --with-zeromq=/Users/mac/tmp/libzmq)
make
sudo make install

3) copy libraries
sudo cp -Rf /usr/share/pkgconfig/lib/* /usr/local/lib
export LD_LIBRARY_PATH=/usr/local/lib



Comments

Post a Comment

Popular posts from this blog

Amazon RDS Blue/Green Deployments

In order to avoid some errors I experienced when proceeding as described in the official documentation, I describe what I did in order. 1) Modify parameters of source_database * error: Blue Green Deployments requires cluster parameter group has binlog enabled. RDS Parameter groups: source-params-group binlog_format => MIXED mysql> show global variables like 'binlog_format'; 2) Insert a row after rebooting the source database, to avoid this error. * error: Correct the replication errors and then switch over. Read Replica Replication Error - IOError: 1236, reason: Got fatal error 1236 from master when reading data from binary log: 'Could not find first log file name in binary log index file' => To Fix: You need to change the data in the source database. INSERT INTO dummy_table ( `favorite_id` , `favorite_order` , `user_id` , `board_id` ) VALUES ('100001', '1', '11111', '11111'); 3) Modify the param...
Post a Comment
Read more

Fluentd for mysql in AWS

(0) preparation ulimit -n If your console shows 1024, it is insufficient. Please add following lines to your /etc/security/limits.conf file and reboot your machine. root soft nofile 65536 root hard nofile 65536 (1) install Fluentd // “Ubuntu 12.04 LTS / Precise” curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-precise.sh | sh /etc/init.d/td-agent start/stop/restart/status // test curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test /etc/init.d/td-agent stop chown: changing ownership of `/var/run/td-agent/td-agent.pid': Operation not permitted chown: changing ownership of `/var/run/td-agent': Operation not permitted  * Stopping td-agent td-agent                                                                                   ...
Post a Comment
Read more

Install CoreOs on linode without VM

Install CoreOs on linode without VM 1. Add a Linode 2. Create a new Disk   CoreOS 3. Rescue > Reboot into Rescue Mode 4. Remote Access   Launch Lish Console 5. make an install script cat <<'EOF1' > install.sh # add needed package sudo apt-get update sudo apt-get install -y curl wget whois sudo apt-get install -y ca-certificates #sudo apt-get install gawk -y # get discovery url discoveryUrl=`curl https://discovery.etcd.io/new` # write cloud-config.yml cat <<EOF2 > cloud-config.yml #cloud-config users:   - name: core     groups:       - sudo       - docker coreos:   etcd:     name: node01     discovery: $discoveryUrl hostname: node01 EOF2 # get the coreos installation script #wget https://raw.github.com/coreos/init/master/bin/coreos-install wget https://raw.githubusercontent.com/coreos/init/master/bin/coreos-install # run installation chmod 75...
31 comments
Read more