Skip to main content

Storm-Esper From Kafka / Zmq / Graphite


This is a Storm-Esper example on vagrant.
(From Kafka / Zmq / Graphite)

https://github.com/doohee323/tzstorm

=====================================

1. build
You can run it on vagrant like this,
./build_deb.sh

and you can see the status of storm at http://192.168.82.157:8080.

2. examples
There are some examples.

1) Simple storm example
/tzstorm/src/main/java/example/tzstorm/TestTopology.java

2) storm + esper example
/tzstorm/src/main/java/example2/tzstorm/TestTopology2.java

3) multiple bolts example
/tzstorm/src/main/java/example3/tzstorm/TestTopology3.java

4) kafka - storm(multiple bolts) example
/tzstorm/src/main/java/example4/tzstorm/TestTopology4.java

5) storm + trident(unique data) + esper example
/tzstorm/src/main/java/example5/tzstorm/TestTopology5.java

6) zmq PUB/SUB + storm + trident(unique data) + esper example

7) zmq REP/REQ + storm + trident(unique data) + esper example
/tzstorm/src/main/java/example7/tzstorm/TestTopology7.java
-classpath .:/home/vagrant/tzstorm/jzmq-3.1.0.jar -Djava.library.path=/usr/local/lib -Xcheck:jni

/tzstorm/src/main/java/example7/tzstorm/zmq/ZMQClient.java
-classpath .:/home/vagrant/tzstorm/jzmq-3.1.0.jar -Djava.library.path=/usr/local/lib -Xcheck:jni

8) graphite + storm + trident(unique data) + esper example
/tzstorm/src/main/java/example8/tzstorm/TestTopology8.java

3. set VM configuration for vagrant
You can define the VM server which you want to run.
for example, if you define the VMs for example7,
- in /tzstorm/setup.conf
{
 'ip'=> {
   'tzstorm7' => "192.168.82.157"
 }
}
- in /tzstorm/setup.rc
cfg_ip_tzstorm7="192.168.82.157"

Then, vagrant will use this shell, /tzstorm/scripts/tzstorm7.sh for build VM.

4. set VM configuration for vagrant
It includes some features, (/tzstorm/scripts/tzstorm.sh)

5. etc
When you run topology, you can use VM arguments.

- in local environment like eclipse, use "-DrunType=local".
- in storm environment, use "-DrunType=storm".
- in kafka-integrate environment, use "-DrunType=kafka".

You can change logback configration file for each examples like this.
-Dlogback.configurationFile=logback6.xml

cf) zmq install on mac

1) libzmq
git clone git://github.com/zeromq/libzmq.git
cd libzmq
./autogen.sh
./configure --prefix=/usr/share/pkgconfig --without-libsodium
make
sudo make install

2) jzmq
git clone https://github.com/zeromq/jzmq.git
cd jzmq
./autogen.sh
./configure --with-zeromq=/home/vagrant/libzmq
(./configure --with-zeromq=/Users/mac/tmp/libzmq)
make
sudo make install

3) copy libraries
sudo cp -Rf /usr/share/pkgconfig/lib/* /usr/local/lib
export LD_LIBRARY_PATH=/usr/local/lib



Comments

Post a Comment

Popular posts from this blog

DevOps JD's required skills from LinkedIn

From some of DevOps JD on linkedIn, I realised that DeveOps should be the leader of the organization. https://docs.google.com/spreadsheets/d/1P520nH0pYcAdN0rJcnMQqsgu9cV9GdknztJ92J8l7-s/pubhtml DevOps' Required Skills From LinkedIn on 8/30/16: DevOps should be the leader of the company! Yahoo Netflix Samsung Salesforce Fortinet SUM OS admin UNIX systems Unix platforms Linux administrator Linux VMs Docker VMs VMware, OpenStack, Hyper-V Openstack, KVM, VMWare Version control version control systems Git, SVN Cloud Amazon AWS AWS AWS, Azure DB MySql Oracle, MySQL, NoSQL Mysql administration and strong command of SQL MySQL RabbitMQ MySql, MongoDB, Redis, Oracle, ProgreSQL N/W TCP/IP networking, DNS, HTTP NAS Understanding of network stack, network tuning, subnet/VLANs. HAProxy, DNS, IPTable Script Lang Shell, Perl, Python, Ruby, PHP bash Python, Bash/tcsh a scripting language: Perl, Python and Unix Shell preferred Python, Perl, Ruby Python, Ruby, Shell, PHP Web LAMP stack
Post a Comment
Read more

Ubuntu GUI with VNC on Xenserver

Xenserver 에서 Ubuntu GUI 를 쓰기 위해서는 VNC 가 답인 듯... Installing Ubuntu Gnome GUI on Ubuntu Server 12.10 with VNC Update Repositories # apt-get update Install gnome and vnc: # apt-get install gnome-core vnc4server Start VNC Server: # vncserver (You’ll then be prompted to create and verify a new VNC connect password) Kill the currently running VNC Session: # vncserver -kill :1 Edit VNC startup config file: # vim .vnc/xstartup Uncomment the following line: unset SESSION_MANAGER Add the following line: gnome-session --session=gnome-classic & Comment Out the following two lines: x-terminal-emulator -geometry 1280x1024+10+10 -ls -title "$VNCDESKTOP Desktop" & x-window-manager & End result should look like: #!/bin/sh # Uncomment the following two lines for normal desktop: unset SESSION_MANAGER # exec /etc/X11/xinit/xinitrc gnome-session --session=gnome-classic & [ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup [ -r $HOME/
Post a Comment
Read more

Install CoreOs on linode without VM

Install CoreOs on linode without VM 1. Add a Linode 2. Create a new Disk   CoreOS 3. Rescue > Reboot into Rescue Mode 4. Remote Access   Launch Lish Console 5. make an install script cat <<'EOF1' > install.sh # add needed package sudo apt-get update sudo apt-get install -y curl wget whois sudo apt-get install -y ca-certificates #sudo apt-get install gawk -y # get discovery url discoveryUrl=`curl https://discovery.etcd.io/new` # write cloud-config.yml cat <<EOF2 > cloud-config.yml #cloud-config users:   - name: core     groups:       - sudo       - docker coreos:   etcd:     name: node01     discovery: $discoveryUrl hostname: node01 EOF2 # get the coreos installation script #wget https://raw.github.com/coreos/init/master/bin/coreos-install wget https://raw.githubusercontent.com/coreos/init/master/bin/coreos-install # run installation chmod 755 coreos-install sudo ./coreos-install \       -d /dev/sda \       -
31 comments
Read more