Skip to main content

Apach Tajo ( with Hadoop Library) 

We wanted to analyse huge user activity logs with some query conditions.
So we started with ELK(Elasticsearch, Logstash, Kibana). but Kibana's query results didn't meet our business needs. so I made some customised search queries and data types or something.
After some struggling, we decided to move to Apache Tajo for flexible query condition and easy for ETL.


And in order to save the cost for Tajo slave instances in AWS, I used this batch application for dynamic spot instances.






1 Tajo Master server (EC2 minimum size)


N Tajo slave servers (large size spot instances )








tz-tajo
=====================================

I made a batch application for cohort analysis with Apache TAJO. the feature is like this,

1. create 10 spot instances
2. check the instances status and Tajo workers' health
3. create a Tajo table on S3 for TETON log files (user, event)
4. create a result Tajo table for result saving on S3
5. execute cohort query and insert the result to the result Tajo table
6. download the result CSV file from S3
7. create a table in mysql
8. load the result CSV file to the table in mysql
9. terminate 10 spot instances

The only thing I have to fix is getting the result CSV file from S3.
Once we got a base for analysis automation tool, so we just need to change the Tajo query for the next requirement.





https://github.com/doohee323/tz-tajo











































Comments











Post a Comment






























Popular posts from this blog



















DevOps JD's required skills from LinkedIn




























 From some of DevOps JD on linkedIn, I realised that DeveOps should be the leader of the organization.   https://docs.google.com/spreadsheets/d/1P520nH0pYcAdN0rJcnMQqsgu9cV9GdknztJ92J8l7-s/pubhtml    DevOps' Required Skills From LinkedIn on 8/30/16: DevOps should be the leader of the company!  Yahoo Netflix Samsung Salesforce Fortinet SUM  OS admin UNIX systems Unix platforms Linux administrator Linux  VMs Docker VMs VMware, OpenStack, Hyper-V Openstack, KVM, VMWare  Version control version control systems Git, SVN  Cloud Amazon AWS AWS AWS, Azure  DB MySql Oracle, MySQL, NoSQL Mysql administration and strong command of SQL MySQL RabbitMQ MySql, MongoDB, Redis, Oracle, ProgreSQL  N/W TCP/IP networking, DNS, HTTP NAS Understanding of network stack, network tuning, subnet/VLANs. HAProxy, DNS, IPTable  Script Lang Shell, Perl, Python, Ruby, PHP bash Python, Bash/tcsh a scripting language: Perl, Python and Unix Shell preferred Python, Perl, Ruby Python, Ruby, Shell, PHP  Web LAMP stack




















Post a Comment









Read more




























Ubuntu GUI with VNC on Xenserver




























Xenserver 에서 Ubuntu GUI 를 쓰기 위해서는 VNC 가 답인 듯...   Installing Ubuntu Gnome GUI on Ubuntu Server 12.10 with VNC   Update Repositories   # apt-get update  Install gnome and vnc:   # apt-get install gnome-core vnc4server  Start VNC Server:   # vncserver  (You’ll then be prompted to create and verify a new VNC connect password)   Kill the currently running VNC Session:   # vncserver -kill :1  Edit VNC startup config file:   # vim .vnc/xstartup  Uncomment the following line:   unset SESSION_MANAGER  Add the following line:   gnome-session --session=gnome-classic &  Comment Out the following two lines:   x-terminal-emulator -geometry 1280x1024+10+10 -ls -title "$VNCDESKTOP Desktop" &  x-window-manager &  End result should look like:   #!/bin/sh   # Uncomment the following two lines for normal desktop:  unset SESSION_MANAGER  # exec /etc/X11/xinit/xinitrc  gnome-session --session=gnome-classic &   [ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup  [ -r $HOME/




















Post a Comment









Read more




























Install CoreOs on linode without VM




























Install CoreOs on linode without VM   1. Add a Linode  2. Create a new Disk    CoreOS  3. Rescue > Reboot into Rescue Mode   4. Remote Access    Launch Lish Console   5. make an install script   cat <<'EOF1' > install.sh   # add needed package  sudo apt-get update  sudo apt-get install -y curl wget whois  sudo apt-get install -y ca-certificates  #sudo apt-get install gawk -y   # get discovery url  discoveryUrl=`curl https://discovery.etcd.io/new`   # write cloud-config.yml  cat <<EOF2 > cloud-config.yml  #cloud-config   users:    - name: core      groups:        - sudo        - docker   coreos:    etcd:      name: node01      discovery: $discoveryUrl   hostname: node01   EOF2   # get the coreos installation script  #wget https://raw.github.com/coreos/init/master/bin/coreos-install  wget https://raw.githubusercontent.com/coreos/init/master/bin/coreos-install   # run installation  chmod 755 coreos-install  sudo ./coreos-install \        -d /dev/sda \        -




















31 comments









Read more