Skip to main content

Fluentd for mysql in AWS


(0) preparation
ulimit -n

If your console shows 1024, it is insufficient.
Please add following lines to your /etc/security/limits.conf file and reboot your machine.

root soft nofile 65536
root hard nofile 65536

(1) install Fluentd
// “Ubuntu 12.04 LTS / Precise”
curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-precise.sh | sh

/etc/init.d/td-agent start/stop/restart/status

// test
curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test

/etc/init.d/td-agent stop
chown: changing ownership of `/var/run/td-agent/td-agent.pid': Operation not permitted
chown: changing ownership of `/var/run/td-agent': Operation not permitted
 * Stopping td-agent td-agent                                                                                                                                                     start-stop-daemon: warning: failed to kill 31734: Operation not permitted

(2) run / test
sudo /etc/init.d/td-agent stop
sudo /etc/init.d/td-agent start

vi /var/log/td-agent/td-agent.log   // log file

(3) output for mysql
https://github.com/tagomoris/fluent-plugin-mysql

- 1.9.2 on your local environment.
1) install rvm
sudo apt-get update
curl -L https://get.rvm.io | bash -s stable --ruby
( or sudo apt-get install ruby-rvm )
source ~/.rvm/scripts/rvm

    rvm install ruby-1.9.2-p320
rvm install 1.9.1
sudo apt-get update

vi .bashrc
export PATH=$PATH:/usr/local/sbin:/usr/sbin:/sbin
export PATH=$PATH:~/.rvm/bin:/usr/lib/fluent/ruby/bin
source .bashrc

 2) install fluentd
  gem install fluentd --no-ri --no-rdoc
  /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-mysql

(4) configure file
sudo vi /etc/fluent/fluent.conf

<source>
  type tail
  format json
  path /mnt/data2/locketAdmin/logs/*
  tag output.by.sql
</source>

<match output.by.sql>
  type mysql
  host 10.97.37.241
  # port 3306 # default
  database locket_admins
  username ***
  password ***
  key_names distinct_id,email
  sql INSERT INTO AT_ACCESS_LOG (user_id,user_nm) VALUES (?,?)
  flush_interval 5s

</match>

https://github.com/tagomoris/fluent-plugin-mysql
https://github.com/toyama0919/fluent-plugin-mysql-bulk

cf. *******************************

    3 - 2) install fluent-plugin-mysql
gem install fluent-plugin-mysql
ERROR:  Error installing fluent-plugin-mysql:
fluentd requires Ruby version >= 1.9.2.

http://rubygems.rubyforge.org
LEK:/usr/lib/fluent/ruby/bin> gem install fluent-plugin-mysql
Fetching: msgpack-0.5.8.gem (100%)
Building native extensions.  This could take a while...
/home/ubuntu/.rvm/rubies/ruby-1.9.1-p431/lib/ruby/site_ruby/1.9.1/rubygems/ext/builder.rb:55: warning: Insecure world writable dir /home/ubuntu/.rvm/gems in PATH, mode 040777
Fetching: json-1.8.1.gem (100%)
Building native extensions.  This could take a while...
Fetching: yajl-ruby-1.2.1.gem (100%)
Building native extensions.  This could take a while...
Fetching: cool.io-1.2.4.gem (100%)
Building native extensions.  This could take a while...
Fetching: http_parser.rb-0.6.0.gem (100%)
Building native extensions.  This could take a while...
Fetching: sigdump-0.2.2.gem (100%)
Fetching: fluentd-0.10.49.gem (100%)
ERROR:  Error installing fluent-plugin-mysql:
fluentd requires Ruby version >= 1.9.2.

> rvm list
=* ruby-1.9.1-p431 [ x86_64 ]
   ruby-1.9.2-p320 [ x86_64 ]
   ruby-2.1.2 [ x86_64 ]
> rvm --default use ruby-1.9.2-p320
> ruby -v
ruby 1.9.2p320 (2012-04-20 revision 35421) [x86_64-linux]

LEK:/home/ubuntu> gem install fluent-plugin-mysql
Fetching: cool.io-1.2.4.gem (100%)
Building native extensions.  This could take a while...
/home/ubuntu/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/ext/builder.rb:73: warning: Insecure world writable dir /home/ubuntu/.rvm/gems in PATH, mode 040777
Successfully installed cool.io-1.2.4
Fetching: http_parser.rb-0.5.3.gem (100%)
Building native extensions.  This could take a while...
Successfully installed http_parser.rb-0.5.3
Fetching: yajl-ruby-1.2.1.gem (100%)
Building native extensions.  This could take a while...
Successfully installed yajl-ruby-1.2.1
Fetching: msgpack-0.5.8.gem (100%)
Building native extensions.  This could take a while...
Successfully installed msgpack-0.5.8
Fetching: sigdump-0.2.2.gem (100%)
Successfully installed sigdump-0.2.2
Fetching: serverengine-1.5.7.gem (100%)
ERROR:  Error installing fluent-plugin-mysql:
serverengine requires Ruby version >= 1.9.3.

> rvmsudo rvm install 1.9.3
> rvm list
   ruby-1.9.1-p431 [ x86_64 ]
 * ruby-1.9.2-p320 [ x86_64 ]
=> ruby-1.9.3-p547 [ x86_64 ]
   ruby-2.1.2 [ x86_64 ]

> rvm --default use ruby-1.9.3-p547
> gem install fluent-plugin-mysql


Comments

Post a Comment

Popular posts from this blog

Amazon RDS Blue/Green Deployments

In order to avoid some errors I experienced when proceeding as described in the official documentation, I describe what I did in order. 1) Modify parameters of source_database * error: Blue Green Deployments requires cluster parameter group has binlog enabled. RDS Parameter groups: source-params-group binlog_format => MIXED mysql> show global variables like 'binlog_format'; 2) Insert a row after rebooting the source database, to avoid this error. * error: Correct the replication errors and then switch over. Read Replica Replication Error - IOError: 1236, reason: Got fatal error 1236 from master when reading data from binary log: 'Could not find first log file name in binary log index file' => To Fix: You need to change the data in the source database. INSERT INTO dummy_table ( `favorite_id` , `favorite_order` , `user_id` , `board_id` ) VALUES ('100001', '1', '11111', '11111'); 3) Modify the param...
Post a Comment
Read more

실리콘밸리 구직 체험기2_201505 - 5. 기억나는 onsite 인터뷰

Groupon: 주워진 restful api를 이용해서 목록 UI를 구현하기, 갑작스런 javascript 문제로 버벅거렸고 javascript 안써본 사람이런 평가를 받았습니다. Frankly: log 분석을 위한 아키텍쳐를 구성해보라고 했고 apache tajo를 이용한 해법을 설명했더니 외부 솔루션 의존적이라 자체 구현 능력이 있는지 모르겠다는 평가를 받았습니다. 로그 분석은 매 프로젝트에서 언제나 하는 것 아닌가? 새로운 해법을 제시했더니 사족이 된 건지 모르겠습니다. Samsung Lab: 10개의 계단을 올라가는 경우의 수를 구하는 프로그램을 작성하라고 해서 node를 이용해서 이것 저것 지도 했는데 그냥 피보나치 수열이라고 했습니다. Phone Veeva: Object가 어떤 method들을 갖고 있는지, 자바 스크립트 inheritance 구현을 어떻게 하는지, generic 타입을 쓰는 이유가 뭔지 어떻게 쓰는지 이것저것 놓친 것이 너무 많아서 챙피했습니다. Resilient: queue 2개로 stack을 구현하라고 했는데 이렇게 작은 회사에서도 코드인터뷰는 쉽지 않았습니다. 한번은 full-stack senior 엔지니어 면접을 보면서 다음과 같은 질문들을 받았습니다. back-end와 front-end 전체 분야를 다 깊이있게 물어볼 경우 탈탈 털리는 느낌이였습니다. 특히 javascript 문제에 대한 준비가 제대로 되어 있지 못했습니다. - java, framework 일반적인 질문 - was, linux 일반적인 질문 - 멀티 쓰레드 프로그래밍 (쓰레드간 순서 통제 등) - architecture, deploy, development 방법 - javascript concept / coding (상속, closure) - SQL 쿼리 문제 - 출력된 자바 코드를 보여주고 JAVA 코드 수정하기 ㅇ full-stack 엔지니어 full-stack은 분명 ...
Post a Comment
Read more