Skip to main content

Fluentd for mysql in AWS


(0) preparation
ulimit -n

If your console shows 1024, it is insufficient.
Please add following lines to your /etc/security/limits.conf file and reboot your machine.

root soft nofile 65536
root hard nofile 65536

(1) install Fluentd
// “Ubuntu 12.04 LTS / Precise”
curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-precise.sh | sh

/etc/init.d/td-agent start/stop/restart/status

// test
curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test

/etc/init.d/td-agent stop
chown: changing ownership of `/var/run/td-agent/td-agent.pid': Operation not permitted
chown: changing ownership of `/var/run/td-agent': Operation not permitted
 * Stopping td-agent td-agent                                                                                                                                                     start-stop-daemon: warning: failed to kill 31734: Operation not permitted

(2) run / test
sudo /etc/init.d/td-agent stop
sudo /etc/init.d/td-agent start

vi /var/log/td-agent/td-agent.log   // log file

(3) output for mysql
https://github.com/tagomoris/fluent-plugin-mysql

- 1.9.2 on your local environment.
1) install rvm
sudo apt-get update
curl -L https://get.rvm.io | bash -s stable --ruby
( or sudo apt-get install ruby-rvm )
source ~/.rvm/scripts/rvm

    rvm install ruby-1.9.2-p320
rvm install 1.9.1
sudo apt-get update

vi .bashrc
export PATH=$PATH:/usr/local/sbin:/usr/sbin:/sbin
export PATH=$PATH:~/.rvm/bin:/usr/lib/fluent/ruby/bin
source .bashrc

 2) install fluentd
  gem install fluentd --no-ri --no-rdoc
  /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-mysql

(4) configure file
sudo vi /etc/fluent/fluent.conf

<source>
  type tail
  format json
  path /mnt/data2/locketAdmin/logs/*
  tag output.by.sql
</source>

<match output.by.sql>
  type mysql
  host 10.97.37.241
  # port 3306 # default
  database locket_admins
  username ***
  password ***
  key_names distinct_id,email
  sql INSERT INTO AT_ACCESS_LOG (user_id,user_nm) VALUES (?,?)
  flush_interval 5s

</match>

https://github.com/tagomoris/fluent-plugin-mysql
https://github.com/toyama0919/fluent-plugin-mysql-bulk

cf. *******************************

    3 - 2) install fluent-plugin-mysql
gem install fluent-plugin-mysql
ERROR:  Error installing fluent-plugin-mysql:
fluentd requires Ruby version >= 1.9.2.

http://rubygems.rubyforge.org
LEK:/usr/lib/fluent/ruby/bin> gem install fluent-plugin-mysql
Fetching: msgpack-0.5.8.gem (100%)
Building native extensions.  This could take a while...
/home/ubuntu/.rvm/rubies/ruby-1.9.1-p431/lib/ruby/site_ruby/1.9.1/rubygems/ext/builder.rb:55: warning: Insecure world writable dir /home/ubuntu/.rvm/gems in PATH, mode 040777
Fetching: json-1.8.1.gem (100%)
Building native extensions.  This could take a while...
Fetching: yajl-ruby-1.2.1.gem (100%)
Building native extensions.  This could take a while...
Fetching: cool.io-1.2.4.gem (100%)
Building native extensions.  This could take a while...
Fetching: http_parser.rb-0.6.0.gem (100%)
Building native extensions.  This could take a while...
Fetching: sigdump-0.2.2.gem (100%)
Fetching: fluentd-0.10.49.gem (100%)
ERROR:  Error installing fluent-plugin-mysql:
fluentd requires Ruby version >= 1.9.2.

> rvm list
=* ruby-1.9.1-p431 [ x86_64 ]
   ruby-1.9.2-p320 [ x86_64 ]
   ruby-2.1.2 [ x86_64 ]
> rvm --default use ruby-1.9.2-p320
> ruby -v
ruby 1.9.2p320 (2012-04-20 revision 35421) [x86_64-linux]

LEK:/home/ubuntu> gem install fluent-plugin-mysql
Fetching: cool.io-1.2.4.gem (100%)
Building native extensions.  This could take a while...
/home/ubuntu/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/ext/builder.rb:73: warning: Insecure world writable dir /home/ubuntu/.rvm/gems in PATH, mode 040777
Successfully installed cool.io-1.2.4
Fetching: http_parser.rb-0.5.3.gem (100%)
Building native extensions.  This could take a while...
Successfully installed http_parser.rb-0.5.3
Fetching: yajl-ruby-1.2.1.gem (100%)
Building native extensions.  This could take a while...
Successfully installed yajl-ruby-1.2.1
Fetching: msgpack-0.5.8.gem (100%)
Building native extensions.  This could take a while...
Successfully installed msgpack-0.5.8
Fetching: sigdump-0.2.2.gem (100%)
Successfully installed sigdump-0.2.2
Fetching: serverengine-1.5.7.gem (100%)
ERROR:  Error installing fluent-plugin-mysql:
serverengine requires Ruby version >= 1.9.3.

> rvmsudo rvm install 1.9.3
> rvm list
   ruby-1.9.1-p431 [ x86_64 ]
 * ruby-1.9.2-p320 [ x86_64 ]
=> ruby-1.9.3-p547 [ x86_64 ]
   ruby-2.1.2 [ x86_64 ]

> rvm --default use ruby-1.9.3-p547
> gem install fluent-plugin-mysql


Comments

Post a Comment

Popular posts from this blog

Install CoreOs on linode without VM

Install CoreOs on linode without VM 1. Add a Linode 2. Create a new Disk   CoreOS 3. Rescue > Reboot into Rescue Mode 4. Remote Access   Launch Lish Console 5. make an install script cat <<'EOF1' > install.sh # add needed package sudo apt-get update sudo apt-get install -y curl wget whois sudo apt-get install -y ca-certificates #sudo apt-get install gawk -y # get discovery url discoveryUrl=`curl https://discovery.etcd.io/new` # write cloud-config.yml cat <<EOF2 > cloud-config.yml #cloud-config users:   - name: core     groups:       - sudo       - docker coreos:   etcd:     name: node01     discovery: $discoveryUrl hostname: node01 EOF2 # get the coreos installation script #wget https://raw.github.com/coreos/init/master/bin/coreos-install wget https://raw.githubusercontent.com/coreos/init/master/bin/coreos-install # run installation chmod 75...
31 comments
Read more

Make vagrant vm for Fedora-16

Make vagrant vm for Fedora-16 # https://cbednarski.com/articles/veewee/ # 1. requirements curl -L https://get.rvm.io | bash -s stable --rails --autolibs=enabled gem update gem install net-scp # 2. install veewee gem install veewee git clone https://github.com/jedi4ever/veewee.git cd veewee # 3. make a veewee vbox #https://github.com/jedi4ever/veewee/tree/master/templates #veewee vbox define awesome-ubuntu-server ubuntu-12.04.2-server-amd64 veewee vbox define awesome-fedora-server Fedora-16-i386 # 4. change url in definition.rb cf. /Users/dhong/tmp/veewee/templates/Fedora-16-x86_64-netboot/definition.rb :iso_file => "Fedora-16-i686-Live-KDE.iso", :iso_src => "http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/16/Live/i686/Fedora-16-i686-Live-KDE.iso", # 5. build vbox veewee vbox build awesome-fedora-server -n ll ~/VirtualBox VMs/ # 6. export to vagrant # need to shutdown virtualbox of awesome-fedora-server veewee...
Post a Comment
Read more

Amazon RDS Blue/Green Deployments

In order to avoid some errors I experienced when proceeding as described in the official documentation, I describe what I did in order. 1) Modify parameters of source_database * error: Blue Green Deployments requires cluster parameter group has binlog enabled. RDS Parameter groups: source-params-group binlog_format => MIXED mysql> show global variables like 'binlog_format'; 2) Insert a row after rebooting the source database, to avoid this error. * error: Correct the replication errors and then switch over. Read Replica Replication Error - IOError: 1236, reason: Got fatal error 1236 from master when reading data from binary log: 'Could not find first log file name in binary log index file' => To Fix: You need to change the data in the source database. INSERT INTO dummy_table ( `favorite_id` , `favorite_order` , `user_id` , `board_id` ) VALUES ('100001', '1', '11111', '11111'); 3) Modify the param...
Post a Comment
Read more