Skip to main content

Fluentd for mysql in AWS


(0) preparation
ulimit -n

If your console shows 1024, it is insufficient.
Please add following lines to your /etc/security/limits.conf file and reboot your machine.

root soft nofile 65536
root hard nofile 65536

(1) install Fluentd
// “Ubuntu 12.04 LTS / Precise”
curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-precise.sh | sh

/etc/init.d/td-agent start/stop/restart/status

// test
curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test

/etc/init.d/td-agent stop
chown: changing ownership of `/var/run/td-agent/td-agent.pid': Operation not permitted
chown: changing ownership of `/var/run/td-agent': Operation not permitted
 * Stopping td-agent td-agent                                                                                                                                                     start-stop-daemon: warning: failed to kill 31734: Operation not permitted

(2) run / test
sudo /etc/init.d/td-agent stop
sudo /etc/init.d/td-agent start

vi /var/log/td-agent/td-agent.log   // log file

(3) output for mysql
https://github.com/tagomoris/fluent-plugin-mysql

- 1.9.2 on your local environment.
1) install rvm
sudo apt-get update
curl -L https://get.rvm.io | bash -s stable --ruby
( or sudo apt-get install ruby-rvm )
source ~/.rvm/scripts/rvm

    rvm install ruby-1.9.2-p320
rvm install 1.9.1
sudo apt-get update

vi .bashrc
export PATH=$PATH:/usr/local/sbin:/usr/sbin:/sbin
export PATH=$PATH:~/.rvm/bin:/usr/lib/fluent/ruby/bin
source .bashrc

 2) install fluentd
  gem install fluentd --no-ri --no-rdoc
  /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-mysql

(4) configure file
sudo vi /etc/fluent/fluent.conf

<source>
  type tail
  format json
  path /mnt/data2/locketAdmin/logs/*
  tag output.by.sql
</source>

<match output.by.sql>
  type mysql
  host 10.97.37.241
  # port 3306 # default
  database locket_admins
  username ***
  password ***
  key_names distinct_id,email
  sql INSERT INTO AT_ACCESS_LOG (user_id,user_nm) VALUES (?,?)
  flush_interval 5s

</match>

https://github.com/tagomoris/fluent-plugin-mysql
https://github.com/toyama0919/fluent-plugin-mysql-bulk

cf. *******************************

    3 - 2) install fluent-plugin-mysql
gem install fluent-plugin-mysql
ERROR:  Error installing fluent-plugin-mysql:
fluentd requires Ruby version >= 1.9.2.

http://rubygems.rubyforge.org
LEK:/usr/lib/fluent/ruby/bin> gem install fluent-plugin-mysql
Fetching: msgpack-0.5.8.gem (100%)
Building native extensions.  This could take a while...
/home/ubuntu/.rvm/rubies/ruby-1.9.1-p431/lib/ruby/site_ruby/1.9.1/rubygems/ext/builder.rb:55: warning: Insecure world writable dir /home/ubuntu/.rvm/gems in PATH, mode 040777
Fetching: json-1.8.1.gem (100%)
Building native extensions.  This could take a while...
Fetching: yajl-ruby-1.2.1.gem (100%)
Building native extensions.  This could take a while...
Fetching: cool.io-1.2.4.gem (100%)
Building native extensions.  This could take a while...
Fetching: http_parser.rb-0.6.0.gem (100%)
Building native extensions.  This could take a while...
Fetching: sigdump-0.2.2.gem (100%)
Fetching: fluentd-0.10.49.gem (100%)
ERROR:  Error installing fluent-plugin-mysql:
fluentd requires Ruby version >= 1.9.2.

> rvm list
=* ruby-1.9.1-p431 [ x86_64 ]
   ruby-1.9.2-p320 [ x86_64 ]
   ruby-2.1.2 [ x86_64 ]
> rvm --default use ruby-1.9.2-p320
> ruby -v
ruby 1.9.2p320 (2012-04-20 revision 35421) [x86_64-linux]

LEK:/home/ubuntu> gem install fluent-plugin-mysql
Fetching: cool.io-1.2.4.gem (100%)
Building native extensions.  This could take a while...
/home/ubuntu/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/ext/builder.rb:73: warning: Insecure world writable dir /home/ubuntu/.rvm/gems in PATH, mode 040777
Successfully installed cool.io-1.2.4
Fetching: http_parser.rb-0.5.3.gem (100%)
Building native extensions.  This could take a while...
Successfully installed http_parser.rb-0.5.3
Fetching: yajl-ruby-1.2.1.gem (100%)
Building native extensions.  This could take a while...
Successfully installed yajl-ruby-1.2.1
Fetching: msgpack-0.5.8.gem (100%)
Building native extensions.  This could take a while...
Successfully installed msgpack-0.5.8
Fetching: sigdump-0.2.2.gem (100%)
Successfully installed sigdump-0.2.2
Fetching: serverengine-1.5.7.gem (100%)
ERROR:  Error installing fluent-plugin-mysql:
serverengine requires Ruby version >= 1.9.3.

> rvmsudo rvm install 1.9.3
> rvm list
   ruby-1.9.1-p431 [ x86_64 ]
 * ruby-1.9.2-p320 [ x86_64 ]
=> ruby-1.9.3-p547 [ x86_64 ]
   ruby-2.1.2 [ x86_64 ]

> rvm --default use ruby-1.9.3-p547
> gem install fluent-plugin-mysql


Comments

Post a Comment

Popular posts from this blog

DevOps JD's required skills from LinkedIn

From some of DevOps JD on linkedIn, I realised that DeveOps should be the leader of the organization. https://docs.google.com/spreadsheets/d/1P520nH0pYcAdN0rJcnMQqsgu9cV9GdknztJ92J8l7-s/pubhtml DevOps' Required Skills From LinkedIn on 8/30/16: DevOps should be the leader of the company! Yahoo Netflix Samsung Salesforce Fortinet SUM OS admin UNIX systems Unix platforms Linux administrator Linux VMs Docker VMs VMware, OpenStack, Hyper-V Openstack, KVM, VMWare Version control version control systems Git, SVN Cloud Amazon AWS AWS AWS, Azure DB MySql Oracle, MySQL, NoSQL Mysql administration and strong command of SQL MySQL RabbitMQ MySql, MongoDB, Redis, Oracle, ProgreSQL N/W TCP/IP networking, DNS, HTTP NAS Understanding of network stack, network tuning, subnet/VLANs. HAProxy, DNS, IPTable Script Lang Shell, Perl, Python, Ruby, PHP bash Python, Bash/tcsh a scripting language: Perl, Python and Unix Shell preferred Python, Perl, Ruby Python, Ruby, Shell, PHP Web LAMP stack
Post a Comment
Read more

Ubuntu GUI with VNC on Xenserver

Xenserver 에서 Ubuntu GUI 를 쓰기 위해서는 VNC 가 답인 듯... Installing Ubuntu Gnome GUI on Ubuntu Server 12.10 with VNC Update Repositories # apt-get update Install gnome and vnc: # apt-get install gnome-core vnc4server Start VNC Server: # vncserver (You’ll then be prompted to create and verify a new VNC connect password) Kill the currently running VNC Session: # vncserver -kill :1 Edit VNC startup config file: # vim .vnc/xstartup Uncomment the following line: unset SESSION_MANAGER Add the following line: gnome-session --session=gnome-classic & Comment Out the following two lines: x-terminal-emulator -geometry 1280x1024+10+10 -ls -title "$VNCDESKTOP Desktop" & x-window-manager & End result should look like: #!/bin/sh # Uncomment the following two lines for normal desktop: unset SESSION_MANAGER # exec /etc/X11/xinit/xinitrc gnome-session --session=gnome-classic & [ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup [ -r $HOME/
Post a Comment
Read more

Install CoreOs on linode without VM

Install CoreOs on linode without VM 1. Add a Linode 2. Create a new Disk   CoreOS 3. Rescue > Reboot into Rescue Mode 4. Remote Access   Launch Lish Console 5. make an install script cat <<'EOF1' > install.sh # add needed package sudo apt-get update sudo apt-get install -y curl wget whois sudo apt-get install -y ca-certificates #sudo apt-get install gawk -y # get discovery url discoveryUrl=`curl https://discovery.etcd.io/new` # write cloud-config.yml cat <<EOF2 > cloud-config.yml #cloud-config users:   - name: core     groups:       - sudo       - docker coreos:   etcd:     name: node01     discovery: $discoveryUrl hostname: node01 EOF2 # get the coreos installation script #wget https://raw.github.com/coreos/init/master/bin/coreos-install wget https://raw.githubusercontent.com/coreos/init/master/bin/coreos-install # run installation chmod 755 coreos-install sudo ./coreos-install \       -d /dev/sda \       -
31 comments
Read more