Installing Fluentd on Unix/Linux

FluentD is a cross — platform software with open source for data collection was originally developed by Treasure Data. It is written mostly in Ruby.

Installing Fluentd on Unix/Linux

To begin with, Cho install and configure something. So called pre-installation.

Configuring NTP

It is HIGHLY recommended to configure ntpd on the node to prevent incorrect timestamp in your logs. Here is an example of the settings:

Install and configure NTP on CentOS/Fedora/RedHat

Increase the number of file descriptors

To check the current size, you can use:

$ ulimit-n

If you have approximately 1024 is not enough. Then, open the:

 $ vim /etc/security/limits.conf

And add the following lines to the file:

root soft nofile 65536 
root hard nofile 65536 
* soft nofile 65536 
* hard nofile 65536

PS: it Is necessary to restart the computer.

The value that I put (65536 is the maximum value), is the most good option. The required number of file descriptors depends on your plugins and settings of FluentD.

Here is a helpful reading:

To increase the Max Open File Limit on Unix/Linux

Optimization of the kernel parameters (network configuration)

For environments with high load, consisting of many Fluentd instances, you should optimize the settings to do this, open:

 $ vim /etc/sysctl.conf

And write:

net.core.somaxconn = 1024
net.core.netdev_max_backlog = 5000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_wmem = 4096 16777216 12582912
net.ipv4.tcp_rmem = 4096 16777216 12582912
net.ipv4.tcp_max_syn_backlog = 8096
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 65535

In order for these settings to take effect, you should do:

$ sysctl -p

Or:

# reboot

If in your environment no problem with TCP_WAIT, these changes are not needed.

Let’s install…

Installing Fluentd in CentOS/Fedora/RedHat

There are several ways to install fluentD on your server, I will give a few of them.

-=== METHOD 1 ===-

Installation is simple running:

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh

To install the td-agent2, do:

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

To start, you will happen team:

$ service td-agent start

Or:

$ /etc/init.d/td-agent start

Well, of course, if yuzaete systemD:

# systemctl start td-agent.service

That’s the whole installation.

-=== METHOD 2 — use the docker ===-

To use Docker. I will tell you how to do it a little lower

-=== METHOD 3 — use gem===-

To use the gem. I will tell you how to do it a little lower

Installing Fluentd on Debian/Ubuntu

There are several ways to install fluentD on your server, I will give a few of them.

-=== METHOD 1 ===-

Installation is very simple, perform one of the following:

If you have Ubuntu Bionic:

$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent3.sh | sh

If you have Ubuntu Xenial:

$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh

If you are on Ubuntu Trusty:

$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent3.sh | sh

If you have a Jessie Debian:

$ curl -L https://toolbelt.treasuredata.com/sh/install-debian-stretch-td-agent3.sh | sh

If you have a Debian Jessie:

$ curl -L https://toolbelt.treasuredata.com/sh/install-debian-jessie-td-agent3.sh | sh

PS: As in example with RPM, you can use the td-agent2.

Run and see what came out ( if you have systemD):

# systemctl start td-agent.service && systemctl status td-agent.service

Run and see what came out ( if you have init.d):

# /etc/init.d/td-agent restart && /etc/init.d/td-agent status

Something like that.

-=== METHOD 2 — use the docker ===-

To use Docker. I will tell you how to do it a little lower

-=== METHOD 3 — use gem===-

To use the gem. I will tell you how to do it a little lower

Installing Fluentd on Mac OS X

There are several ways to install fluentD on your server, I will give a few of them.

-=== METHOD 1 ===-

Go to the link ( at the time of writing, was the most recent version) and downloaded:

$ wget https://packages.treasuredata.com.s3.amazonaws.com/3/macosx/td-agent-3.1.1-0.dmg

To set there team, if you don’t know how — here is the reading:

Install dmg packages using CLI (command line) to MacOS X

Then, run:

$ sudo launchctl load /Library/LaunchDaemons/td-agent.plist

To see the logs:

$ less /var/log/td-agent/td-agent.log

To pause a service, follow these:

$ sudo launchctl unload /Library/LaunchDaemons/td-agent.plist

That’s all!

-=== METHOD 2 — use the docker ===-

To use Docker. I will tell you how to do it a little lower

-=== METHOD 3 — use gem===-

To use the gem. I will tell you how to do it a little lower

Installing Fluentd on other Unix/Linux OS

There are several ways to install fluentD on your server, I will give a few of them.

-=== METHOD 1 — use docker ===-

Install Docker:

Install Docker on Debian/Ubuntu

Installing Docker on CentOS/RedHat/Fedora

Let’s search for packages:

$ docker search fluentd

To put performed:

$ docker pull fluent/fluentd

Create config, for example:

$ vim /tmp/fluentd.conf

And it prescribe:

<source>
 @type http
 port 9880
 bind 0.0.0.0
</source>
<match **>
 @type stdout
</match>

Run the container:

$ docker run-d 
 -p 9880:9880 -v /tmp:/fluentd/etc -e FLUENTD_CONF=fluentd.conf 
 fluent/fluentd

For a test you can perform:

$ curl -X POST -d 'json={"json":"message"}' https://localhost:9880/sample.test

-=== METHOD 2 — using source code===-

Set up git and clone the code:

$ git clone https://github.com/fluent/fluentd.git && cd fluentd

Perform the build:

$ bundle install
Fetching gem metadata from https://rubygems.org/.........
...
Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.
$ bundle exec rake build
fluentd xxx built to pkg/fluentd-xxx.gem.
$ gem install pkg/fluentd-xxx.gem

Run:

$ fluentd --setup ./fluent
$ fluentd -c ./fluent/fluent.conf-vv &
$ echo '{"json":"message"}' | fluent-cat debug.test

To stop, you can use:

$ pkill -f fluentd

Go ahead…

-=== METHOD 3 — use gem===-

First, you need Ruby >= 2.1.

After that, execute:

$ gem install fluentd --no-ri --no-rdoc

Then execute:

$ fluentd --setup ./fluent
$ fluentd -c ./fluent/fluent.conf-vv &
$ echo '{"json":"message"}' | fluent-cat debug.test

To stop the service, run:

$ pkill -f fluentd

Something like that.

Configuring Fluentd on Unix/Linux

After installation, you can install plugins. Here are some examples:

The use of plugins for FluentD

I’m using Docker, ispolzoval:

$ /opt/td-agent/embedded/bin/fluent-gem install --no-ri --no-rdoc 
 fluent-plugin-elasticsearch 
 fluent-plugin-record-modifier 
 fluent-plugin-exclude-filter 
 fluent-plugin-splunk-enterprise

You can drive to use:

$ fluent-gem install --no-ri --no-rdoc 
 fluent-plugin-elasticsearch 
 fluent-plugin-record-modifier 
 fluent-plugin-exclude-filter 
 fluent-plugin-splunk-enterprise

Or if you use td-agent:

# td-agent-gem install fluent-plugin-elasticsearch 
 fluent-plugin-record-modifier 
 fluent-plugin-exclude-filter 
 fluent-plugin-splunk-enterprise

Something like that.

Configuration Of FluentD

My fluentd.config as follows:

# http
<source>
 @type http
 @id http_input
 json format
 port 8888
 self_hostname "#{ENV['HOSTNAME']}"
<parse>
 @type json
</parse>
 tag http.events
</source>

# tcp
<source>
 @type tcp
 @tcp_input id
 format none
 format /^(?<field1>d+):(?<field2>w+)$/
 port 5170
 bind 0.0.0.0
 delimiter n
 #log-level debug
 self_hostname "#{ENV['HOSTNAME']}"
 #source_hostname_key "client_host"
<parse>
 @type json
</parse>
 tag tcp.events
</source>

# Forwarding 
<source>
 @type forward
 @id forward_input
 port 24224
 self_hostname "#{ENV['HOSTNAME']}"
 bind 0.0.0.0
 tag forward.events
</source>

# live debugging agent
<source>
 @type debug_agent
 @id debug.agent
 bind 127.0.0.1
 port 24230
 tag debug_agent.debug
</source>

###############################################################################
## match directives determine the output destinations.
## filter directives determine the event processing pipelines.
## directives system set system wide configuration.
## label directives group the output and filter for internal routing
## @include directives include other files.
## Output descriptions:
###############################################################################

# <match **.access>
<match **>
 @type copy
<store>
 @type stdout
 </store> 
</match>

## match tag=debug.** and dump to console
<match **.debug>
 @type stdout
 @id output_stdout
</match>

<filter esb_bw>
 @type record_transformer
<record>
 hostname "#{Socket.gethostname}"
</record>
</filter>

<system>
 # equal to -qq option
 log_level error

 process_name fluentd
</system>

# Include config files in the ./config.d directory
@include config.d/*.conf

To work with files you can use:

<source>
 @type tail
 @id in_tail_container_logs
 path /logs/docker_logs/*/*.log
 pos_file /data/docker.pos
 read_from_head true
 tag docker.**
 skip_refresh_on_startup false
 refresh_interval 10s
 enable_watch_timer true
 enable_stat_watcher false
<parse>
 @type json
 time_format %Y-%m-%dT%H:%M:%S.%NZ
</parse>
</source>

Filter, ( In this case, appends the name of the host system that is running the container):

<filter docker.**>
 @type record_transformer
enable_ruby
<record>
 hostname "#{Socket.gethostname}"
</record>
</filter>

Handles splitting the data into portions (chunks) and writes them to the buffer (buffer), which is located on the disk. Portions of the data queues. Sends the header:

<match docker.**>
 @type forward
 @id forward_docker
<server>
 name main_server
 host collector.example.com
 port 24224
</server>
<buffer>
 @type file
 path /data/buffers/containers
 flush_mode interval
 retry_type exponential_backoff
 flush_thread_count 2
 flush_interval 2s
retry_forever
 retry_max_interval 30
 chunk_limit_size 4M
 queue_limit_length 512
 overflow_action block
</buffer>
</match>

The same way you do for other types.

That’s all, “Install Fluentd on Unix/Linux” is completed.