Installing Fluentd on Unix/Linux
FluentD is a cross — platform software with open source for data collection was originally developed by Treasure Data. It is written mostly in Ruby.
Installing Fluentd on Unix/Linux
To begin with, Cho install and configure something. So called pre-installation.
Configuring NTP
It is HIGHLY recommended to configure ntpd on the node to prevent incorrect timestamp in your logs. Here is an example of the settings:
Install and configure NTP on CentOS/Fedora/RedHat
Increase the number of file descriptors
To check the current size, you can use:
$ ulimit-n
If you have approximately 1024 is not enough. Then, open the:
$ vim /etc/security/limits.conf
And add the following lines to the file:
root soft nofile 65536 root hard nofile 65536 * soft nofile 65536 * hard nofile 65536
PS: it Is necessary to restart the computer.
The value that I put (65536 is the maximum value), is the most good option. The required number of file descriptors depends on your plugins and settings of FluentD.
Here is a helpful reading:
To increase the Max Open File Limit on Unix/Linux
Optimization of the kernel parameters (network configuration)
For environments with high load, consisting of many Fluentd instances, you should optimize the settings to do this, open:
$ vim /etc/sysctl.conf
And write:
net.core.somaxconn = 1024 net.core.netdev_max_backlog = 5000 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_wmem = 4096 16777216 12582912 net.ipv4.tcp_rmem = 4096 16777216 12582912 net.ipv4.tcp_max_syn_backlog = 8096 net.ipv4.tcp_slow_start_after_idle = 0 net.ipv4.tcp_tw_reuse = 1 net.ipv4.ip_local_port_range = 10240 65535
In order for these settings to take effect, you should do:
$ sysctl -p
Or:
# reboot
If in your environment no problem with TCP_WAIT, these changes are not needed.
Let’s install…
Installing Fluentd in CentOS/Fedora/RedHat
There are several ways to install fluentD on your server, I will give a few of them.
-=== METHOD 1 ===-
Installation is simple running:
$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh
To install the td-agent2, do:
$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh
To start, you will happen team:
$ service td-agent start
Or:
$ /etc/init.d/td-agent start
Well, of course, if yuzaete systemD:
# systemctl start td-agent.service
That’s the whole installation.
-=== METHOD 2 — use the docker ===-
To use Docker. I will tell you how to do it a little lower
-=== METHOD 3 — use gem===-
To use the gem. I will tell you how to do it a little lower
Installing Fluentd on Debian/Ubuntu
There are several ways to install fluentD on your server, I will give a few of them.
-=== METHOD 1 ===-
Installation is very simple, perform one of the following:
If you have Ubuntu Bionic:
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent3.sh | sh
If you have Ubuntu Xenial:
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh
If you are on Ubuntu Trusty:
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent3.sh | sh
If you have a Jessie Debian:
$ curl -L https://toolbelt.treasuredata.com/sh/install-debian-stretch-td-agent3.sh | sh
If you have a Debian Jessie:
$ curl -L https://toolbelt.treasuredata.com/sh/install-debian-jessie-td-agent3.sh | sh
PS: As in example with RPM, you can use the td-agent2.
Run and see what came out ( if you have systemD):
# systemctl start td-agent.service && systemctl status td-agent.service
Run and see what came out ( if you have init.d):
# /etc/init.d/td-agent restart && /etc/init.d/td-agent status
Something like that.
-=== METHOD 2 — use the docker ===-
To use Docker. I will tell you how to do it a little lower
-=== METHOD 3 — use gem===-
To use the gem. I will tell you how to do it a little lower
Installing Fluentd on Mac OS X
There are several ways to install fluentD on your server, I will give a few of them.
-=== METHOD 1 ===-
Go to the link ( at the time of writing, was the most recent version) and downloaded:
$ wget https://packages.treasuredata.com.s3.amazonaws.com/3/macosx/td-agent-3.1.1-0.dmg
To set there team, if you don’t know how — here is the reading:
Install dmg packages using CLI (command line) to MacOS X
Then, run:
$ sudo launchctl load /Library/LaunchDaemons/td-agent.plist
To see the logs:
$ less /var/log/td-agent/td-agent.log
To pause a service, follow these:
$ sudo launchctl unload /Library/LaunchDaemons/td-agent.plist
That’s all!
-=== METHOD 2 — use the docker ===-
To use Docker. I will tell you how to do it a little lower
-=== METHOD 3 — use gem===-
To use the gem. I will tell you how to do it a little lower
Installing Fluentd on other Unix/Linux OS
There are several ways to install fluentD on your server, I will give a few of them.
-=== METHOD 1 — use docker ===-
Install Docker:
Install Docker on Debian/Ubuntu
Installing Docker on CentOS/RedHat/Fedora
Let’s search for packages:
$ docker search fluentd
To put performed:
$ docker pull fluent/fluentd
Create config, for example:
$ vim /tmp/fluentd.conf
And it prescribe:
<source> @type http port 9880 bind 0.0.0.0 </source> <match **> @type stdout </match>
Run the container:
$ docker run-d -p 9880:9880 -v /tmp:/fluentd/etc -e FLUENTD_CONF=fluentd.conf fluent/fluentd
For a test you can perform:
$ curl -X POST -d 'json={"json":"message"}' https://localhost:9880/sample.test
-=== METHOD 2 — using source code===-
Set up git and clone the code:
$ git clone https://github.com/fluent/fluentd.git && cd fluentd
Perform the build:
$ bundle install Fetching gem metadata from https://rubygems.org/......... ... Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed. $ bundle exec rake build fluentd xxx built to pkg/fluentd-xxx.gem. $ gem install pkg/fluentd-xxx.gem
Run:
$ fluentd --setup ./fluent
$ fluentd -c ./fluent/fluent.conf-vv &
$ echo '{"json":"message"}' | fluent-cat debug.test
To stop, you can use:
$ pkill -f fluentd
Go ahead…
-=== METHOD 3 — use gem===-
First, you need Ruby >= 2.1.
After that, execute:
$ gem install fluentd --no-ri --no-rdoc
Then execute:
$ fluentd --setup ./fluent
$ fluentd -c ./fluent/fluent.conf-vv &
$ echo '{"json":"message"}' | fluent-cat debug.test
To stop the service, run:
$ pkill -f fluentd
Something like that.
Configuring Fluentd on Unix/Linux
After installation, you can install plugins. Here are some examples:
The use of plugins for FluentD
I’m using Docker, ispolzoval:
$ /opt/td-agent/embedded/bin/fluent-gem install --no-ri --no-rdoc fluent-plugin-elasticsearch fluent-plugin-record-modifier fluent-plugin-exclude-filter fluent-plugin-splunk-enterprise
You can drive to use:
$ fluent-gem install --no-ri --no-rdoc fluent-plugin-elasticsearch fluent-plugin-record-modifier fluent-plugin-exclude-filter fluent-plugin-splunk-enterprise
Or if you use td-agent:
# td-agent-gem install fluent-plugin-elasticsearch fluent-plugin-record-modifier fluent-plugin-exclude-filter fluent-plugin-splunk-enterprise
Something like that.
Configuration Of FluentD
My fluentd.config as follows:
# http
<source>
@type http
@id http_input
json format
port 8888
self_hostname "#{ENV['HOSTNAME']}"
<parse>
@type json
</parse>
tag http.events
</source>
# tcp
<source>
@type tcp
@tcp_input id
format none
format /^(?<field1>d+):(?<field2>w+)$/
port 5170
bind 0.0.0.0
delimiter n
#log-level debug
self_hostname "#{ENV['HOSTNAME']}"
#source_hostname_key "client_host"
<parse>
@type json
</parse>
tag tcp.events
</source>
# Forwarding
<source>
@type forward
@id forward_input
port 24224
self_hostname "#{ENV['HOSTNAME']}"
bind 0.0.0.0
tag forward.events
</source>
# live debugging agent
<source>
@type debug_agent
@id debug.agent
bind 127.0.0.1
port 24230
tag debug_agent.debug
</source>
###############################################################################
## match directives determine the output destinations.
## filter directives determine the event processing pipelines.
## directives system set system wide configuration.
## label directives group the output and filter for internal routing
## @include directives include other files.
## Output descriptions:
###############################################################################
# <match **.access>
<match **>
@type copy
<store>
@type stdout
</store>
</match>
## match tag=debug.** and dump to console
<match **.debug>
@type stdout
@id output_stdout
</match>
<filter esb_bw>
@type record_transformer
<record>
hostname "#{Socket.gethostname}"
</record>
</filter>
<system>
# equal to -qq option
log_level error
process_name fluentd
</system>
# Include config files in the ./config.d directory
@include config.d/*.conf
To work with files you can use:
<source> @type tail @id in_tail_container_logs path /logs/docker_logs/*/*.log pos_file /data/docker.pos read_from_head true tag docker.** skip_refresh_on_startup false refresh_interval 10s enable_watch_timer true enable_stat_watcher false <parse> @type json time_format %Y-%m-%dT%H:%M:%S.%NZ </parse> </source>
Filter, ( In this case, appends the name of the host system that is running the container):
<filter docker.**>
@type record_transformer
enable_ruby
<record>
hostname "#{Socket.gethostname}"
</record>
</filter>
Handles splitting the data into portions (chunks) and writes them to the buffer (buffer), which is located on the disk. Portions of the data queues. Sends the header:
<match docker.**> @type forward @id forward_docker <server> name main_server host collector.example.com port 24224 </server> <buffer> @type file path /data/buffers/containers flush_mode interval retry_type exponential_backoff flush_thread_count 2 flush_interval 2s retry_forever retry_max_interval 30 chunk_limit_size 4M queue_limit_length 512 overflow_action block </buffer> </match>
The same way you do for other types.
That’s all, “Install Fluentd on Unix/Linux” is completed.