Ulimit (limits.conf) configuring common problems in Linux

ulimit – restrictions of the current process, for example, the number of simultaneously open files. Reference: man limits.conf. The configuration file is located at /etc/security/limits.conf.

Setting limits using ulimit can prevent disaster in your Linux systems, but you need to anticipate where constraints will make sense and where they will cause problems.

To see the limits associated with your login, use the command ulimit -a. If you are using a regular user account, you will most likely see something like this:

~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7728
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7728
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

It is immediately striking that you cannot create core dumps because your maximum core file size is 0. As long as the core file size is zero, core dumps are not allowed. This makes sense for most users, as they probably won’t do anything with a core dump other than delete it, but if you need a core dump to debug your application problems, you can adjust your core file size to unlimited.

You can change the limits in the configuration file, usually /etc/security/limits.conf, or via the command line. Remember that when changing limits via the command line, your changes will not persist after an OS reboot. For example, to change the maximum number of user processes (max user processes), use the u switch (from the ulimit -a command output listing):

ulimit -u 8000

In limits.conf, limits can be set for each user or group. Each limit is specified on a separate line. For example, for the developers group, let’s configure the limits for opening files, where hard strictly defines the maximum number, and upon reaching the lower soft limit value, users will receive warnings:

@developers        soft    nofile         2048
@developers        hard    nofile         8192

Read also: ProcFS the file system for processes (/proc)

Don’t know what parameters to set? Set them as indicated below!

Soft limit is usually set at a lower level and acts as a warning to the user. This is not the final limit, but when it is reached, the system can warn the user that resources are running low. The recommended value for soft nofile can vary, but is often set in the range from 1024 to 4096.

Hard limit sets an absolute maximum that cannot be exceeded under any circumstances. This limit serves to protect the system from overload. The recommended value for hard nofile is usually significantly higher and can be set between 4096 and 65535, depending on the system and application requirements.

For example, a line in limits.conf might look like this:

* soft nofile 4096
* hard nofile 65535

This means that for all users, the soft limit on the number of open files is set to 4096, and the hard limit is set to 65535.

Too many open files

Asterisk IP-PBX: Features, Requirements, and Configuration rtp.c: Unable to allocate RTP socket: Too many open files res_timing_timerfd.c: Read error: Bad file descriptor

The “Too many open files” error means that the server is running out of resource for the maximum limit of open files (max open file limit).

You need to increase the limit on the number of simultaneously open files in the limits.conf file. To start with, let’s check what limit is set in the OS, using data from the ProcFS file system. And we will check the parameter specifying the number of simultaneously open files:

# cat /proc/sys/fs/file-max
348651

Let’s set it for the asterisk user:

# nano /etc/security/limits.conf
...
asterisk         soft    nofile           65535
asterisk         hard    nofile           65535
...

Let’s check ulimit -a:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

ulimit file sizes

Mainly when working in a UNIX system, we assume that its resources are limitless. For example, we do not care if a created file turns out to be “too large”, which is not so rare on personal computers with floppy disks. If we are involved in the maintenance and administration of a UNIX system, then we must be prepared to deal with situations where various system limits are exceeded. It is always better to investigate these issues in advance in non-critical situations, so let’s look at file size limits and their meaning.

Some parameters are “hardcoded” into the system kernel during its generation. One such value is the maximum file size. It determines the largest number of blocks a file can occupy. This parameter is closely related to the UNIX method of using file index descriptors (inodes). These are sets of pointers, among which the first ten point to data blocks, the next points to another table, the next points to a table pointing to a table, etc.

There is another file size limitation defined for each user while working in the system – the ulimit number (user limit). This value is set at the time of your login to the system and represents the number of 512-byte blocks that you can write to any given file. There is a ulimit command in the shell which, when called without arguments, outputs this number. This same command allows you to decrease your ulimit value. Only the superuser (root) can INCREASE ulimit values.

A side effect of decreasing the ulimit value is that you cannot increase it back to the registration value. The ulimit value remains the same for the entire duration of your shell session, so to restore the registration value, you need to log out of the system and then log in again.

Another interesting point is that if you set your ulimit value to 0, you won’t be able to create any files! The maximum allowed file size in this case is zero, so no file can be created. This seems reasonable enough, however, there are situations where a file of zero size CAN exist. Again, to restore your normal ulimit value, you need to log out of the system and then log in again.

As noted earlier, only the superuser can increase the ulimit value. This procedure is quite simple. First, you need to increase the ulimit value with the ulimit command, and then start a shell. This new shell has the new ulimit value. If we want the system to boot with a shell having a higher ulimit value, we can set a program in inittab (system initialization table) so that this operation is performed automatically.

Nginx number of open sockets, files (file descriptors)

In Nginx, file descriptors are required for each connection with a client, and for returning a static file from the cache. The number of required descriptors can grow rapidly. However, the total number of descriptors is limited by the operating system. Limit on the number of simultaneously open sockets: Typically, most *nix systems limit the number of sockets to 1024.

We assume that Nginx server processes are running under the nginx user. For convenience of viewing the operating system limits for the nginx user, let’s log in under it. The standard su launch won’t help (we will get the error: This account is currently not available) since the user is a system user, so we need to use switches:

# su -l nginx -s /bin/bash
$ whoami
nginx

Or simply run the command on behalf of nginx:

# su nginx -s /bin/bash -c 'ulimit  -a'

Let’s gather the current values.

Current limit (Soft limit) of open files per process for the current user:

su nginx -s /bin/bash -c 'ulimit -Sn'
1024

Maximum limit (Hard limit) of open files per process for the current user:

su nginx -s /bin/bash -c 'ulimit -Hn'
4096

Count the number of files opened by all processes:

# lsof | wc -l
9973

Count the number of files opened by all processes, according to Linux kernel data. We will see 3 numbers (the number of currently open files; the number of open but unused files; the maximum number of open files):

# cat /proc/sys/fs/file-nr
2112    0       94547

Maximum limit of open files for the entire system:

# cat /proc/sys/fs/file-max
94547

Based on the numbers obtained above, the minimum recommended numbers for Nginx ulimit turned out as follows. Let’s add them to the /etc/security/limits.conf file:

nginx soft nofile 4096
nginx hard nofile 10240

Now we change the limits in nginx; to do this, at the very beginning of the /etc/nginx/nginx.conf file, approximately after the lines:

user nginx;

we write
worker_rlimit_nofile 10240;

The worker_rlimit_nofile directive changes the limit on the number of files opened by a worker process.

Conclusion or Recommendation

Do not set the limit number higher than 65535. In the case of incoming connections, there is no obvious limit of 65535 connections, you will most likely hit the limit on file descriptors (sockets), as a socket is needed for each connection. At the same time, limits can be at the user limit level (and it must be taken into account that separate limits apply to services started via systemd, and not those specified in limits.conf). Usually, by default, descriptor limits are around 1024-4096, significantly lower than 65535. For a very large number of sockets, it will also be necessary to change sysctl for the maximum number of open files, and possibly rebuild the kernel to bypass the hardcoded upper constraints.