Scenario: How to limit memory usage per user on CentOS/RHEL
System: CentOS/RHEL 6/7

I would like to cap the amount of memory the users on a server can use. The applications developed by our users have memory leaks and I want to prevent them from using all the memory and crashing the system.

Resolution
It is possible to limit the amount of memory that each process can allocate with the RLIMIT facility in Red Hat Enterprise Linux. RLIMIT_AS limits the size of the virtual address space and it is a suitable solution in many cases which require limiting per-process memory consumption even though there are many limits related to memory.

Steps:
1) Edit limit.conf
# vi /etc/security/limits.conf

#<domain> <type> <item> <value>
# Example limit 2GB memory usage on user bachem
bachem hard as 2097152

2) Reboot the server
3) Show the ulimit

[bachem@centos7ht ~]$ ulimit -v
2097152

In the example above, processes running as ‘bachem‘ user can only use 2GB virtual address space. So, each application used by bachem cannot allocate more than 2 GB memory space.
Note : this limitation does not affect an amount of physical memory directly, and the values are in KiB (1024-byte).

Show the ulimit hard type:

[bachem@centos7ht ~]$ ulimit -a -H
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 7193
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 7193
virtual memory (kbytes, -v) 2097152
file locks (-x) unlimited
[bachem@centos7ht ~]$

Show the ulimit soft type:

[bachem@centos7ht ~]$ ulimit -a -S
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 7193
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) 2097152
file locks (-x) unlimited

For more information, refer to its manual page or examples of the configuration file:
# man ulimit
# cat /etc/security/limits.conf

[root@centos7ht ~]# cat /etc/security/limits.conf
# /etc/security/limits.conf
#
#This file sets the resource limits for the users logged in via PAM.
#It does not affect resource limits of the system services.
#
#Also note that configuration files in /etc/security/limits.d directory,
#which are read in alphabetical order, override the settings in this
#file in case the domain is the same or more specific.
#That means for example that setting a limit for wildcard domain here
#can be overriden with a wildcard setting in a config file in the
#subdirectory, but a user specific setting here can be overriden only
#with a user specific setting in the subdirectory.
#
#Each line describes a limit for a user in the form:
#
#<domain> <type> <item> <value>
#
#Where:
#<domain> can be:
# - a user name
# - a group name, with @group syntax
# - the wildcard *, for default entry
# - the wildcard %, can be also used with %group syntax,
# for maxlogin limit
#
#<type> can have the two values:
# - "soft" for enforcing the soft limits
# - "hard" for enforcing hard limits
#
#<item> can be one of the following:
# - core - limits the core file size (KB)
# - data - max data size (KB)
# - fsize - maximum filesize (KB)
# - memlock - max locked-in-memory address space (KB)
# - nofile - max number of open file descriptors
# - rss - max resident set size (KB)
# - stack - max stack size (KB)
# - cpu - max CPU time (MIN)
# - nproc - max number of processes
# - as - address space limit (KB)
# - maxlogins - max number of logins for this user
# - maxsyslogins - max number of logins on the system
# - priority - the priority to run user process with
# - locks - max number of file locks the user can hold
# - sigpending - max number of pending signals
# - msgqueue - max memory used by POSIX message queues (bytes)
# - nice - max nice priority allowed to raise to values: [-20, 19]
# - rtprio - max realtime priority
#
#<domain> <type> <item> <value>
#

#* soft core 0
#* hard rss 10000
#@student hard nproc 20
#@faculty soft nproc 20
#@faculty hard nproc 50
#ftp hard nproc 0
#@student - maxlogins 4

# End of file

# Example limit 2GB memory usage on user bachem
bachem hard as 2097152
[root@centos7ht ~]#