/proc/sys/vm/overcommit_memory
/proc/sys/vm/overcommit_ratio
This is system level setting that control how process can allocate memory from system. In C++, it directly working under malloc

echo 2 > /proc/sys/vm/overcommit_memory
echo 50 > /proc/sys/vm/overcommit_ratio

grep Committed_AS /proc/meminfo

/proc/sys/vm/overcommit_memory

Since 2.5.30 the values are:
0 (default): as before: guess about how much overcommitment is reasonable,
1: never refuse any malloc(),
2: be precise about the overcommit – never commit a virtual address space larger than swap space plus a fraction overcommit_ratio of the physical memory.

Here /proc/sys/vm/overcommit_ratio (by default 50) is another user-settable parameter. It is possible to set overcommit_ratio to values larger than 100.

After
# echo 2 > /proc/sys/vm/overcommit_memory

# echo 80 > /proc/sys/vm/overcommit_ratio

We set the system overcommit to be “allow malloc, but refust when 80% of Physical mem+SWAP been used.

One can view the currently committed amount of memory in /proc/meminfo, in the field Committed_AS.

OK, what about process level?

ulimit this control at per shell basis.
run ulimit directly will show it’s current setting.

ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control. The -H and -S options specify that the hard or soft limit
is set for the given resource. A hard limit cannot be increased by a non-root user once it is set; a soft limit may be increased up to the value of the hard limit. If neither -H nor -S
is specified, both the soft and hard limits are set. The value of limit can be a number in the unit specified for the resource or one of the special values hard, soft, or unlimited,
which stand for the current hard limit, the current soft limit, and no limit, respectively. If limit is omitted, the current value of the soft limit of the resource is printed, unless
the -H option is given. When more than one resource is specified, the limit name and unit are printed before the value. Other options are interpreted as follows:
-a All current limits are reported
-b The maximum socket buffer size
-c The maximum size of core files created
-d The maximum size of a process’s data segment
-e The maximum scheduling priority (“nice”)
-f The maximum size of files written by the shell and its children
-i The maximum number of pending signals
-l The maximum size that may be locked into memory
-m The maximum resident set size (many systems do not honor this limit)
-n The maximum number of open file descriptors (most systems do not allow this value to be set)
-p The pipe size in 512-byte blocks (this may not be set)
-q The maximum number of bytes in POSIX message queues
-r The maximum real-time scheduling priority
-s The maximum stack size
-t The maximum amount of cpu time in seconds
-u The maximum number of processes available to a single user
-v The maximum amount of virtual memory available to the shell and, on some systems, to its children
-x The maximum number of file locks
-T The maximum number of threads

If limit is given, and the -a option is not used, limit is the new value of the specified resource. If no option is given, then -f is assumed. Values are in 1024-byte increments, except
for -t, which is in seconds; -p, which is in units of 512-byte blocks; and -T, -b, -n, and -u, which are unscaled values. The return status is 0 unless an invalid option or argument is
supplied, or an error occurs while setting a new limit.

ulimit -v 1024 this limit the Virtual memory max usage for this current shell process.

To know how much Virtual memory used by one process, run ps -aux look for it’s VSZ value.

RSS is the Resident Set Size and is used to show how much memory is allocated to that process and is in RAM. It does not include memory that is swapped out. It does include memory from shared libraries as long as the pages from those libraries are actually in memory. It does include all stack and heap memory.

VSZ is the Virtual Memory Size. It includes all memory that the process can access, including memory that is swapped out and memory that is from shared libraries.