This article is meant as a starting point to gain knowledge about CGroups.
It links to several other documents that should also be read to get a better grasp of the concept.
Total reading time will be several hours.
Let me first start with explaining the reason why I wrote this.

Introduction

This week I was asked to do research on finding a way to CPU limit process on Red Hat 7 (RHEL7).
The reason is that my client will soon get a bare-metal server on which he wants to run JAVA Springboot process. Instead of running virtual servers (VMs) of any kind. A POC showed that running the processes directly on the OS would improve performance. The only worry is that some processes might go haywire and consume all available resources. As the heap-size of each process is already under control, the only worry left is CPU abuse. I was asked to investigate the best way to prevent such an abuse.

The environment:

• Bare metal hardware with dozens of multi-core and multi-threaded CPUs
• Red Hat 7.4
• Automated installation using Ansible and Jenkins.
• Running several hundreds Springboot(Java Virtual Machines) instances with different priorities (some must remain available, some can be lagged)

Nice and CPULimit where shortly taken into consideration but discarded. Nice is too hard to maintain when you have so many different process with so many different priority groups. CPULimit will also limit access to CPU even when the system itself has resources available, so it's not very cost efficient.

CGroups caught the attention as it only limits resources when the total usage is (near to) 100% and you can fine-grain resources per group.

CGroups pre-RHEL7

There is many documentation available for CGroups. Dozes of documents describing how to install and use it. When running on RHEL6 or below those documents are valid. When running RHEL7 the implementation of CGroups has changed entirely. It is not recommended to use the previous tooling. Instead the whole CGroups is handled by systemd.

CGroups RHEL7

RHEL7.0 uses the systemd system to set restrictions on resources. Note that the popular option "CPUQuota" was only introduced in version RHEL7.2
Once you start doing research about CGroups one is quickly drowned in many articles about this subject. Most cover just the basics and the more advanced articles are hardcore performance tuning guides.

Gathering detailed information

I will try to guide you through the reading process to quickly get an understanding of the CGroups concept.
The following documents and video's helped me explain the entire concept to a level that was good enough for me to start using it.

While searching for information about how to implement CGgroups on Red Hat 7 I came across the following websites which I recommend reading:

• https://www.redhat.com/files/summit/session-assets/2017/S103870-Demystifying-systemd.pdf
• https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/pdf/resource_management_guide/Red_Hat_Enterprise_Linux-7-Resource_Management_Guide-en-US.pdf
• https://access.redhat.com/documentation/en-us/red_hat_jboss_data_grid/7.1/html/performance_tuning_guide/java_virtual_machine_settings
• https://developers.redhat.com/blog/2017/04/04/openjdk-and-containers/
• https://www.redhat.com/en/blog/world-domination-cgroups-part-1-cgroup-basics
• https://engineering.linkedin.com/blog/2016/11/application-pauses-when-running-jvm-inside-linux-control-groups
• https://www.certdepot.net/rhel7-get-started-cgroups/
• https://www.certdepot.net/rhel-7-interesting-cgroup-behaviour/
• https://www.certdepot.net/rhel-7-2-cpuquota-option/

A very good video to watch:

Sander van Vugt: Managing performance parameters through systemd

Personal summary

After reading and watching all this materials I needed to make a summary for myself.
Some important findings:

• There are three levels at which we can manage resources like CPU: Slices, Scopes, Services
• We need to configure the slices properly otherwise a user might be able to 'steal' resources from the system.
• CPUshares and CPUquotas are two fundemental different approaches on how to manage CPU resources
• Java Garbage Collection (GC) can be impacted and need to be closely monitored
• Systemd can do so much more then I knew :)

Slices, Scopes, Services

By default, service and scope units are placed in system.slice, virtual machines and containers registered with systemd-machined are found in machine.slice, and user sessions handled by systemd-logind(1) in user.slice. See systemd.special(5) for more information.

Slice

Unit type for creating the cgroup hierarchy for resource management.
There are three slices:

• User
• system
• machine (default not used)

These three top level slices are each granted equal CPU time with each other - but this only happens once the system ends up under load.
If a single process in the User slice wants 100% and nothing else does, this process will be granted that CPU time.

Scope

Organizational unit that groups a daemon’s worker processes. You can look at it as 'slices within a slice'.

Service

Process or group of processes controlled by systemd. Basically the process that you run.

CPUShares vs CPUQuota

After RHEL7.2 you have the option to set CPUQuota. This is a simple way to make sure a certain process does not use more CPU then the quota allows. When more CPUQuota is given there is available CPU, normal

With the CPUShares option, you assigned a percentage of CPU time to a service.
With the CPUQuota option, you now set a duration in millisecond. This duration is the maximum of CPU time allowed to a service per second. This service can get an amount of CPU time below but not above this limit. Limit is set on one CPU, go above 100% to grant multiple CPUs.

Examples

Limiting CPU Consumption of a Unit

To assign the Apache service CPU quota of 20%, add the following content to
the /etc/systemd/system/httpd.service.d/cpu.conf configuration file:
[Service]
CPUQuota=20%

or

[Service]
CPUShares=1500

Tips and tricks

Show all parameters that can be used (or currently set)
systemctl show [name].service

Explanation of most of the parameters:
man systemd.resource-control

ps with cgroups
alias ps='ps xawf -eo pid,user,cgroup,args'