SchedMD - Slurm Support – Full Text Ticket Listing
- Home
- | New
- | Browse
- | Search
- | [?]
- | Reports
- | Help
- | New Account
- | Log In
- | Forgot Password
| Summary: | cgroup constrain devices not enforced | ||
|---|---|---|---|
| Product: | Slurm | Reporter: | Dylan Simon <dsimon> |
| Component: | slurmd | Assignee: | Marshall Garey <marshall> |
| Status: | RESOLVED INFOGIVEN | QA Contact: | |
| Severity: | 4 - Minor Issue | ||
| Priority: | --- | ||
| Version: | 17.11.8 | ||
| Hardware: | Linux | ||
| OS: | Linux | ||
| See Also: |
https://bugs.schedmd.com/show_bug.cgi?id=6062 https://bugs.schedmd.com/show_bug.cgi?id=5361 |
||
| Site: | Simons Foundation & Flatiron Institute | Slinky Site: | --- |
| Alineos Sites: | --- | Atos/Eviden Sites: | --- |
| Confidential Site: | --- | Coreweave sites: | --- |
| Cray Sites: | --- | DS9 clusters: | --- |
| Google sites: | --- | HPCnow Sites: | --- |
| HPE Sites: | --- | IBM Sites: | --- |
| NOAA SIte: | --- | NoveTech Sites: | --- |
| Nvidia HWinf-CS Sites: | --- | OCF Sites: | --- |
| Recursion Pharma Sites: | --- | SFW Sites: | --- |
| SNIC sites: | --- | Tzag Elita Sites: | --- |
| Linux Distro: | --- | Machine Name: | |
| CLE Version: | Version Fixed: | ||
| Target Release: | --- | DevPrio: | --- |
| Emory-Cloud Sites: | --- | ||
| Attachments: |
slurm.conf
cgroup.conf |
||
|
Description
Dylan Simon
2018-12-17 12:52:56 MST
Created attachment 8673 [details]
cgroup.conf
On further testing, the gres deny rules for unallocated /dev/nvidia devices may be working, but the default whitelist is definitely not -- unlisted devices can still be used. At one point in time, the way constraining devices in cgroups works changed. You'll find some helpful background here: https://bugs.schedmd.com/show_bug.cgi?id=5361https://www.kernel.org/doc/Documentation/cgroup-v1/devices.txt The second paragraph: "The root device cgroup starts with rwm to 'all'. A child device cgroup gets a copy of the parent. Administrators can then remove devices from the whitelist or add new entries. A child cgroup can never receive a device access which is denied by its parent." The last paragraph: "device cgroups is implemented internally using a behavior (ALLOW, DENY) and a list of exceptions. The internal state is controlled using the same user interface to preserve compatibility with the previous whitelist-only implementation. Removal or addition of exceptions that will reduce the access to devices will be propagated down the hierarchy. For every propagated exception, the effective rules will be re-evaluated based on current parent's access rules." It sounds like there was a previous behavior ("previous whitelist-only implementation"). But with the new behavior, ***the cgroup_allowed_devices file doesn't do anything anymore.*** We haven't yet included the contribution in bug 5361. There is additional work that needs to be done to cleanup the task/cgroup plugin. Currently the plugin whitelists everything in the cgroup_allowed_devices file (which isn't needed, since everything is already whitelisted). Then it whitelists any GRES the job has in its allocation (also not needed), and blacklists every GRES the job does not have in its allocation (this is how devices are constrained). So, unless you specify a device in your gres.conf, it will be available to be used. Does that answer your question? That makes sense and sounds like what we're seeing, yes -- only explicit gres devices are removed from the whitelist. We can explicitly set the whitelist in some parent cgroup, or this may be improved in some future release. Thanks. You're welcome. Closing |