Ticket 17363 - GCP a2-highgpu-8g nodes unable to be used with hpc toolkit
Summary: GCP a2-highgpu-8g nodes unable to be used with hpc toolkit
Status: OPEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: GCP (show other tickets)
Version: 23.11.x
Hardware: Linux Linux
: 6 - No support contract
Assignee: Jacob Jenson
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2023-08-07 08:17 MDT by Chris Raynor
Modified: 2023-08-07 08:17 MDT (History)
0 users

See Also:
Site: -Other-
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Chris Raynor 2023-08-07 08:17:46 MDT 
Since https://github.com/SchedMD/slurm-gcp/commit/c5095d0bc2f38054d678e9a32568883dec0132c5 instances of type a2-highgpu-8g (or similar) nodes cannot be created - the controller node fails with:

2023-08-07 12:04:08,247 INFO: Setting up controller
2023-08-07 12:04:08,249 INFO: installing custom scripts: compute.d/ghpc_startup.sh,controller.d/ghpc_startup.sh,partition.d/a2/ghpc_startup.sh,partition.d/dev/ghpc_startup.sh
2023-08-07 12:04:08,249 DEBUG: install_custom_scripts: compute.d/ghpc_startup.sh
2023-08-07 12:04:08,251 DEBUG: install_custom_scripts: controller.d/ghpc_startup.sh
2023-08-07 12:04:08,252 DEBUG: install_custom_scripts: partition.d/a2/ghpc_startup.sh
2023-08-07 12:04:08,254 DEBUG: install_custom_scripts: partition.d/dev/ghpc_startup.sh
2023-08-07 12:04:08,259 DEBUG: compute_service: Using version=v1 of Google Compute Engine API
2023-08-07 12:04:44,512 WARNING: core count in machine type a2-highgpu-8g is not an integer. Default to 1 socket.
2023-08-07 12:04:44,512 ERROR: invalid literal for int() with base 10: '8g'
--
  File "/slurm/scripts/setup.py", line 1071, in setup_controller
    gen_cloud_conf()
  File "/slurm/scripts/setup.py", line 341, in gen_cloud_conf
    content = make_cloud_conf(lkp, cloud_parameters=cloud_parameters)
  File "/slurm/scripts/setup.py", line 330, in make_cloud_conf
    lines = [
  File "/slurm/scripts/setup.py", line 333, in <genexpr>
    *(partitionlines(p, lkp) for p in lkp.cfg.partitions.values()),
  File "/slurm/scripts/setup.py", line 272, in partitionlines
    group_lines = [
  File "/slurm/scripts/setup.py", line 273, in <listcomp>
    node_group_lines(group, part_name, lkp)
  File "/slurm/scripts/setup.py", line 210, in node_group_lines
    machine_conf = lkp.template_machine_conf(node_group.instance_template)
  File "/slurm/scripts/util.py", line 1540, in template_machine_conf
    _div = 2 if getThreadsPerCore(template) == 1 else 1
  File "/slurm/scripts/util.py", line 1117, in getThreadsPerCore
    if not isSmt(template):
  File "/slurm/scripts/util.py", line 1099, in isSmt
    machineTypeCore: int = int(matches["core"])
ValueError: invalid literal for int() with base 10: '8g'

https://github.com/SchedMD/slurm-gcp/commit/6cb0fd3f706ee5fc742a7c243bf2648ee45a9729 also seems to make the same assumption