Ticket 5661

Summary:	error: Database settings not recommended values: innodb_buffer_pool_size
Product:	Slurm	Reporter:	Kilian Cavalotti <kilian>
Component:	slurmdbd	Assignee:	Broderick Gardner <broderick>
Status:	RESOLVED FIXED	QA Contact:
Severity:	4 - Minor Issue
Priority:	---	CC:	isaid
Version:	18.08.0
Hardware:	Linux
OS:	Linux
Site:	Stanford	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Linux Distro:	---
Machine Name:		CLE Version:
Version Fixed:	18.08.1	Target Release:	---
DevPrio:	---	Emory-Cloud Sites:	---

Description Kilian Cavalotti 2018-08-31 18:53:26 MDT

Hi there,

We're using a datacenter planned maintenance over the week-end to upgrade Sherlock to 18.08.0.

One of the first things I noted when starting the new slurmdbd is this error message:

error: Database settings not recommended values: innodb_buffer_pool_size


In our MySQL config, we have:

innodb_buffer_pool_size=32G


Looking at the code (_check_database_variables() in accounting_storage_mysql.c), it looks like the error message is displayed if the values are not at least half the recommendation. 
The recommended value for innodb_buffer_pool_size appears to be 1GB, and 32GB is definitely larger than half of 1GB. :)

Maybe the _check_database_variables() function expects values from the config to be in bytes? Or is there something else?

I tried to set DebugLevelSyslog to debug2, but didn't see any additional message in the logs, only this:

Aug 31 17:52:10 sh-sl01 systemd: Started Slurm DBD accounting daemon.
Aug 31 17:52:10 sh-sl01 slurmdbd[171888]: error: Database settings not recommended values: innodb_buffer_pool_size
Aug 31 17:52:10 sh-sl01 slurmdbd[171888]: pre-converting step table for sherlock


Thanks!
--
Kilian

Comment 1 Tim Wickberg 2018-08-31 19:01:44 MDT

Do you mind dropping into the database and running a quick query?

What we're relying on is the output from 

show variables like 'innodb_buffer_pool_size';

which should be in bytes. Unless for some reason it isn't on your system?

There should be some minor additional details at debug2, I'm not sure why you aren't seeing that though.

As you've presumably noticed, this is just an advisory and you can safely ignore the warning if you've done this configuration already.

- Tim

Comment 2 Kilian Cavalotti 2018-08-31 19:04:33 MDT

(In reply to Tim Wickberg from comment #1)
> Do you mind dropping into the database and running a quick query?
> 
> What we're relying on is the output from 
> 
> show variables like 'innodb_buffer_pool_size';
> 
> which should be in bytes. Unless for some reason it isn't on your system?

You;re right, it's in bytes:

MariaDB [slurm_acct_db]> show variables like 'innodb_buffer_pool_size';
+-------------------------+-------------+
| Variable_name           | Value       |
+-------------------------+-------------+
| innodb_buffer_pool_size | 34359738368 |
+-------------------------+-------------+

That still seems larger than the hard-coded (mmpf) recommended value of 1073741824.

> There should be some minor additional details at debug2, I'm not sure why
> you aren't seeing that though.

Don't know either.
 
> As you've presumably noticed, this is just an advisory and you can safely
> ignore the warning if you've done this configuration already.

I assumed so. May be in that case it could be better to display this as a warning rather than an error?

Cheers,
--
Kilian

Comment 3 Tim Wickberg 2018-08-31 19:10:19 MDT

> You;re right, it's in bytes:
> 
> MariaDB [slurm_acct_db]> show variables like 'innodb_buffer_pool_size';
> +-------------------------+-------------+
> | Variable_name           | Value       |
> +-------------------------+-------------+
> | innodb_buffer_pool_size | 34359738368 |
> +-------------------------+-------------+
> 
> That still seems larger than the hard-coded (mmpf) recommended value of
> 1073741824.

I see the mistake, we'll have this fixed in 18.08.1. You can ignore this for now, it's not worth patching out.

Comment 4 Kilian Cavalotti 2018-08-31 19:12:37 MDT

(In reply to Tim Wickberg from comment #3)
> > You;re right, it's in bytes:
> > 
> > MariaDB [slurm_acct_db]> show variables like 'innodb_buffer_pool_size';
> > +-------------------------+-------------+
> > | Variable_name           | Value       |
> > +-------------------------+-------------+
> > | innodb_buffer_pool_size | 34359738368 |
> > +-------------------------+-------------+
> > 
> > That still seems larger than the hard-coded (mmpf) recommended value of
> > 1073741824.
> 
> I see the mistake, we'll have this fixed in 18.08.1. You can ignore this for
> now, it's not worth patching out.

Got it, thanks!

Cheers,
-- 
Kilian

Comment 12 Broderick Gardner 2018-10-04 09:52:38 MDT

The fix is in 18.08.1, commit ba446ec62333eeec

Closing

Comment 13 Kilian Cavalotti 2018-10-04 10:04:39 MDT

(In reply to Broderick Gardner from comment #12)
> The fix is in 18.08.1, commit ba446ec62333eeec

Thank you!
-- 
Kilian

Comment 14 Issam SAID 2020-06-25 23:26:10 MDT

we are still seeing this in 20.02.x