Ticket 25716

Created attachment 46044 [details]
slurmdbd.log

Created attachment 46045 [details]
slurmdbd.conf

Hi Ole. You may be hitting a temporary space limit during the conversion. The failure happens partway through the schema-upgrade ALTER TABLE niflheim_job_table (started 15:16:40, failed 15:19:47). This ALTER changes the primary key (drops it and re-adds job_db_inx as the primary key) and rewrites indexes. Under MariaDB, a primary-key change requires a full table rebuild.

Error 1114 "table is full" means InnoDB couldn't grow files it's writing to during this rebuild, even though df shows the file system is not full.

So the space to check is wherever your MariaDB datadir and tmpdir/innodb_tmpdir actually live. Check these:

SHOW VARIABLES LIKE 'datadir';
SHOW VARIABLES LIKE 'tmpdir';
SHOW VARIABLES LIKE 'innodb_tmpdir';
SHOW VARIABLES LIKE 'innodb_data_file_path';
SHOW VARIABLES LIKE 'innodb_file_per_table';
SELECT data_length/1024/1024/1024 AS data_gb, index_length/1024/1024/1024 AS index_gb FROM information_schema.tables WHERE table_name='niflheim_job_table';

Then run df on whatever paths datadir/tmpdir/innodb_tmpdir point to.

Note that innodb_buffer_pool_size=32768M is unrelated to this error. It's a memory cache for reads/writes, not disk space for a table rebuild so no need to change it for this issue.

Once you locate and free up space or point to a large file system, you can rerun slurmdbd and hopefully it will just work...let me know how this goes.

Hi Chad,

Thanks a lot for pointing to the database disk space as being the source of the present 
problem!  As the test server for the dry run slurmdbd upgrade [1] I had chosen a compute 
node with a root (/) filesystem of only 32 GB.  I was able to expand this to 100 GB, 
and then I repeated [1] from the beginning.  This time the slurmdbd upgrade worked 
like a charm :-)

During the upgrade the disk usage grew to 28 GB:

$ du -h /var/lib/mysql/
2.6M	/var/lib/mysql/mysql
4.0K	/var/lib/mysql/performance_schema
27G	/var/lib/mysql/slurm_acct_db
28G	/var/lib/mysql/

and it fell to 17 GB afterwards.

So we should now be ready for the upgrade to 26.05.3.

However, I wonder if the slurmdbd upgrade process could print to the log some
estimates of the disk space requirements for datadir/tmpdir/innodb_tmpdir?
The error messages reported in comment 0 are not very helpful for pointing 
in the right direction :-(

If the mysql command were able to execute external "df" commands for 
datadir/tmpdir/innodb_tmpdir, then the slurmdbd upgrade might be able 
to terminate with warnings before starting the upgrade.  That would be 
a "nice to have" feature.

Best regards,
Ole


(In reply to Chad Vizino from comment #3)
> Hi Ole. You may be hitting a temporary space limit during the conversion.
> The failure happens partway through the schema-upgrade ALTER TABLE
> niflheim_job_table (started 15:16:40, failed 15:19:47). This ALTER changes
> the primary key (drops it and re-adds job_db_inx as the primary key) and
> rewrites indexes. Under MariaDB, a primary-key change requires a full table
> rebuild.
> 
> Error 1114 "table is full" means InnoDB couldn't grow files it's writing to
> during this rebuild, even though df shows the file system is not full.

[1] https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_installation/#make-a-dry-run-database-upgrade

(In reply to Ole.H.Nielsen@fysik.dtu.dk from comment #4)
> However, I wonder if the slurmdbd upgrade process could print to the log some
> estimates of the disk space requirements for datadir/tmpdir/innodb_tmpdir?
> The error messages reported in comment 0 are not very helpful for pointing 
> in the right direction :-(
> 
> If the mysql command were able to execute external "df" commands for 
> datadir/tmpdir/innodb_tmpdir, then the slurmdbd upgrade might be able 
> to terminate with warnings before starting the upgrade.  That would be 
> a "nice to have" feature.
Glad this info helped you complete the conversion. We'll take your suggestion for checking database space in advance of the upgrade under consideration for a future release--thanks for that.

I'll go ahead and close out this ticket.