2022-12-30 15:36:24

by Holger Kiehl

[permalink] [raw]
Subject: memcpy: detected field-spanning write (size 128) of single field "&r1_cmd->io_request->SGL" at drivers/scsi/megaraid/megaraid_sas_fusion.c:3326 (size 16)

Hello,

I am getting this at boot when filesystems are being mounted on all
systems with megaraid_sas:

Starting File System Checkā€¦b5c5-42c7-93bc-06a5c05f0141...
[ 25.705445] ------------[ cut here ]------------
[ 25.718769] memcpy: detected field-spanning write (size 128) of single field "&r1_cmd->io_request->SGL" at drivers/scsi/megaraid/megaraid_sas_fusion.c:3326 (size 16)
[ 25.751153] WARNING: CPU: 107 PID: 2741 at drivers/scsi/megaraid/megaraid_sas_fusion.c:3326 megasas_prepare_secondRaid1_IO+0x13d/0x150 [megaraid_sas]
[ 25.783375] Modules linked in: sd_mod sg nvme nvme_core t10_pi crct10dif_pclmul crc32_pclmul crc32c_intel ahci crc64_rocksoft_generic ghash_clmulni_intel crc64_rocksoft ice libahci sha512_ssse3 bnxt_en crc64 sp5100_tco megaraid_sas
[ 25.834579] CPU: 107 PID: 2741 Comm: fsck.ext4 Not tainted 6.1.1 #1
[ 25.854737] Hardware name: Dell Inc. PowerEdge R7525/XXXXXX, BIOS 2.9.3 08/05/2022
[ 25.872776] RIP: 0010:megasas_prepare_secondRaid1_IO+0x13d/0x150 [megaraid_sas]
[ 25.891150] Code: 00 00 0f 85 2f ff ff ff b9 10 00 00 00 48 c7 c2 70 92 6f c0 4c 89 f6 48 c7 c7 c8 92 6f c0 c6 05 51 49 02 00 01 e8 fd 2a 3d d9 <0f> 0b e9 06 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 90 0f 1f 44
[ 25.931873] RSP: 0018:ffffa5588625f900 EFLAGS: 00010286
[ 25.947337] RAX: 0000000000000000 RBX: ffff8df60693a180 RCX: 0000000000000000
[ 25.966847] RDX: ffff8ff13e4ec700 RSI: ffff8ff13e4e0560 RDI: ffff8ff13e4e0560
[ 25.985757] RBP: ffff8df60680e000 R08: 0000000000000000 R09: ffffffff9b184de0
[ 26.012930] R10: 0000000000000001 R11: 0000000000000001 R12: ffffa5587374f058
[ 26.035873] R13: ffff8df5f9470820 R14: 0000000000000080 R15: ffff8df6191b0750
[ 26.054510] FS: 00007f8dac1e2780(0000) GS:ffff8ff13e4c0000(0000) knlGS:0000000000000000
[ 26.073109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 26.087901] CR2: 00007ffcd3263fe8 CR3: 00000100ad48e004 CR4: 0000000000770ee0
[ 26.105898] PKRU: 55555554
[ 26.118517] Call Trace:
[ 26.130464] <TASK>
[ 26.144148] megasas_build_and_issue_cmd_fusion+0x185/0x230 [megaraid_sas]
[ 26.166138] scsi_dispatch_cmd+0x8b/0x220
[ 26.181020] scsi_queue_rq+0x279/0x610
[ 26.197569] blk_mq_dispatch_rq_list+0x18b/0x690
[ 26.216612] __blk_mq_do_dispatch_sched+0xba/0x330
[ 26.230539] ? release_pages+0x15b/0x450
[ 26.244164] ? elv_attempt_insert_merge+0xbc/0x100
[ 26.257915] blk_mq_do_dispatch_sched+0x3b/0x70
[ 26.271189] __blk_mq_sched_dispatch_requests+0xf0/0x140
[ 26.285749] blk_mq_sched_dispatch_requests+0x34/0x60
[ 26.299851] __blk_mq_run_hw_queue+0x35/0x90
[ 26.316085] blk_mq_sched_insert_requests+0x6a/0x150
[ 26.337224] blk_mq_flush_plug_list+0x122/0x2f0
[ 26.352449] __blk_flush_plug+0x102/0x160
[ 26.365502] ? __wake_up_common_lock+0x8a/0xc0
[ 26.379836] blk_finish_plug+0x25/0x40
[ 26.393678] generic_writepages+0x5a/0x80
[ 26.408024] do_writepages+0xcf/0x1d0
[ 26.422927] filemap_fdatawrite_wbc+0x66/0x90
[ 26.438551] __filemap_fdatawrite_range+0x54/0x80
[ 26.456288] file_write_and_wait_range+0x43/0xa0
[ 26.469451] blkdev_fsync+0x14/0x40
[ 26.481500] __x64_sys_fsync+0x33/0x60
[ 26.495285] do_syscall_64+0x5c/0x90
[ 26.509468] ? ksys_write+0xab/0xe0
[ 26.521848] ? syscall_exit_to_user_mode+0x12/0x30
[ 26.534957] ? do_syscall_64+0x69/0x90
[ 26.546588] ? handle_mm_fault+0xee/0x2e0
[ 26.557927] ? do_user_addr_fault+0x1d6/0x690
[ 26.569652] ? exc_page_fault+0x5d/0x120
[ 26.580668] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 26.597062] RIP: 0033:0x7f8dac32c067
[ 26.608530] Code: ff ff ff ff eb b7 e8 38 82 01 00 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 4a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 41 c3 48 83 ec 18 89 7c 24 0c e8 73 6b f5 ff
[ 26.641441] RSP: 002b:00007ffcd3266108 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
[ 26.655857] RAX: ffffffffffffffda RBX: 000055f73742d400 RCX: 00007f8dac32c067
[ 26.670764] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000003
[ 26.685898] RBP: 00007ffcd3266170 R08: 0000000000000000 R09: 0000000000000000
[ 26.702612] R10: 00000000000000c8 R11: 0000000000000246 R12: 0000000000000000
[ 26.721338] R13: 00007ffcd3266178 R14: 000055f73742d1b0 R15: 000055f73742d1b0
[ 26.739231] </TASK>
[ 26.748275] ---[ end trace 0000000000000000 ]---

Think this was introduced with kernel hardening code

54d9469bc515 fortify: Add run-time WARN for cross-field memcpy()

CONFIG_FORTIFY_SOURCE=y, committed in 6.0-rc2 as mentioned in
https://bugzilla.kernel.org/show_bug.cgi?id=216563#c3

Regards,
Holger


2023-06-09 21:24:59

by Frank Reppin

[permalink] [raw]
Subject: Re: memcpy: detected field-spanning write (size 128) of single field "&r1_cmd->io_request->SGL" at drivers/scsi/megaraid/megaraid_sas_fusion.c:3326 (size 16)

Dear all,

at first - my apologies to bring this up again here.

But may I please ask/request to have this fix committed
to longterm 6.1 too?

Reason: Upcoming Debian Bookworm (currently RC4) comes with 6.1 but
does not include this fix yet - as it is only present in 6.3. - and
probably nobody noticed this one yet.

We do encounter this issue on brand new test machines (which should go
live once Bookworm is released) and this is a real showstopper
when it comes to show logs to QA audit people... ;)

Another reason: I see vanished /dev/disk/by-uuid/ entries when
this issue hits us

For example...

cryptsetup -v -y luksFormat
/dev/disk/by-uuid/926943a2-8e40-445f-aad4-2ee96807cd32
-> this command should succeed - but returns with error because
somehow this (some seconds earlier perfectly valid and existing)
by-uuid entry vanished during the issue.
Other entries pointing to the same virtual drive are not affected.
(by-id,by-path,by-diskseq)

Last but not least... is this really a warning only?!
While I don't think that something on our brand new servers is broken
(it affects all btw - same observation as Holger mentioned here earlier)
it is really disturbing to see vanishing /dev/disk/by-uuid/ entries
since they might be used somewhere else and their sudden disappearance
might cause severe havoc for other daemons looking for them (server
monitoring comes to mind ... nagios... zabbix)

Thankyou all!
cheers
Frank Reppin


--
43rd Law of Computing:
Anything that can go wr
fortune: Segmentation violation -- Core dumped