Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32;
Content-Type: multipart/mixed; boundary="------------7Ax4MydM3pHKUDdN0cmwlaxt"
Message-ID: <c866bcfa-85cc-44fb-9b54-bb4840f588e6@sapience.com>
Date:   Thu, 7 Dec 2023 10:58:04 -0500
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: md raid6 oops in 6.6.4 stable
Content-Language: en-US
To:     Guoqing Jiang <guoqing.jiang@linux.dev>,
        Bagas Sanjaya <bagasdotme@gmail.com>, snitzer@kernel.org,
        song@kernel.org, yukuai3@huawei.com, axboe@kernel.dk,
        mpatocka@redhat.com, heinzm@redhat.com,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Linux RAID <linux-raid@vger.kernel.org>,
        Linux Regressions <regressions@lists.linux.dev>
Cc:     Bhanu Victor DiCara <00bvd0+linux@gmail.com>,
        Xiao Ni <xni@redhat.com>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>
References: <6e6816dd-2ec5-4bca-9558-60cfde46ef8c@sapience.com>
 <ZXHJEkwIJ5zKTMjV@archie.me>
 <be56b5df-fef8-4dbe-bb98-f6370a692d6e@sapience.com>
 <714b22c7-b8dd-008d-a1ea-a184dc8ec1cf@linux.dev>
From:   Genes Lists <lists@sapience.com>
In-Reply-To: <714b22c7-b8dd-008d-a1ea-a184dc8ec1cf@linux.dev>
Precedence: bulk

This is a multi-part message in MIME format.
--------------7Ax4MydM3pHKUDdN0cmwlaxt
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

On 12/7/23 09:42, Guoqing Jiang wrote:
> Hi,
> 
> On 12/7/23 21:55, Genes Lists wrote:
>> On 12/7/23 08:30, Bagas Sanjaya wrote:
>>> On Thu, Dec 07, 2023 at 08:10:04AM -0500, Genes Lists wrote:
>>>> I have not had chance to git bisect this but since it happened in 
>>>> stable I
>>>> thought it was important to share sooner than later.
>>>>
>>>> One possibly relevant commit between 6.6.3 and 6.6.4 could be:
>>>>
>>>>    commit 2c975b0b8b11f1ffb1ed538609e2c89d8abf800e
>>>>    Author: Song Liu <song@kernel.org>
>>>>    Date:   Fri Nov 17 15:56:30 2023 -0800
>>>>
>>>>      md: fix bi_status reporting in md_end_clone_io
>>>>
>>>> log attached shows page_fault_oops.
>>>> Machine was up for 3 days before crash happened.
> 
> Could you decode the oops (I can't find it in lore for some reason) 
> ([1])? And
> can it be reproduced reliably? If so, pls share the reproduce step.
> 
> [1]. https://lwn.net/Articles/592724/
> 
> Thanks,
> Guoqing

   - reproducing
     An rsync runs 2 x / day. It copies to this server from another. The 
copy is from a (large) top level directory. On the 3rd day after booting 
6.6.4,  the second of these rysnc's triggered the oops. I need to do 
more testing to see if I can reliably reproduce. I have not seen this 
oops on earlier stable kernels.

   - decoding oops with scripts/decode_stacktrace.sh had errors :
    readelf: Error: Not an ELF file - it has the wrong magic bytes at 
the start

    It appears that the decode script doesn't handle compressed modules. 
  I changed the readelf line to decompress first. This fixes the above 
script complaint and the result is attached.

gene


--------------7Ax4MydM3pHKUDdN0cmwlaxt
Content-Type: text/plain; charset=UTF-8; name="raid6-stacktrace"
Content-Disposition: attachment; filename="raid6-stacktrace"
Content-Transfer-Encoding: base64

RGVjIDA2IDE5OjIwOjU0IHM2IGtlcm5lbDogQlVHOiB1bmFibGUgdG8gaGFuZGxlIHBhZ2Ug
ZmF1bHQgZm9yIGFkZHJlc3M6IGZmZmY4ODgxMDE5MzEyZTgKRGVjIDA2IDE5OjIwOjU0IHM2
IGtlcm5lbDogI1BGOiBzdXBlcnZpc29yIHdyaXRlIGFjY2VzcyBpbiBrZXJuZWwgbW9kZQpE
ZWMgMDYgMTk6MjA6NTQgczYga2VybmVsOiAjUEY6IGVycm9yX2NvZGUoMHgwMDAzKSAtIHBl
cm1pc3Npb25zIHZpb2xhdGlvbgpEZWMgMDYgMTk6MjA6NTQgczYga2VybmVsOiBQR0QgMzM2
ZTAxMDY3IFA0RCAzMzZlMDEwNjcgUFVEIDEwMTllZTA2MyBQTUQgMTAxOWYwMDYzIFBURSA4
MDAwMDAwMTAxOTMxMDIxCkRlYyAwNiAxOToyMDo1NCBzNiBrZXJuZWw6IE9vcHM6IDAwMDMg
WyMxXSBQUkVFTVBUIFNNUCBQVEkKRGVjIDA2IDE5OjIwOjU0IHM2IGtlcm5lbDogQ1BVOiAz
IFBJRDogNzczIENvbW06IG1kMTI3X3JhaWQ2IE5vdCB0YWludGVkIDYuNi40LXN0YWJsZS0x
ICM0IDc4NGMxYzcxMDY0NmNmZmMxZThjYzU5NzhmOGY2Y2VjOTc0YWExNzkKRGVjIDA2IDE5
OjIwOjU0IHM2IGtlcm5lbDogSGFyZHdhcmUgbmFtZTogVG8gQmUgRmlsbGVkIEJ5IE8uRS5N
LiBUbyBCZSBGaWxsZWQgQnkgTy5FLk0uL1ozNzAgRXh0cmVtZTQsIEJJT1MgUDQuMjAgMTAv
MzEvMjAxOQpEZWMgMDYgMTk6MjA6NTQgczYga2VybmVsOiBSSVA6IHVwZGF0ZV9pb190aWNr
cysweDJjLzB4NjAgCkRlYyAwNiAxOToyMDo1NCBzNiBrZXJuZWw6IENvZGU6IDFmIDAwIDBm
IDFmIDQ0IDAwIDAwIDQ4IDhiIDRmIDI4IDQ4IDM5IGYxIDc4IDE3IDgwIDdmIDMxIDAwIDc0
IDNiIDQ4IDhiIDQ3IDEwIDQ4IDhiIDc4IDQwIDQ4IDhiIDRmIDI4IDQ4IDM5IGYxIDc5IGU5
IDQ4IDg5IGM4IDxmMD4gNDggMGYgYjEgNzcgMjggNzUgZGUgNDggODkgZjAgNDggMjkgYzgg
ODQgZDIgYjkgMDEgMDAgPgpBbGwgY29kZQo9PT09PT09PQogICAwOgkxZiAgICAgICAgICAg
ICAgICAgICAJKGJhZCkKICAgMToJMDAgMGYgICAgICAgICAgICAgICAgCWFkZCAgICAlY2ws
KCVyZGkpCiAgIDM6CTFmICAgICAgICAgICAgICAgICAgIAkoYmFkKQogICA0Ogk0NCAwMCAw
MCAgICAgICAgICAgICAJYWRkICAgICVyOGIsKCVyYXgpCiAgIDc6CTQ4IDhiIDRmIDI4ICAg
ICAgICAgIAltb3YgICAgMHgyOCglcmRpKSwlcmN4CiAgIGI6CTQ4IDM5IGYxICAgICAgICAg
ICAgIAljbXAgICAgJXJzaSwlcmN4CiAgIGU6CTc4IDE3ICAgICAgICAgICAgICAgIAlqcyAg
ICAgMHgyNwogIDEwOgk4MCA3ZiAzMSAwMCAgICAgICAgICAJY21wYiAgICQweDAsMHgzMSgl
cmRpKQogIDE0Ogk3NCAzYiAgICAgICAgICAgICAgICAJamUgICAgIDB4NTEKICAxNjoJNDgg
OGIgNDcgMTAgICAgICAgICAgCW1vdiAgICAweDEwKCVyZGkpLCVyYXgKICAxYToJNDggOGIg
NzggNDAgICAgICAgICAgCW1vdiAgICAweDQwKCVyYXgpLCVyZGkKICAxZToJNDggOGIgNGYg
MjggICAgICAgICAgCW1vdiAgICAweDI4KCVyZGkpLCVyY3gKICAyMjoJNDggMzkgZjEgICAg
ICAgICAgICAgCWNtcCAgICAlcnNpLCVyY3gKICAyNToJNzkgZTkgICAgICAgICAgICAgICAg
CWpucyAgICAweDEwCiAgMjc6CTQ4IDg5IGM4ICAgICAgICAgICAgIAltb3YgICAgJXJjeCwl
cmF4CiAgMmE6KglmMCA0OCAwZiBiMSA3NyAyOCAgICAJbG9jayBjbXB4Y2hnICVyc2ksMHgy
OCglcmRpKQkJPC0tIHRyYXBwaW5nIGluc3RydWN0aW9uCiAgMzA6CTc1IGRlICAgICAgICAg
ICAgICAgIAlqbmUgICAgMHgxMAogIDMyOgk0OCA4OSBmMCAgICAgICAgICAgICAJbW92ICAg
ICVyc2ksJXJheAogIDM1Ogk0OCAyOSBjOCAgICAgICAgICAgICAJc3ViICAgICVyY3gsJXJh
eAogIDM4Ogk4NCBkMiAgICAgICAgICAgICAgICAJdGVzdCAgICVkbCwlZGwKICAzYToJYjkg
ICAgICAgICAgICAgICAgICAgCS5ieXRlIDB4YjkKICAzYjoJMDEgMDAgICAgICAgICAgICAg
ICAgCWFkZCAgICAlZWF4LCglcmF4KQoJLi4uCgpDb2RlIHN0YXJ0aW5nIHdpdGggdGhlIGZh
dWx0aW5nIGluc3RydWN0aW9uCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT0KICAgMDoJZjAgNDggMGYgYjEgNzcgMjggICAgCWxvY2sgY21weGNoZyAlcnNp
LDB4MjgoJXJkaSkKICAgNjoJNzUgZGUgICAgICAgICAgICAgICAgCWpuZSAgICAweGZmZmZm
ZmZmZmZmZmZmZTYKICAgODoJNDggODkgZjAgICAgICAgICAgICAgCW1vdiAgICAlcnNpLCVy
YXgKICAgYjoJNDggMjkgYzggICAgICAgICAgICAgCXN1YiAgICAlcmN4LCVyYXgKICAgZToJ
ODQgZDIgICAgICAgICAgICAgICAgCXRlc3QgICAlZGwsJWRsCiAgMTA6CWI5ICAgICAgICAg
ICAgICAgICAgIAkuYnl0ZSAweGI5CiAgMTE6CTAxIDAwICAgICAgICAgICAgICAgIAlhZGQg
ICAgJWVheCwoJXJheCkKCS4uLgpEZWMgMDYgMTk6MjA6NTQgczYga2VybmVsOiBSU1A6IDAw
MTg6ZmZmZmM5MDAwMGMwYmI3OCBFRkxBR1M6IDAwMDEwMjk2CkRlYyAwNiAxOToyMDo1NCBz
NiBrZXJuZWw6IFJBWDogY2NjY2NjY2NjY2NjY2NjYyBSQlg6IGZmZmY4ODgxMDE5MzEyYzAg
UkNYOiBjY2NjY2NjY2NjY2NjY2NjCkRlYyAwNiAxOToyMDo1NCBzNiBrZXJuZWw6IFJEWDog
MDAwMDAwMDAwMDAwMDAwMSBSU0k6IDAwMDAwMDAxMTBmMjhmNGUgUkRJOiBmZmZmODg4MTAx
OTMxMmMwCkRlYyAwNiAxOToyMDo1NCBzNiBrZXJuZWw6IFJCUDogMDAwMDAwMDAwMDAwMDAw
MSBSMDg6IGZmZmY4ODgxMDRjYzE3NjAgUjA5OiAwMDAwMDAwMDgwMjAwMDE2CkRlYyAwNiAx
OToyMDo1NCBzNiBrZXJuZWw6IFIxMDogZmZmZjg4ODUxZjBjZWQwMCBSMTE6IGZmZmY4ODg4
YmVmZmIwMDAgUjEyOiAwMDAwMDAwMDAwMDAwMDA4CkRlYyAwNiAxOToyMDo1NCBzNiBrZXJu
ZWw6IFIxMzogMDAwMDAwMDAwMDAwMDAyOCBSMTQ6IDAwMDAwMDAwMDAwMDAwMDggUjE1OiAw
MDAwMDAwMDAwMDAwMDQ4CkRlYyAwNiAxOToyMDo1NCBzNiBrZXJuZWw6IEZTOiAgMDAwMDAw
MDAwMDAwMDAwMCgwMDAwKSBHUzpmZmZmODg4ODllZWMwMDAwKDAwMDApIGtubEdTOjAwMDAw
MDAwMDAwMDAwMDAKRGVjIDA2IDE5OjIwOjU0IHM2IGtlcm5lbDogQ1M6ICAwMDEwIERTOiAw
MDAwIEVTOiAwMDAwIENSMDogMDAwMDAwMDA4MDA1MDAzMwpEZWMgMDYgMTk6MjA6NTQgczYg
a2VybmVsOiBDUjI6IGZmZmY4ODgxMDE5MzEyZTggQ1IzOiAwMDAwMDAwMzM2MDIwMDAyIENS
NDogMDAwMDAwMDAwMDM3MDZlMApEZWMgMDYgMTk6MjA6NTQgczYga2VybmVsOiBEUjA6IDAw
MDAwMDAwMDAwMDAwMDAgRFIxOiAwMDAwMDAwMDAwMDAwMDAwIERSMjogMDAwMDAwMDAwMDAw
MDAwMApEZWMgMDYgMTk6MjA6NTQgczYga2VybmVsOiBEUjM6IDAwMDAwMDAwMDAwMDAwMDAg
RFI2OiAwMDAwMDAwMGZmZmUwZmYwIERSNzogMDAwMDAwMDAwMDAwMDQwMApEZWMgMDYgMTk6
MjA6NTQgczYga2VybmVsOiBDYWxsIFRyYWNlOgpEZWMgMDYgMTk6MjA6NTQgczYga2VybmVs
OiAgPFRBU0s+CkRlYyAwNiAxOToyMDo1NCBzNiBrZXJuZWw6ID8gX19kaWUrMHgyMy8weDcw
IApEZWMgMDYgMTk6MjA6NTQgczYga2VybmVsOiA/IHBhZ2VfZmF1bHRfb29wcysweDE3MS8w
eDRlMCAKRGVjIDA2IDE5OjIwOjU0IHM2IGtlcm5lbDogPyBleGNfcGFnZV9mYXVsdCsweDE3
NS8weDE4MCAKRGVjIDA2IDE5OjIwOjU0IHM2IGtlcm5lbDogPyBhc21fZXhjX3BhZ2VfZmF1
bHQrMHgyNi8weDMwIApEZWMgMDYgMTk6MjA6NTQgczYga2VybmVsOiA/IHVwZGF0ZV9pb190
aWNrcysweDJjLzB4NjAgCkRlYyAwNiAxOToyMDo1NCBzNiBrZXJuZWw6IGJkZXZfZW5kX2lv
X2FjY3QrMHg2My8weDE2MCAKRGVjIDA2IDE5OjIwOjU0IHM2IGtlcm5lbDogbWRfZW5kX2Ns
b25lX2lvKzB4NzUvMHhhMCBtZF9tb2QKRGVjIDA2IDE5OjIwOjU0IHM2IGtlcm5lbDogaGFu
ZGxlX3N0cmlwZV9jbGVhbl9ldmVudCsweDFlZS8weDQzMCByYWlkNDU2CkRlYyAwNiAxOToy
MDo1NCBzNiBrZXJuZWw6IGhhbmRsZV9zdHJpcGUrMHg3YjYvMHgxYWMwIHJhaWQ0NTYKRGVj
IDA2IDE5OjIwOjU0IHM2IGtlcm5lbDogaGFuZGxlX2FjdGl2ZV9zdHJpcGVzLmlzcmEuMCsw
eDM4ZC8weDU1MCByYWlkNDU2CkRlYyAwNiAxOToyMDo1NCBzNiBrZXJuZWw6IHJhaWQ1ZCsw
eDQ4OC8weDc1MCByYWlkNDU2CkRlYyAwNiAxOToyMDo1NCBzNiBrZXJuZWw6ID8gbG9ja190
aW1lcl9iYXNlKzB4NjEvMHg4MCAKRGVjIDA2IDE5OjIwOjU0IHM2IGtlcm5lbDogPyBwcmVw
YXJlX3RvX3dhaXRfZXZlbnQrMHg2MC8weDE4MCAKRGVjIDA2IDE5OjIwOjU0IHM2IGtlcm5l
bDogPyBfX3BmeF9tZF90aHJlYWQrMHgxMC8weDEwIG1kX21vZApEZWMgMDYgMTk6MjA6NTQg
czYga2VybmVsOiBtZF90aHJlYWQrMHhhYi8weDE5MCBtZF9tb2QKRGVjIDA2IDE5OjIwOjU0
IHM2IGtlcm5lbDogPyBfX3BmeF9hdXRvcmVtb3ZlX3dha2VfZnVuY3Rpb24rMHgxMC8weDEw
IApEZWMgMDYgMTk6MjA6NTQgczYga2VybmVsOiBrdGhyZWFkKzB4ZTUvMHgxMjAgCkRlYyAw
NiAxOToyMDo1NCBzNiBrZXJuZWw6ID8gX19wZnhfa3RocmVhZCsweDEwLzB4MTAgCkRlYyAw
NiAxOToyMDo1NCBzNiBrZXJuZWw6IHJldF9mcm9tX2ZvcmsrMHgzMS8weDUwIApEZWMgMDYg
MTk6MjA6NTQgczYga2VybmVsOiA/IF9fcGZ4X2t0aHJlYWQrMHgxMC8weDEwIApEZWMgMDYg
MTk6MjA6NTQgczYga2VybmVsOiByZXRfZnJvbV9mb3JrX2FzbSsweDFiLzB4MzAgCkRlYyAw
NiAxOToyMDo1NCBzNiBrZXJuZWw6ICA8L1RBU0s+CkRlYyAwNiAxOToyMDo1NCBzNiBrZXJu
ZWw6IE1vZHVsZXMgbGlua2VkIGluOiBhbGdpZl9oYXNoIGFmX2FsZyBtcHRjcF9kaWFnIHhz
a19kaWFnIHRjcF9kaWFnIHVkcF9kaWFnIHJhd19kaWFnIGluZXRfZGlhZyB1bml4X2RpYWcg
YWZfcGFja2V0X2RpYWcgbmV0bGlua19kaWFnIHJwY3NlY19nc3Nfa3JiNSBuZnN2NCBkbnNf
cmVzb2x2ZXIgbmZzIGZzY2FjaGUgbmV0ZnMgbmZ0X2N0PgpEZWMgMDYgMTk6MjA6NTQgczYg
a2VybmVsOiAgc25kX2hkYV9jb2RlYyBrdm0gc25kX2hkYV9jb3JlIGRybV9idWRkeSBzbmRf
aHdkZXAgaVRDT193ZHQgaTJjX2FsZ29fYml0IG1laV9weHAgaW50ZWxfcG1jX2J4dCBzbmRf
cGNtIG1laV9oZGNwIGVlMTAwNCBpcnFieXBhc3MgdHRtIGlUQ09fdmVuZG9yX3N1cHBvcnQg
cmFwbCBkcm1fZGlzcGxheV9oZWxwZXIgbmxzX2lzbzg4NTlfMT4KRGVjIDA2IDE5OjIwOjU0
IHM2IGtlcm5lbDogQ1IyOiBmZmZmODg4MTAxOTMxMmU4CkRlYyAwNiAxOToyMDo1NCBzNiBr
ZXJuZWw6IC0tLVsgZW5kIHRyYWNlIDAwMDAwMDAwMDAwMDAwMDAgXS0tLQoK

--------------7Ax4MydM3pHKUDdN0cmwlaxt--