Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Content-Type: multipart/mixed; boundary="------------od96F16BmcMNYjWNf40wDvbg"
Message-ID: <2b6ed542-b3e0-1a87-33ac-d52fc0e0339c@oracle.com>
Date:   Tue, 12 Apr 2022 11:28:58 -0500
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.6.1
Subject: Re: [PATCH v5] locking/rwsem: Make handoff bit handling more
 consistent
Content-Language: en-US
To:     Waiman Long <longman@redhat.com>,
        chenguanyou <chenguanyou9338@gmail.com>,
        gregkh@linuxfoundation.org
Cc:     dave@stgolabs.net, hdanton@sina.com, linux-kernel@vger.kernel.org,
        mazhenhua@xiaomi.com, mingo@redhat.com, peterz@infradead.org,
        quic_aiquny@quicinc.com, will@kernel.org, sashal@kernel.org
References: <20211116012912.723980-1-longman@redhat.com>
 <20220214154741.12399-1-chenguanyou@xiaomi.com>
 <3f02975c-1a9d-be20-32cf-f1d8e3dfafcc@oracle.com>
 <e873727e-22db-3330-015d-bd6581a2937a@redhat.com>
 <31178c33-e25c-c3e8-35e2-776b5211200c@oracle.com>
 <161c2e25-3d26-4dd7-d378-d1741f7bcca8@redhat.com>
From:   john.p.donnelly@oracle.com
In-Reply-To: <161c2e25-3d26-4dd7-d378-d1741f7bcca8@redhat.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Wm8rdU4xRGIxVWRXY1FKVmNpNkZzRDRJdXJJbWh5UDk1R2U2K2NlMTd0L2FI?=
 =?utf-8?B?d1VJQXJTSTY3STBCMnJFY24rZzJiZHdCMk9VNmppYWlLcElpNytjZTk4bEpW?=
 =?utf-8?B?TzZsT2ZPbElzWnFXYkt1N2hGT0pEOEh1NmpZVWYzbFMrcS9EL1dMd0o3TmdU?=
 =?utf-8?B?S2xrQUx6cWNVL1MvREhWOGt3dExQTzVhTGVTN1FKM2wwZjlmWUNHUk51aEl4?=
 =?utf-8?B?RXp1eTdrTUlyb2FRSVhKU1NIL1AyZFJZYjZEY1ZUY01xS0dQWG15UUdlb3Jp?=
 =?utf-8?B?UlRUdHJIeGlrVWNQbG0vazVuY3g2UStTWVpoQm5adXcraXJwS25iV05Bd1Ni?=
 =?utf-8?B?OWZ0L01oU3Z6b3VieDIyVkNsSG5CTTF0OW5DNVM4dGdVdkU0TmJaS1FNZzJO?=
 =?utf-8?B?Q3pIbGFjL3dQbU1ram5Nd1hZeDhSY25JOHJhZ2JyL2czV0RqUmNBMjNLN0or?=
 =?utf-8?B?T21jbGVZZXowME1jR0ZTUnRmM0FtK1RmYlkxV2xmUUQ1S3laRElRUXdlZ2tD?=
 =?utf-8?B?L1paNkczQXlucHZyU0V0MnRTVDcwTVFjT00rK2R5cnJoWWxCblNPUElKb2Ur?=
 =?utf-8?B?aWtsVUNNalNSSFdla2Z4WktCcEd6S0g0TFF0QVMvTVBTNXJ2SlNEaWowWTdX?=
 =?utf-8?B?eEt6dCs0cVZ3RXlJc0greE9vbnd4WEFaSiszSys0cnZ1bW5iNTZJTTFxWnYv?=
 =?utf-8?B?WnhJMk9VRm5EcER6M21vTEk3dnAzUnIycjZ6Rit5K2JyMHJJVmtRMHl4dmlt?=
 =?utf-8?B?ZGlBdkNYSTJ3azJLV3NoT091Uk9hNnN1QmtlVC9ZdmVUN1VFL2E5bDRYWnEy?=
 =?utf-8?B?ZjNNaVhUQUYyRHErZ0Y0LzcyS3dhQ2ZHdys1QlpOT1Z5Y2J5dC9yWXdIc05N?=
 =?utf-8?B?SnJ6RGpnMXlJcjBobGZuOEhJT0FoQlhuWUthK29FZWpIMEZOSDRzc1hvYVZs?=
 =?utf-8?B?czIzWHY4RHBheGJXSHFrdzN6RUxLZVVtZFpGSzFNdC9hbTVMbjg5UFJmOFhB?=
 =?utf-8?B?SCs5SE5mbU9xVEZycUVoaG10MXVrWmJodEdJQ0xudnh6QjRvVTdxWkJSSjFK?=
 =?utf-8?B?U1hNek15b1JEVFF2UzJiQ0FMb01EVzNsMzJ2WGdiS214ejFqOTdNRy9XeFhy?=
 =?utf-8?B?V29oMnhmYWNFbC9CTU1UMmxmZFBEYkJWSmlxYUhab1paSUMxcG5aK3QxTzRV?=
 =?utf-8?B?ZTJmMlp0am0xUjRERWQxV1FYODhpQ242NUJCR2NveXVkMEV0NjdQejJMaGhQ?=
 =?utf-8?B?akFYalFOTXdQZXR2N3ZLRVAzTWRmTWN6NXo0eFdUNkF6c1RpR1FWSVY1Mnlt?=
 =?utf-8?B?YkZ5aG5PZysxL3plU3lHaVIxakVQRGZvZ0hKT3YyckdWMGV2OE1kcjFDNG1w?=
 =?utf-8?B?RVVJY3BRVTlPMGZtUWpGZzZzZzZnYzI2akR0ekg2b0JVYVpTMHpNaDh2Tno4?=
 =?utf-8?B?RjhmelZ5YnBoa09KUXo3dEZ5OUdZVVZrL3RyOWd4bnZYQzkzbEtQRDFXclEr?=
 =?utf-8?B?eHBBWFViYVkrOFdkN3E2aFZTUlMzU3JtdlBDNDFVLzRVS2xickxBMHJOeDA0?=
 =?utf-8?B?QThwWC9melpjWWU1SXArVGpVWEtYSzh2QVhxd2IxOHRPY0Q1dFBNYVVEUnFX?=
 =?utf-8?B?anBqb0UxYjQ0dHNOdEE2Y3RCcmdGZExoUnhVeHNldy9XOUhGTG55aWJWWlNv?=
 =?utf-8?B?cFpoQW1odU9sS0xJNVpHVDVrdzRMR2FyVGFmdlNZVkJLQUhEdmhhYVFNdDUz?=
 =?utf-8?B?dWxCVThNcHBxSDhiVVhmc0pScmRzdVVGVDlFS0NQSlZHN1hKNGtlV3JLSHpM?=
 =?utf-8?B?OFNGTW5wWGlEOTRab2lsYzNIR0VNa1F0MndmQmZPanNvMjNpQThuK0FFUXFG?=
 =?utf-8?B?VlFBK1d2RDVWcnE4RTZ5YlhzK2ZjclhKajZhSXdRdVp4MlJQVXZ1TXE0UHo0?=
 =?utf-8?B?aXNWanRvN1huZlhmVk0zVldNdU9mT05nVlUxVVNRV0YyWFNRRExaLzVGbktE?=
 =?utf-8?B?MVlXWWh1SzF5RjFOZ01yc1VDSURNUEVFR1RQNGxHWHVXM2xUaUhUbjROZ0VN?=
 =?utf-8?B?YnBnTXJMc0s3b3RvUDlHQ3M0NVU2YVpBV3QvSFVBVm04WHU0b1UzOUJUQk5q?=
 =?utf-8?B?c3VEcnNsajhHbWFOMmFGUzZPUG14Q21aTkpvalg3OHVBSjR3aHJETFEycWNv?=
 =?utf-8?B?cnM5SGhzZUpkZ3ZBeVZmZ2EvNnl5VXRrVk9ZVUhpWU9Ja2VmNVRXRE9WcWxS?=
 =?utf-8?B?aWNER2dVY25OM3l2VW5aQm1Ub3pZYUpSWHhtUnZyMHNZNmVna3FyYmdabXVx?=
 =?utf-8?B?ZlNFci9jOVNiZDBlcjQwTGNGN2d4WjJuaWxJTjEwZkRlV3ltd3FsVGllRXFa?=
 =?utf-8?Q?23B3neHmQVcsOgwM=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5782c7c6-5fbc-4bbf-a46a-08da1ca191ce
X-MS-Exchange-CrossTenant-AuthSource: CO1PR10MB4722.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Apr 2022 16:29:08.9616
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: DYSvekDwgfecnrVoa6zOeZi59Hr0lYeAt2l7Qbgp627q5o0mG2LkDND/8KlgdovvKMt1G6YqpJ8MpatZvYTANAurJdsPQHQNoibl7QIoC8c=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW5PR10MB5665
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858
 definitions=2022-04-12_06:2022-04-11,2022-04-12 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0
 malwarescore=0 mlxlogscore=999 spamscore=0 adultscore=0 phishscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2202240000 definitions=main-2204120078
X-Proofpoint-ORIG-GUID: pIvj8nXrU4Q5JaX0qW80hHLERtrD5iLI
X-Proofpoint-GUID: pIvj8nXrU4Q5JaX0qW80hHLERtrD5iLI
X-Spam-Status: No, score=-3.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

--------------od96F16BmcMNYjWNf40wDvbg
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

On 4/11/22 4:07 PM, Waiman Long wrote:
> 
> On 4/11/22 17:03, john.p.donnelly@oracle.com wrote:
>>
>>>>
>>>> I have reached out to Waiman and he suggested this for our next test 
>>>> pass:
>>>>
>>>>
>>>> 1ee326196c6658 locking/rwsem: Always try to wake waiters in 
>>>> out_nolock path
>>>
>>> Does this commit help to avoid the lockup problem?
>>>
>>> Commit 1ee326196c6658 fixes a potential missed wakeup problem when a 
>>> reader first in the wait queue is interrupted out without acquiring 
>>> the lock. It is actually not a fix for commit d257cc8cb8d5. However, 
>>> this commit changes the out_nolock path behavior of writers by 
>>> leaving the handoff bit set when the wait queue isn't empty. That 
>>> likely makes the missed wakeup problem easier to reproduce.
>>>
>>> Cheers,
>>> Longman
>>>
>>
>> Hi,
>>
>>
>> We are testing now
>>
>> ETA for fio soak test completion is  ~15hr from now.
>>
>> I wanted to share the stack traces for future reference + occurrences.
>>
> I am looking forward to your testing results tomorrow.
> 
> Cheers,
> Longman
> 
Hi

  Our 24hr fio soak test with :

  1ee326196c6658 locking/rwsem: Always try to wake waiters in out_nolock 
path


  applied to 5.15.30  passed.

  I suggest you append  1ee326196c6658 with :


  cc: stable

   Fixes: d257cc8cb8d5 ("locking/rwsem: Make handoff bit handling more 
consistent")


I'll leave the implementation details up to the core maintainers how to 
do that ;-)

...

Thank you

John.

--------------od96F16BmcMNYjWNf40wDvbg
Content-Type: message/rfc822; name="[PATCH v5] locking/rwsem: Make handoff bit
 handling more consistent.eml"
Content-Disposition: attachment; filename*0="[PATCH v5] locking/rwsem: Make
 handoff bit handling more con"; filename*1="sistent.eml"
Content-Transfer-Encoding: 8bit

Received: from SN6PR10MB2976.namprd10.prod.outlook.com (2603:10b6:805:d6::12)
 by CO1PR10MB4722.namprd10.prod.outlook.com with HTTPS; Mon, 11 Apr 2022
 21:03:51 +0000
Authentication-Results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=oracle.com;
Received: from CO1PR10MB4722.namprd10.prod.outlook.com (2603:10b6:303:9e::12)
 by SN6PR10MB2976.namprd10.prod.outlook.com (2603:10b6:805:d6::12) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5144.29; Mon, 11 Apr
 2022 21:03:50 +0000
Received: from CO1PR10MB4722.namprd10.prod.outlook.com
 ([fe80::4427:92a2:50a5:50b6]) by CO1PR10MB4722.namprd10.prod.outlook.com
 ([fe80::4427:92a2:50a5:50b6%6]) with mapi id 15.20.5144.029; Mon, 11 Apr 2022
 21:03:50 +0000
Message-ID: <31178c33-e25c-c3e8-35e2-776b5211200c@oracle.com>
Date: Mon, 11 Apr 2022 16:03:40 -0500
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.6.1
Subject: Re: [PATCH v5] locking/rwsem: Make handoff bit handling more
 consistent
Content-Language: en-US
To: Waiman Long <longman@redhat.com>, chenguanyou
 <chenguanyou9338@gmail.com>, gregkh@linuxfoundation.org
Cc: dave@stgolabs.net, hdanton@sina.com, linux-kernel@vger.kernel.org,
 mazhenhua@xiaomi.com, mingo@redhat.com, peterz@infradead.org,
 quic_aiquny@quicinc.com, will@kernel.org, sashal@kernel.org,
 John Donnelly <john.p.donnelly@oracle.com>
References: <20211116012912.723980-1-longman@redhat.com>
 <20220214154741.12399-1-chenguanyou@xiaomi.com>
 <3f02975c-1a9d-be20-32cf-f1d8e3dfafcc@oracle.com>
 <e873727e-22db-3330-015d-bd6581a2937a@redhat.com>
From: john.p.donnelly@oracle.com
In-Reply-To: <e873727e-22db-3330-015d-bd6581a2937a@redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-MS-Exchange-Organization-ExpirationStartTime: 11 Apr 2022 21:03:50.3769
 (UTC)
X-MS-Exchange-Organization-ExpirationStartTimeReason: OriginalSubmit
X-MS-Exchange-Organization-ExpirationInterval: 1:00:00:00.0000000
X-MS-Exchange-Organization-ExpirationIntervalReason: OriginalSubmit
X-MS-Exchange-Organization-Network-Message-Id: 
 f24d91a9-2ff3-4b28-83b6-08da1bfec72f
X-MS-Exchange-Organization-AuthSource: CO1PR10MB4722.namprd10.prod.outlook.com
X-MS-Exchange-Organization-AuthAs: Internal
X-MS-Exchange-Organization-AuthMechanism: 14
X-ClientProxiedBy: DM6PR01CA0011.prod.exchangelabs.com (2603:10b6:5:296::16)
 To CO1PR10MB4722.namprd10.prod.outlook.com (2603:10b6:303:9e::12)
X-MS-Exchange-Organization-MessageDirectionality: Originating
Return-Path: john.p.donnelly@oracle.com
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: f24d91a9-2ff3-4b28-83b6-08da1bfec72f
X-MS-TrafficTypeDiagnostic: SN6PR10MB2976:EE_
X-Oracle-Tenancy: 1
X-MS-Exchange-Organization-SCL: -1
X-Microsoft-Antispam: BCL:0;
X-Forefront-Antispam-Report: 
 CIP:255.255.255.255;CTRY:;LANG:en;SCL:-1;SRV:;IPV:NLI;SFV:SKI;H:CO1PR10MB4722.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:;DIR:INB;
X-MS-Exchange-CrossTenant-Network-Message-Id: f24d91a9-2ff3-4b28-83b6-08da1bfec72f
X-MS-Exchange-CrossTenant-AuthSource: CO1PR10MB4722.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Apr 2022 21:03:50.5019
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: tcelX3jxfFDN045gruLrZ2dACptl396b4YXHtueSzxP2ZSdRB1JVqdtcjlyp24kTEwcFJInw++LhoR5T3dZTikO7hj3I2IQmIvfJUCVfIuI=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB2976
X-MS-Exchange-Transport-EndToEndLatency: 00:00:01.1239883
X-MS-Exchange-Processed-By-BccFoldering: 15.20.5144.029
X-Microsoft-Antispam-Mailbox-Delivery: 
 ucf:1;jmr:0;auth:0;dest:C;OFR:CustomRules;ENG:(910001)(944506458)(944626604)(920097)(425001)(930097);
X-Microsoft-Antispam-Message-Info: 
 AbvWkKWJpj1NTrW84R+PCJyLV1AzCqa9QzLNCPK0UmjgIkqGaisWYtx/P5zMCUWRM+ui8NdiySYW5x3Ebf7zyZViG8Do0QlmPrgsgtejHDoE/wHWnfbUfRuFjZd0gWZptxYxH87G2TY6GfERsU/DjkKnbXl1LvqlCABh0u0g2yTxRcAZIORx2ZNUdTvBgQK2bGRgEH5CXFKGC8KnLzxoMkZ2lH9E9P5c8rorz41zmuNS/l9Si181uOn2kOwFwQJI0u12AL5MNNMEn/GrYBKC4bJPTpj/nG9cid9GvwpzF23i2xVhYQ4kYpE39eH49STpnxAU9/2x3f647jKD8ZnTFXphOyqABq8TeYZBRJjGXQr1DZjDRa6+jk2940w/+YCHfBkICvjyk6E9j4603/AY/C76lY103yFaRgJo0o9kUT+SQjj6AUheBABkbqm9gn3QkraoFRyX3ji3ixrm4CBY7/U+HNzfZnCS7SHd/FJY9Uv5vzmymB7bwvarevNvzN5v1Zi7ev4y4M485tl+iBN869VX6j0sAEAAQn/Fh4lJTAudLTEfdonRIBBFG9Q3lpfUNXr+6K6C+r/2XiGhhCnCiEzX96aRgUsDi9qxxZlVFJfYGLCm0sDJIgjzn1tP03FytgcRwvMPUpzpBph0zSIbLWhAzekHPQDfXKWY/ixSkmHrhg7uLztYJFWIGCo6c6+0mDp2lmQeMhfcat9N5w0/ZGkv0ZscVtOsLsrshBBPBNY=
MIME-Version: 1.0

On 4/11/22 1:40 PM, Waiman Long wrote:
> On 4/11/22 14:26, john.p.donnelly@oracle.com wrote:
>> On 2/14/22 9:47 AM, chenguanyou wrote:
>>> Hi Waiman, Greg,
>>> This patch has been merged in branch linux-5.16.y.
>>> Can we take it to the linux-5.10.y LTS version?
>>>
>>> thanks,
>>
>> Hi,
>>
>> As a FYI:
>>
>> We have observed that following lockup with this commit added to 
>> 5.15.LTS:
>>
>> d257cc8cb8d5 - locking/rwsem: Make handoff bit handling more 
>> consistent (4 months ago) <Waiman Long>
>>
>> The "fio" test suit fails with LVM devices composed of four NVME 
>> devices with these observed lockup, panic.
>>
>>
>>
>> ext4:
>>
>> PID: 3682   TASK: ffff8f489ae34bc0  CPU: 2   COMMAND: "dio/dm-0"
>>  #0 [fffffe0000083e50] crash_nmi_callback at ffffffff828772b3
>>  #1 [fffffe0000083e58] nmi_handle at ffffffff82840778
>>  #2 [fffffe0000083ea0] default_do_nmi at ffffffff8337a1e2
>>  #3 [fffffe0000083ec8] exc_nmi at ffffffff8337a48d
>>  #4 [fffffe0000083ef0] end_repeat_nmi at ffffffff8340153b
>>     [exception RIP: _raw_spin_lock_irq+23]
>>     RIP: ffffffff8338b2e7  RSP: ffff9c4409b47c78  RFLAGS: 00000046
>>     RAX: 0000000000000000  RBX: ffff8f489ae34bc0  RCX: 0000000000000000
>>     RDX: 0000000000000001  RSI: 0000000000000000  RDI: ffff8f47f7b90104
>>     RBP: ffff9c4409b47d20   R8: 0000000000000000   R9: 0000000000000000
>>     R10: 0000000000000000  R11: 0000000000000000  R12: ffff8f47f7b90104
>>     R13: ffff9c4409b47cb0  R14: ffff8f47f7b900f0  R15: 0000000000000000
>>     ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
>>     <NMI exception stack> ---
>>  #5 [ffff9c4409b47c78] _raw_spin_lock_irq at ffffffff8338b2e7
>>  #6 [ffff9c4409b47c78] rwsem_down_write_slowpath at ffffffff82925be9
>>  #7 [ffff9c4409b47d28] ext4_map_blocks at ffffffffc11c26dc [ext4]
>>  #8 [ffff9c4409b47d98] ext4_convert_unwritten_extents at
>> ffffffffc11ad9e0 [ext4]
>>  #9 [ffff9c4409b47df0] ext4_dio_write_end_io at ffffffffc11b22aa [ext4]
>>
>> xfs:
>>
>> PID: 3719   TASK: ffff9f81d2d74bc0  CPU: 37  COMMAND: "dio/dm-0"
>>  #0 [fffffe0000894e50] crash_nmi_callback at ffffffffad6772b3
>>  #1 [fffffe0000894e58] nmi_handle at ffffffffad640778
>>  #2 [fffffe0000894ea0] default_do_nmi at ffffffffae17a1e2
>>  #3 [fffffe0000894ec8] exc_nmi at ffffffffae17a48d
>>  #4 [fffffe0000894ef0] end_repeat_nmi at ffffffffae20153b
>>     [exception RIP: _raw_spin_lock_irq+23]
>>     RIP: ffffffffae18b2e7  RSP: ffffbb7ec9637c48  RFLAGS: 00000046
>>     RAX: 0000000000000000  RBX: ffff9f81d2d74bc0  RCX: 0000000000000000
>>     RDX: 0000000000000001  RSI: 0000000000000000  RDI: ffff9f81c04a918c
>>     RBP: ffffbb7ec9637ce8   R8: 0000000000000000   R9: 0000000000000000
>>     R10: 0000000000000000  R11: 0000000000000000  R12: ffff9f81c04a918c
>>     R13: ffffbb7ec9637c80  R14: ffff9f81c04a9178  R15: 0000000000000000
>>     ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
>>     <NMI exception stack> ---
>>  #5 [ffffbb7ec9637c48] _raw_spin_lock_irq at ffffffffae18b2e7
>>  #6 [ffffbb7ec9637c48] rwsem_down_write_slowpath at ffffffffad725be9
>>  #7 [ffffbb7ec9637cf0] xfs_trans_alloc_inode at ffffffffc074f2bd [xfs]
>>  #8 [ffffbb7ec9637d50] xfs_iomap_write_unwritten at ffffffffc073ad15
>> [xfs]
>>  #9 [ffffbb7ec9637dd0] xfs_dio_write_end_io at ffffffffc072db62 [xfs]
>>
>>
>> I have reached out to Waiman and he suggested this for our next test 
>> pass:
>>
>>
>> 1ee326196c6658 locking/rwsem: Always try to wake waiters in out_nolock 
>> path
> 
> Does this commit help to avoid the lockup problem?
> 
> Commit 1ee326196c6658 fixes a potential missed wakeup problem when a 
> reader first in the wait queue is interrupted out without acquiring the 
> lock. It is actually not a fix for commit d257cc8cb8d5. However, this 
> commit changes the out_nolock path behavior of writers by leaving the 
> handoff bit set when the wait queue isn't empty. That likely makes the 
> missed wakeup problem easier to reproduce.
> 
> Cheers,
> Longman
> 

Hi,


We are testing now

ETA for fio soak test completion is  ~15hr from now.

I wanted to share the stack traces for future reference + occurrences.


Cheers.

JD


...


--------------od96F16BmcMNYjWNf40wDvbg--