Date:   Wed, 1 Feb 2023 09:43:14 -0500
From:   "Liam R. Howlett" <Liam.Howlett@Oracle.com>
To:     "Paul E. McKenney" <paulmck@kernel.org>
Cc:     Yujie Liu <yujie.liu@intel.com>, oe-lkp@lists.linux.dev,
        lkp@intel.com, linux-kernel@vger.kernel.org,
        Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org,
        linux-fsdevel@vger.kernel.org
Subject: Re: [linus:master] [maple_tree] 120b116208:
 INFO:task_blocked_for_more_than#seconds
Message-ID: <20230201144314.ukbaxy7vbgftaebr@revolver>
Mail-Followup-To: "Liam R. Howlett" <Liam.Howlett@Oracle.com>,
        "Paul E. McKenney" <paulmck@kernel.org>,
        Yujie Liu <yujie.liu@intel.com>, oe-lkp@lists.linux.dev,
        lkp@intel.com, linux-kernel@vger.kernel.org,
        Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org,
        linux-fsdevel@vger.kernel.org
References: <202301310940.4a37c7af-yujie.liu@intel.com>
 <20230131202635.GA3019407@paulmck-ThinkPad-P17-Gen-1>
 <20230131204520.ad6cf4lvtw5uf27s@revolver>
 <20230131205221.GX2948950@paulmck-ThinkPad-P17-Gen-1>
 <Y9okrO+GOYP2Gh4K@yujie-X299>
 <20230201143655.GE2948950@paulmck-ThinkPad-P17-Gen-1>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230201143655.GE2948950@paulmck-ThinkPad-P17-Gen-1>
User-Agent: NeoMutt/20220429
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?sa3hPk7YLwXqMHNhMl77FgJxc0TUVr8jmlMRV8cRI51zBGmhcROgxc9Li+fY?=
 =?us-ascii?Q?1jxpC/sl9VEA/OeRV1CbC96B0meeqfinheb2zsTs5OIYy5QXBObbwW/FKqo2?=
 =?us-ascii?Q?fyjtyBkXSUEKGaNN/4UKowLjPi+1t6Va+vRSaIZwuMvdqV0MOLRqMN0FD33V?=
 =?us-ascii?Q?7TwIXmDlXViKdteIkq84uL7whBLnSNVktaj2YU867MwKiG4RZaGWhfZoZcYc?=
 =?us-ascii?Q?rwkxmUjdYTzZgyMU9GY1SHeFKRzMKqfXAmqyiHOEHXiXAjqrqF2v+hlCl0Zy?=
 =?us-ascii?Q?Lf5voJ0VsXtfomjJRAX/b8Jw6/GTc+gOb+G31/wT/6RRBxsqjxxcrq43qY5g?=
 =?us-ascii?Q?ennnk7JqqHjoxX9yZaYctF0E9oJfr6LaC6baQFrxx48AafH8+pyMWGEmjWCd?=
 =?us-ascii?Q?+34pbsZDo7HOMVwOC5kRa6pjba8w5FTqZBz0I7eSWwFmaKcpTnagup/0bBFV?=
 =?us-ascii?Q?FVaUGqlZHICeqfFgA75SRe6WKPQT1WqF8OfNbddIe5mjwORiHxjtubtBuq+b?=
 =?us-ascii?Q?5yAJsTsFFboD7mk1Eclo8CyUkPmxSnjYJVi06bFc7ynM4XcA59+WDkBMZNs/?=
 =?us-ascii?Q?BnAYChoChrhBfoit2rJSgURXLUMDFyIrFJpvI5BGA1aR4iqeMVMD5L+EhBBF?=
 =?us-ascii?Q?vWzDbqUUEILJN/qDjskuocu2HOnoB+nYtKGtbya5iaLZCwTVSgmPpy7FapFy?=
 =?us-ascii?Q?pdprkfbz6YGIGFyEapHegmVETNrnzrehTvfByhXMkNK6gjOrRQ3420c3qeve?=
 =?us-ascii?Q?teM1mhz1Irq4ASnajaFfc0nM0Er1I0Y4WwZjtMKXYW/AOKgRv7NhZMuvjjO7?=
 =?us-ascii?Q?iH6uZWXTtj9yeLKYTAuZ7zHqgZ/pKBf0Uirqr9Vd5Eo6N23fpZIaWhtVSdkv?=
 =?us-ascii?Q?EgMUVPBNYpq8q/MC3XA3mxOCRIv5tnIU/Q96SKX3/oag7SJFjLmtnPDE6NX+?=
 =?us-ascii?Q?BwyN1RqJPiFA0Uq0dw8rJjm/T/+hGjAc60v8zLXGbzs9yaWOACOXORnOS616?=
 =?us-ascii?Q?BOS+qm+AyQvqKhU9gQqAC6tamMKR1mys238Mxth8l4X2y+zG7UzJ2mQeywif?=
 =?us-ascii?Q?BvRbx9XemIMNSzWqzhMaef+1EocoO+W1lPk2kCvvDadxnxAk/9Vo3hO7HOQQ?=
 =?us-ascii?Q?i1BuqvrmJQYbRISXSDorH5aAS6oRp/ZtVpzpDGQoymbVStkfpGZyiBQwMQ/2?=
 =?us-ascii?Q?ZVXgYb+4AfNln1b6FNzc+JACdbhdycAtQiC+4MsHhxWllqCN2wwD8PlSAoGX?=
 =?us-ascii?Q?sg/Qb743v74L94UEHFDY0GqYEJu0I2pVblk28kWbtTs4DaXHyYBt8w9Jm5Ml?=
 =?us-ascii?Q?6ieUs0NHMid6/K1u1fiXFAA5yLIiiBzGTm3IBVf3PgcAq9TcIuLEePl/hKYI?=
 =?us-ascii?Q?cYe+3TIxaAAN4I5D44DuZDtk/5GvjOWdx+5qTPzNnwqv3eV+FljSu25+h+FI?=
 =?us-ascii?Q?OssoB2KSlwYWkq29EoFXEXe/GWwQmBGd3SHaUIFx3ROlHJy8mt903+GEburN?=
 =?us-ascii?Q?jYTbsgsnx9bxzgLECOdakaZKEdb2qdgZUuT+x2i5FTGSDDA8b+zIA3pP0rin?=
 =?us-ascii?Q?2KObZELk3rXVRxs2PlCWxnGK1FZ+5Ilj0bABYjb+aqZ/zpM/uPIBaqsLJC+e?=
 =?us-ascii?Q?9A=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?rxAt6v6YWkjPueQiEaZSJPsJLLB2wp/HbcoqdAOoO5BTyr+4TLktMS9pwLI4?=
 =?us-ascii?Q?Xz2eGRCREznsLAgjulaW1LJcWI5SJaStnAGmX2aOc5SBxvL+N8x4B6gqYSSG?=
 =?us-ascii?Q?r/IWfRA7wyUGjGdZOREy+WNXCKlRInK7Ka5xwPEZdZ80RuCLBu0VIGXSzoaW?=
 =?us-ascii?Q?MMkWPQtK/Y+p8pF8uefTOdrCGnzMM1bDyiGuaWqfc7qWxi9DqqrVhd+KY2FH?=
 =?us-ascii?Q?eIW6uHhaz+hrj1o7LeA/A5pbnoRNGMApp2vLOHa5879ky6zO4XP8pDxtUj0B?=
 =?us-ascii?Q?OvWm5GISHXHLm/SPRkQ2eJBzcxg9HupZtPCjj6Uh2tqQtUpalC3OhhVFCqFn?=
 =?us-ascii?Q?Q7hjyMjdBU56oMwgJqNpKqh4oefR2Z/Rj0wJbvAL9IVf2sugkitdSXPGnIFI?=
 =?us-ascii?Q?0y0d02KyJ5nknZuJexoMV862gz0woUuhskv5BQee2ySUsjbApLX700GqSeXm?=
 =?us-ascii?Q?JeMiEu1NezyVgS0NLJo9sfjP7VFaaK0upJqIGw7VHztdhev+rlQo8p5VYmLL?=
 =?us-ascii?Q?A1QAycuQuo/8b3vckSaV1mUb+XV6CQRBX0OPPJzoVAyXdzo9876yZvX8n9h+?=
 =?us-ascii?Q?l0QKSZK1wrWorSFAnwy17FGU98HLj8Iv7zpXi6SmhAphZ/wnUbL8hAOxC+OD?=
 =?us-ascii?Q?RsyR5jAWQjO2qFH1TtUFVpoPWplQZcNZfe/TSk2py+OyP+wl8WPM1nc54+Gw?=
 =?us-ascii?Q?iG5y76X28zUjDqapTegQZwqzVSi7fcuy9hB2LrbA03qFOCU5gYJSLxOxCe8G?=
 =?us-ascii?Q?T4Qz92x8vFp3xkzocRVWHpy5cHe0hl4v4NB7jhXW3ZV0o5QVfhsw7ZfwzvUr?=
 =?us-ascii?Q?gw9t46P3rXnC4r/e7DAWIr5DlLaYbSojPmCaAuY4xUa77Cehz4f7Cv567j/7?=
 =?us-ascii?Q?Z+zAszVNOomgBssrC+JNka5wqXYsBM/Ff/efWNZUYPfLE4Gmn/bGcn5iQD9X?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6375e949-2a04-4afc-6d9b-08db0462a7c2
X-MS-Exchange-CrossTenant-AuthSource: SN6PR10MB3022.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Feb 2023 14:43:17.4337
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: usKgNW+eOH9MK4Lf7YcP7VgKQt+EzuZdxrRoSSNrQjANdWMRXTfrJ/uZ8xuWN8i2lgk4BUyuX8RRVfaSXC9TWQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB4517
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1
 definitions=2023-02-01_04,2023-01-31_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0
 suspectscore=0 mlxscore=0 spamscore=0 phishscore=0 bulkscore=0
 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2212070000 definitions=main-2302010127
X-Proofpoint-ORIG-GUID: B5-6Kw7ykIt7c3ke09_59i49uNvDOBjk
X-Proofpoint-GUID: B5-6Kw7ykIt7c3ke09_59i49uNvDOBjk
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

...

> > > > > > 
> > > > > > FYI, we noticed INFO:task_blocked_for_more_than#seconds due to commit (built with clang-14):
> > > > > > 
> > > > > > commit: 120b116208a0877227fc82e3f0df81e7a3ed4ab1 ("maple_tree: reorganize testing to restore module testing")
> > > > > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > > > > > 
> > > > > > in testcase: boot
> > > > > > 
> > > > > > on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
> > > > > > 
...

> > > > > > If you fix the issue, kindly add following tag
> > > > > > | Reported-by: kernel test robot <yujie.liu@intel.com>
> > > > > > | Link: https://lore.kernel.org/oe-lkp/202301310940.4a37c7af-yujie.liu@intel.com
> > > > > 
> > > > > Liam brought this to my attention on IRC, and it looks like the root
> > > > > cause is that the rcuscale code does not deal gracefully with grace
> > > > > periods that are in much excess of a second in duration.
> > > > > 
> > > > > Now, it might well be worth looking into why the grace periods were taking
> > > > > that long, but if you were running Maple Tree stress tests concurrently
> > > > > with rcuscale, this might well be expected behavior.
> > > > > 
> > > > 
> > > > This could be simply cpu starvation causing no foward progress in your
> > > > tests with the number of concurrent running tests and "-smp 2".
> > > > 
> > > > It's also worth noting that building in the rcu test module makes the
> > > > machine turn off once the test is complete.  This can be seen in your
> > > > console message:
> > > > [   13.254240][    T1] rcu-scale:--- Start of test: nreaders=2 nwriters=2 verbose=1 shutdown=1
> > > > 
> > > > so your machine may not have finished running through the array of tests
> > > > you have specified to build in - which is a lot.  I'm not sure if this
> > > > is the best approach considering the load that produces on the system
> > > > and how difficult it is (was) to figure out which test is causing a
> > > > stall, or other issue.
> > > 
> > > Agreed, both rcuscale and refscale when built in turn the machine off at
> > > the end of the test.  For providing background stress for some other test
> > > (in this case Maple Tree tests), rcutorture, locktorture, or scftorture
> > > might be better choices.
> > 
> > Thanks for looking into this. This is a boot test on a randconfig
> > kernel, and yes, it happend to select CONFIG_RCU_SCALE_TEST=y together
> > with CONFIG_TEST_MAPLE_TREE=y, leading to the situation in this case.

Ah, I see.  Thanks for that information, this makes more sense now.

> > 
> > I've tested the patch on same config, it does clear up the "task
> > blocked" log, though it still waits a long time at this step. The test
> > result is as follows:
> > 
> > [   18.397784][    T1] calling  maple_tree_seed+0x0/0x15d0 @ 1
> > [   18.398646][    T1]
> > [   18.398646][    T1] TEST STARTING
> > [   18.398646][    T1]
> > [ 1266.450656][    T1] maple_tree: 12610686 of 12610686 tests passed
> > [ 1266.451749][    T1] initcall maple_tree_seed+0x0/0x15d0 returned 0 after 1248053116 usecs
> > ...

Thanks.  Yes, I have a lot of tests in there that add up to taking a
while.  This is the expected output.

...

Thanks,
Liam