Received-SPF: pass (google.com: domain of linux-kernel+bounces-69671-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
References: <20240213055554.1802415-1-ankur.a.arora@oracle.com>
 <a7e785f8-d4c3-4b1b-9abe-36ac0b971e44@paulmck-laptop>
 <87le7mpjpr.fsf@oracle.com>
 <4e070ae0-29dc-41ee-aee6-0d3670304825@paulmck-laptop>
 <0d4a4eec-ce91-48da-91b6-1708a97edaeb@paulmck-laptop>
 <871q9dmndg.fsf@oracle.com>
 <9916c73f-510c-47a6-a9b4-ea6b438e82c0@paulmck-laptop>
 <87le7lkzj6.fsf@oracle.com>
 <4bc4ea06-e3e9-4d22-bacf-71cae0ba673d@paulmck-laptop>
 <0be4df28-99be-41a3-9e24-2b7cfc740b4a@paulmck-laptop>
User-agent: mu4e 1.4.10; emacs 27.2
From: Ankur Arora <ankur.a.arora@oracle.com>
To: paulmck@kernel.org
Cc: Ankur Arora <ankur.a.arora@oracle.com>, linux-kernel@vger.kernel.org,
        tglx@linutronix.de, peterz@infradead.org,
        torvalds@linux-foundation.org, akpm@linux-foundation.org,
        luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com,
        hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com,
        vincent.guittot@linaro.org, willy@infradead.org, mgorman@suse.de,
        jpoimboe@kernel.org, mark.rutland@arm.com, jgross@suse.com,
        andrew.cooper3@citrix.com, bristot@kernel.org,
        mathieu.desnoyers@efficios.com, glaubitz@physik.fu-berlin.de,
        anton.ivanov@cambridgegreys.com, mattst88@gmail.com,
        krypton@ulrich-teichert.org, rostedt@goodmis.org,
        David.Laight@aculab.com, richard@nod.at, jon.grimm@amd.com,
        bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com
Subject: Re: [PATCH 00/30] PREEMPT_AUTO: support lazy rescheduling
In-reply-to: <0be4df28-99be-41a3-9e24-2b7cfc740b4a@paulmck-laptop>
Date: Fri, 16 Feb 2024 19:59:45 -0800
Message-ID: <87r0hbkafi.fsf@oracle.com>
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?rF5KFuSYyz0mfRQdkJ2ZYgRFUpe4DYxuW3ZBJkXA6SgGEh0/uobfN1qyTPEY?=
 =?us-ascii?Q?aEpOaWPz08/Br2NTlcDc89Lxzobg8h+EepPjBHC1ATiJguKsAQwHrUhRou0j?=
 =?us-ascii?Q?3zzXxvZMCIYe0omomNVVdHwKi0SZF3RgczuBT9+l/tpvf8urjBruj3qJVx4G?=
 =?us-ascii?Q?EXlHHJzwTO0ZGVvURP28wmobcCNPn8TsxyIXkglgUjBpEuGaopU3/44x7URS?=
 =?us-ascii?Q?eU/Drt6jBmXDbQtjBguDU+dQ1u/LIIxlK3rXJd9L5zCBhtKkFBiy+i0xlBac?=
 =?us-ascii?Q?jqlaxsCp8YyJftYPNS8cgeHKqweo5kF0mMm2cg+wcQNsEfmXWOguvIwTpAAs?=
 =?us-ascii?Q?3RH1OIvCOex8B7/FDpr1POg0otEs5LkUr0NbQRXKCSTidX1oXwe+8mxVxjwo?=
 =?us-ascii?Q?KqwnwTojEpvsOv5JuNWB7Ibl0bsy204aybLB13LruEXVUmm5oCkQL6QW4CGB?=
 =?us-ascii?Q?x3Un526JZMYMzgszXpbv+3rak6dGAY4QQkgdPc4o67PwDxN9LVbjCfJXEVgG?=
 =?us-ascii?Q?FFTFv4pmBCNhvKPozb/el9h2rKQzvxetTuAURkGxDPoSYP8ZtEbPs4vXD8UG?=
 =?us-ascii?Q?urJFWoLGN41jFTYfxBvff7aJxF1iBNi6zDk1EJebM+xTrTYU1OshZbol+D9t?=
 =?us-ascii?Q?Bn0PqeNa17QNQ+tiVX9K1Qp4My29vwijHdVTxyXBLf04hufncw/xMbTxjJEl?=
 =?us-ascii?Q?CW1DJ/j3VmWCc9sPhyOnpZSlSCvBqXcSIfhpfPQI3y50Wm7vLFYOJmvg5Ja8?=
 =?us-ascii?Q?7Xpk1SekIqUTbviRF+4gqUll0xnabahbpkCSnTn+Px/nngxPLGYQ5R5/mKDk?=
 =?us-ascii?Q?az+vPHZhqyzZAOZB57+RXRcwuxRiXEDr7O/4Ai3wq+3HTtSHqLZMXncrDH2B?=
 =?us-ascii?Q?uXJIJLK3IUJSlznx4m6JOJx0Nv67fK7+qW4C3uomU1zd60Csp0rPJ5Ru87UB?=
 =?us-ascii?Q?oHwA9B0JR2efpUqVkuD52uNNIv2M2AxokYAIikCv0F9jYFvuwHGZ13iqAL/O?=
 =?us-ascii?Q?/pEwHYgGqiV+3jJOxy67cBBx3EDDkiLvADQjpj6MagxlTG+r51AdHFHbv7dX?=
 =?us-ascii?Q?iERJnh68iy93DQQqMHfc9NtViQaIh0x7QE09WwazOz9ODGojT/rMm/uG36KV?=
 =?us-ascii?Q?KCbWE0R5sYh5dbw/DwMmWUG/J/+41S6JK64rIyi3cWP/zzNyN5LOetvva23j?=
 =?us-ascii?Q?HowzhQGWWuIl8SVIxQzXyi2ekgfOkeWnfvhdjDZnsihpyFsupkrFWgeqRNrP?=
 =?us-ascii?Q?fZFDOb/FngRWlDLGDalRoelJEmolO4JmTbYL3IC64YVixCKCPlwuGB3/LAI5?=
 =?us-ascii?Q?SuP8dFFNKD2oWN2JR7XgFBGpeLwgkWCmIT8peR7OOdbIy6USZ3uljzCYK2G2?=
 =?us-ascii?Q?KStbOJb2p0/xuW9GbKRdkCjhL4bKE5UjDJuCS2df3KObjYIQBrOkLMrN1Rg7?=
 =?us-ascii?Q?djZeineRPRAdFRNYlTAX570OqnAkzobsTnbCOUa9yV+96uaqbcVqfGp0yjmt?=
 =?us-ascii?Q?qZyxawnfQnD9dON6ZS8CYN1x65zBGBnb7rX5x56vlKt54tfkCk3biFC/zwuw?=
 =?us-ascii?Q?caqK/2b2qr5K/xdMtIHTiFUiXzmuUoF7MM7Qx8Jp1BBFbiBIh5BxZ2Mm58OC?=
 =?us-ascii?Q?Yw=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	Hf+ifcFcjTMxRjbfez6KKgH+KH+aWgNrG35AK73mgbtty8ilqceKDjBgKN+NoLU2Jt5wWXKHCOQx1sNscrgI3zK86Y7vHZaHMY7r2MAPrzAZO6wZQjX0IWJGVqvHs6MgRxVgTIyfGXd1+ok3EQeH0gFYBbxNI+e2PL7UqNHKcHBYUPkm1MN8AWeuMYcKFTi3lIRE23LcZ7VVFDGeh/Kcw/em5ACp6NGmCAOEVWSPzZP2stQXt1zwtv8VhxL0FWIu3Ybb+0TvCcVVB6BGQhGlDRgM8ZKPw3a4GClpYmfsS/w9jyUA5ouAOeWPIT6XNLU2Hyxd8AeCfkfE3j112HJw8DMw7YmGeLA6I1E54oTfSNl4pZTEzToXfCW+5ef0rDk/I95zR4PlNALRzonLFk9kHcCNyoPuLSKmhvhgaBVPbw9GbyyAKmD3q+7UOcY2xNfUJJ23WitVMpzAPeTY1nmAuZKajFtB7XaDW9f/Wa21q84z0+OLODn48VNEmF3Fh714rVD/E5G0OdI1TAJtLDjwLD30o2fccXNA/SjWBFVTKmHjh0QJqhdUYvyyqglE6U2bmxmX6NjdV4UQuvnmYmMElTQ8StLfSLNHtSAxBz/qQ4k=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 13493258-729e-4c5d-e244-08dc2f6cd4ab
X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Feb 2024 03:59:25.2740
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: dSFEvgTSo6cg88OQsRZtZXzOHxlLqfO/yoBpxUtsrKe0wxHPwNY/Hr/5baZq5dP9h0/eE3Nlff5VQOm7X8LUpSdd5kKTLka93RarUwyLBjY=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR10MB7742
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-02-17_01,2024-02-16_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 phishscore=0
 adultscore=0 mlxlogscore=999 mlxscore=0 bulkscore=0 malwarescore=0
 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2402170028
X-Proofpoint-GUID: -mgPLSHMbWkVD5CGizagaLNIR7Ut73NG
X-Proofpoint-ORIG-GUID: -mgPLSHMbWkVD5CGizagaLNIR7Ut73NG


Paul E. McKenney <paulmck@kernel.org> writes:

> On Thu, Feb 15, 2024 at 06:59:25PM -0800, Paul E. McKenney wrote:
>> On Thu, Feb 15, 2024 at 04:45:17PM -0800, Ankur Arora wrote:
>> >
>> > Paul E. McKenney <paulmck@kernel.org> writes:
>> >
>> > > On Thu, Feb 15, 2024 at 01:24:59PM -0800, Ankur Arora wrote:
>> > >>
>> > >> Paul E. McKenney <paulmck@kernel.org> writes:
>> > >>
>> > >> > On Wed, Feb 14, 2024 at 07:45:18PM -0800, Paul E. McKenney wrote:
>> > >> >> On Wed, Feb 14, 2024 at 06:03:28PM -0800, Ankur Arora wrote:
>> > >> >> >
>> > >> >> > Paul E. McKenney <paulmck@kernel.org> writes:
>> > >> >> >
>> > >> >> > > On Mon, Feb 12, 2024 at 09:55:24PM -0800, Ankur Arora wrote:
>> > >> >> > >> Hi,
>> > >> >> > >>
>> > >> >> > >> This series adds a new scheduling model PREEMPT_AUTO, which like
>> > >> >> > >> PREEMPT_DYNAMIC allows dynamic switching between a none/voluntary/full
>> > >> >> > >> preemption model. However, unlike PREEMPT_DYNAMIC, it doesn't depend
>> > >> >> > >> on explicit preemption points for the voluntary models.
>> > >> >> > >>
>> > >> >> > >> The series is based on Thomas' original proposal which he outlined
>> > >> >> > >> in [1], [2] and in his PoC [3].
>> > >> >> > >>
>> > >> >> > >> An earlier RFC version is at [4].
>> > >> >> > >
>> > >> >> > > This uncovered a couple of latent bugs in RCU due to its having been
>> > >> >> > > a good long time since anyone built a !SMP preemptible kernel with
>> > >> >> > > non-preemptible RCU.  I have a couple of fixes queued on -rcu [1], most
>> > >> >> > > likely for the merge window after next, but let me know if you need
>> > >> >> > > them sooner.
>> > >> >> >
>> > >> >> > Thanks. As you can probably tell, I skipped out on !SMP in my testing.
>> > >> >> > But, the attached diff should tide me over until the fixes are in.
>> > >> >>
>> > >> >> That was indeed my guess.  ;-)
>> > >> >>
>> > >> >> > > I am also seeing OOM conditions during rcutorture testing of callback
>> > >> >> > > flooding, but I am still looking into this.
>> > >> >> >
>> > >> >> > That's on the PREEMPT_AUTO && PREEMPT_VOLUNTARY configuration?
>> > >> >>
>> > >> >> On two of the PREEMPT_AUTO && PREEMPT_NONE configurations, but only on
>> > >> >> two of them thus far.  I am running a longer test to see if this might
>> > >> >> be just luck.  If not, I look to see what rcutorture scenarios TREE10
>> > >> >> and TRACE01 have in common.
>> > >> >
>> > >> > And still TRACE01 and TREE10 are hitting OOMs, still not seeing what
>> > >> > sets them apart.  I also hit a grace-period hang in TREE04, which does
>> > >> > CONFIG_PREEMPT_VOLUNTARY=y along with CONFIG_PREEMPT_AUTO=y.  Something
>> > >> > to dig into more.
>> > >>
>> > >> So, the only PREEMPT_VOLUNTARY=y configuration is TREE04. I wonder
>> > >> if you would continue to hit the TREE04 hang with CONFIG_PREEMTP_NONE=y
>> > >> as well?
>> > >> (Just in the interest of minimizing configurations.)
>> > >
>> > > I would be happy to, but in the spirit of full disclosure...
>> > >
>> > > First, I have seen that failure only once, which is not enough to
>> > > conclude that it has much to do with TREE04.  It might simply be low
>> > > probability, so that TREE04 simply was unlucky enough to hit it first.
>> > > In contrast, I have sufficient data to be reasonably confident that the
>> > > callback-flooding OOMs really do have something to do with the TRACE01 and
>> > > TREE10 scenarios, even though I am not yet seeing what these two scenarios
>> > > have in common that they don't also have in common with other scenarios.
>> > > But what is life without a bit of mystery?  ;-)
>> >
>> > :).
>> >
>> > > Second, please see the attached tarball, which contains .csv files showing
>> > > Kconfig options and kernel boot parameters for the various torture tests.
>> > > The portions of the filenames preceding the "config.csv" correspond to
>> > > the directories in tools/testing/selftests/rcutorture/configs.
>> >
>> > So, at least some of the HZ_FULL=y tests don't run into problems.
>> >
>> > > Third, there are additional scenarios hand-crafted by the script at
>> > > tools/testing/selftests/rcutorture/bin/torture.sh.  Thus far, none of
>> > > them have triggered, other than via the newly increased difficulty
>> > > of configurating a tracing-free kernel with which to test, but they
>> > > can still be useful in ruling out particular Kconfig options or kernel
>> > > boot parameters being related to a given issue.
>> > >
>> > > But please do take a look at the .csv files and let me know what
>> > > adjustments would be appropriate given the failure information.
>> >
>> > Nothing stands out just yet. Let me start a run here and see if
>> > that gives me some ideas.
>>
>> Sounds good, thank you!
>>
>> > I'm guessing the splats don't give any useful information or
>> > you would have attached them ;).
>>
>> My plan is to extract what can be extracted from the overnight run
>> that I just started.  Just in case the fixes have any effect on things,
>> unlikely though that might be given those fixes and the runs that failed.
>
> And I only got no failures from either TREE10 or TRACE01 on last night's
> run.

Oh that's great news. Same for my overnight runs for TREE04 and TRACE01.

Ongoing: a 24 hour run for those. Let's see how that goes.

> I merged your series on top of v6.8-rc4 with the -rcu tree's
> dev branch, the latter to get the RCU fixes.  But this means that last
> night's results are not really comparable to earlier results.
>
> I did get a few TREE09 failures, but I get those anyway.  I took it
> apart below for you because I got confused and thought that it was a
> TREE10 failure.  So just in case you were curious what one of these
> looks like and because I am too lazy to delete it.  ;-)

Heh. Well, thanks for being lazy /after/ dissecting it nicely.

> So from the viewpoint of moderate rcutorture testing, this series
> looks good.  Woo hoo!!!

Awesome!

> We did uncover a separate issue with Tasks RCU, which I will report on
> in more detail separately.  However, this issue does not (repeat, *not*)
> affect lazy preemption as such, but instead any attempt to remove all
> of the cond_resched() invocations.

So, that sounds like it happens even with (CONFIG_PREEMPT_AUTO=n,
CONFIG_PREEMPT=y)?
Anyway will look out for it when you go into the detail.

> My next step is to try this on bare metal on a system configured as
> is the fleet.  But good progress for a week!!!

Yeah this is great. Fingers crossed for the wider set of tests.

Thanks

--
ankur