Received-SPF: pass (google.com: domain of linux-kernel+bounces-68965-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Fri, 16 Feb 2024 11:33:18 -0500
From: "Liam R. Howlett" <Liam.Howlett@Oracle.com>
To: Jan Kara <jack@suse.cz>
Cc: Chuck Lever <cel@kernel.org>, viro@zeniv.linux.org.uk, brauner@kernel.org,
        hughd@google.com, akpm@linux-foundation.org, oliver.sang@intel.com,
        feng.tang@intel.com, linux-kernel@vger.kernel.org,
        linux-fsdevel@vger.kernel.org, maple-tree@lists.infradead.org,
        linux-mm@kvack.org, lkp@intel.com
Subject: Re: [PATCH RFC 7/7] libfs: Re-arrange locking in offset_iterate_dir()
Message-ID: <20240216163318.w66ywrhpr5at46pi@revolver>
Mail-Followup-To: "Liam R. Howlett" <Liam.Howlett@Oracle.com>,
	Jan Kara <jack@suse.cz>, Chuck Lever <cel@kernel.org>,
	viro@zeniv.linux.org.uk, brauner@kernel.org, hughd@google.com,
	akpm@linux-foundation.org, oliver.sang@intel.com,
	feng.tang@intel.com, linux-kernel@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, maple-tree@lists.infradead.org,
	linux-mm@kvack.org, lkp@intel.com
References: <170785993027.11135.8830043889278631735.stgit@91.116.238.104.host.secureserver.net>
 <170786028847.11135.14775608389430603086.stgit@91.116.238.104.host.secureserver.net>
 <20240215131638.cxipaxanhidb3pev@quack3>
 <20240215170008.22eisfyzumn5pw3f@revolver>
 <20240215171622.gsbjbjz6vau3emkh@quack3>
 <20240215210742.grjwdqdypvgrpwih@revolver>
 <20240216101546.xjcpzyb3pgf2eqm4@quack3>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240216101546.xjcpzyb3pgf2eqm4@quack3>
User-Agent: NeoMutt/20220429
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?geNzMG0vp1JKMq0jn9jIkZnU73t1yMHklH53/o0nHJXJnpEG3PkBxmrqmZfl?=
 =?us-ascii?Q?IN4+2c3nSYYUIb+/a28TuHV4EmFQw6ckZTcbhKRDEHN6VmclDNaz2P/S0NOa?=
 =?us-ascii?Q?kvCo5Rid2t6Z0nJiz1UL3A8ITWkcb1xcTd5JgupxOLe8Qbt5kj0wWaITSsl/?=
 =?us-ascii?Q?nRw2Wu1KVZWC9GccQGu87ZyXS/D7OSRCVk/XzS6tTr4aWr7RcoV2awELERsl?=
 =?us-ascii?Q?RjIK0S5BdrI25z18O/6inMHQGknjf4S12a29/YqpsUAOxzNp+5+G1o4ZOZzh?=
 =?us-ascii?Q?AIKp464d7hJPtM8EyGZDdBUueNeDqo7NsIR9/mThBSbqjQn1EH54S6y5zeLP?=
 =?us-ascii?Q?q5Tf6DbSTf/1TTD293HglB4XeAsP2IsRLoLG7oJtYbF7tveDBH+H61sIEd9N?=
 =?us-ascii?Q?Do4Idgvs0QSQtOYjPgxRbE5ozu0BVDPMflJyTGe+zpHfvQ/ydhahQK/9C6ss?=
 =?us-ascii?Q?mgmNAolCy16sZrkjoJ8tBEd2RuK7mv7wtZpLiaaTW5I/UGAP5VlTagJuQdML?=
 =?us-ascii?Q?DfwR5AFINF8zsIsFfkimuMTetyUQ4ew+RMeI6oBp2JBrDiCBZb+2cdHVjTQ2?=
 =?us-ascii?Q?IlcWkSr1ptFCCQICY0JnM1nkqwTsNRX8zUedaXI+jLJQWJT8Zm51rULcoDjM?=
 =?us-ascii?Q?ZqAO/SgNLPabgoxgJU5XxvCIWrKeoInD6ja0d5sAA1CFThjSGXNT+v6wZWxy?=
 =?us-ascii?Q?0h6YW1CB8KAFPAbx8NV+Irdv6zJcELSKzoT+uHZ8iZkIvsl2a/QyIN0GZ/hs?=
 =?us-ascii?Q?f9ORxEVNmtofpORFNetdGkV73y3noptbhCtaQcxYmD0qycx8sHJLIkcElRY5?=
 =?us-ascii?Q?17LzJ/5ICeZCmvIISGGEvqoPAOCzJASkx5N9I6YIjubSXbB/Wq6sO3CirhSR?=
 =?us-ascii?Q?oqFOMkpXqcNr7Ou7TrXVRDDuW/cdwaf+7NAuqzazoTOxcsJn05jzUqnA4gqw?=
 =?us-ascii?Q?zJO8sPqHBX0RKl9I37+fGYGnG/GKfo9l4V/pTy/aZSm4ZV3eioUFXX3zqfWA?=
 =?us-ascii?Q?hYNnxb49nRA8PnMLpsWUjNxR2vxwY5BjuYAgEtMaeWBadVTYd2zrJ0rKDLHi?=
 =?us-ascii?Q?rgHyA02fqNQF6/yUgG7u4L5lQGD8+7y3Sj9NpPLNEeOK164yhtuXVrFDyWCN?=
 =?us-ascii?Q?Vo5hg62J0ciDA8Cct1N72AuzlWQvj+UbeW7dRY3jxY8EZvEV9BDHzKtFSGHF?=
 =?us-ascii?Q?ZP0DwFn9TYTMS0qi8IXmhEt557Yt6dEXLos4/Sq8ky7tWynccAY5bXbtX0Kc?=
 =?us-ascii?Q?fw17DL3wRyL3D3g9Q62rkjBY6BUNokDGdYpAvP1fQ0ZkyU7ZzARK5qkSXOPt?=
 =?us-ascii?Q?ZvxOWD4TXUNCDryKU3OuQL9uDbxHeM7NJBJaHKB66XNUOPbgRfJGkfwYVAJv?=
 =?us-ascii?Q?8yXPHFsxxviQxph3NRBgxgQvMhQUuuyjATetiFnPltY1Km9rh0scct0vUo69?=
 =?us-ascii?Q?gQANKjRtANI4651m2gX6OayxonHqGKrqfznFaOk3OyRmqAJAS0R/9r257njA?=
 =?us-ascii?Q?Un83UPStJ3WC7nYUVOpfFrGPHtKZ6we/M0KKIaGCxVN38p5Ylp324N3o6BZk?=
 =?us-ascii?Q?/rw8QYpjlVehkvgADLZflkGTJNzIx6aqDntL7OIQ?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	k5Q/0X9Cv0b3RaQRyNegKq//P85RTse7ej5hVpYvbMqKA/v3GUX2AKDFyXO8CMBBpY8hNqSLERVCUFSvK7lmgYkwCdIpfZ6OVfukPV2Md5uE7NSVlAXqpVGegHTgKr6hji4AN3fwCzBAohFLxNCQOLI1CKtchzlvNg5vLkbk2kV9Miio+QhXigHmrtsnTrhJmSiTAK31TBQLbdSBpUX7BYKL93+WbOSvBB1R9mBLK+F25aY0M1jpEttJzl+Zi5MsW1VVQzDmLffYb4YStsmXWmjsuaoQBJRPE+wjJra8n0JOj+TI2maTCegceDMELJDL4n2vux9uCnKCG7/i6rcOULUFZYvYBRicUz4oheCNUEriyByX8sWpS0j5O7sCKcvTXQ+HB8+Ve+Lq9bAnbqiVE0re71qOxWOmHJqyeuVSGVLV7a8ndjU1CGk08VxAGdSnsNDwYryG4IAlIiQxeHOfvMgsM5BX67uURcLtq97xSM4w4d0P1/0/5OfKVJx1TotAOuiJBRnkd+PBmKSK/aqH/7fP1rE2uCxmqS27xRR02yoNNzwp0z46gqxnBI2xkUsE6JAgAU8HcMtLiI69vxS0KrIY0w6hNzMnG8f+rrRy7TM=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 50a76638-8c40-4f01-9a6b-08dc2f0cfcf3
X-MS-Exchange-CrossTenant-AuthSource: DS0PR10MB7933.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Feb 2024 16:33:21.0872
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: kaYJKTLDzi/yKjL9Hof0W1MwXhDbuFDfxnXroa6WwOpYia6hS21wdGkk6+BMJcuIPjHwNE2yZy/Psv9J+geA9Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR10MB7354
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-02-16_16,2024-02-16_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0
 mlxlogscore=999 malwarescore=0 mlxscore=0 spamscore=0 suspectscore=0
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2402160131
X-Proofpoint-GUID: sVE4T54mjSZJ5YkOANB0wLB4i2Yu5OcT
X-Proofpoint-ORIG-GUID: sVE4T54mjSZJ5YkOANB0wLB4i2Yu5OcT

* Jan Kara <jack@suse.cz> [240216 05:15]:
> On Thu 15-02-24 16:07:42, Liam R. Howlett wrote:
> > * Jan Kara <jack@suse.cz> [240215 12:16]:
> > > On Thu 15-02-24 12:00:08, Liam R. Howlett wrote:
> > > > * Jan Kara <jack@suse.cz> [240215 08:16]:
> > > > > On Tue 13-02-24 16:38:08, Chuck Lever wrote:
> > > > > > From: Chuck Lever <chuck.lever@oracle.com>
> > > > > > 
> > > > > > Liam says that, unlike with xarray, once the RCU read lock is
> > > > > > released ma_state is not safe to re-use for the next mas_find() call.
> > > > > > But the RCU read lock has to be released on each loop iteration so
> > > > > > that dput() can be called safely.
> > > > > > 

..

> > 
> > > Then how do you imagine serializing the
> > > background operations like compaction? As much as I agree your argument is
> > > "theoretically clean", it seems a bit like a trap and there are definitely
> > > xarray users that are going to be broken by this (e.g.
> > > tag_pages_for_writeback())...
> > 
> > I'm not sure I follow the trap logic.  There are locks for the data
> > structure that need to be followed for reading (rcu) and writing
> > (spinlock for the maple tree).  If you don't correctly lock the data
> > structure then you really are setting yourself up for potential issues
> > in the future.
> > 
> > The limitations are outlined in the documentation as to how and when to
> > lock.  I'm not familiar with the xarray users, but it does check for
> > locking with lockdep, but the way this is written bypasses the lockdep
> > checking as the locks are taken and dropped without the proper scope.
> > 
> > If you feel like this is a trap, then maybe we need to figure out a new
> > plan to detect incorrect use?
> 
> OK, I was a bit imprecise. What I wanted to say is that this is a shift in
> the paradigm in the sense that previously, we mostly had (and still have)
> data structure APIs (lists, rb-trees, radix-tree, now xarray) that were
> guaranteeing that unless you call into the function to mutate the data
> structure it stays intact. Now maple trees are shifting more in a direction
> of black-box API where you cannot assume what happens inside. Which is fine
> but then we have e.g. these iterators which do not quite follow this
> black-box design and you have to remember subtle details like calling
> "mas_pause()" before unlocking which is IMHO error-prone. Ideally, users of
> the black-box API shouldn't be exposed to the details of the internal
> locking at all (but then the performance suffers so I understand why you do
> things this way). Second to this ideal variant would be if we could detect
> we unlocked the lock without calling xas_pause() and warn on that. Or maybe
> xas_unlock*() should be calling xas_pause() automagically and we'd have
> similar helpers for RCU to do the magic for you?
> 
..

Fair enough.  You can think of the radix-tree and xarray as black boxes
as well; not everyone knows what's going on in there.  The rbtree and
list are embedded in your own data structure and you have to do a lot of
work for your operations.

Right now, it is the way you have described; it's a data structure API.
It will work this way, but you lose the really neat and performant part
of the tree.  If we do this right, we can compact at the same time as
reading data.  We cannot do that when depending on the locks you use
today. 

You don't have to use the mas_pause() functions, there are less
error-prone methods such as mt_find() that Chuck used here.  If you want
to do really neat stuff though, you should be looking at the mas_*
interface.  And yes, we totally hand you enough rope to hang yourself.
I'm not sure we can have an advanced interface without doing this.

Using the correct calls will allow for us to smoothly transition to a
model where you don't depend on the data remaining in the same place in
the tree (or xarray).

> 
> > If you have other examples you think are unsafe then I can have a look
> > at them as well.
> 
> I'm currently not aware of any but I'll let you know if I find some.
> Missing xas/mas_pause() seems really easy.

What if we convert the rcu_read_lock() to a mas_read_lock() or
xas_read_lock() and we can check to ensure the state isn't being locked
without being in the 'parked' state (paused or otherwise)?

mas_read_lock(struct ma_state *mas) {
	assert(!mas_active(mas));
	rcu_read_lock();
}

Would that be a reasonable resolution to your concern?  Unfortunately,
what was done with the locking in this case would not be detected with
this change unless the rcu_read_lock() was replaced.  IOW, people could
still use the rcu_read_lock() and skip the detection.

Doing the same in the mas_unlock() doesn't make as much sense since that
may be called without the intent to reuse the state at all.  So we'd be
doing more work than necessary at the end of each loop or use.

Thanks,
Liam