Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 03 May 2022 10:32:58 +0100
Message-ID: <87fslr7ygl.wl-maz@kernel.org>
From:   Marc Zyngier <maz@kernel.org>
To:     Marek =?UTF-8?B?QmVow7pu?= <kabel@kernel.org>
Cc:     linux-kernel@vger.kernel.org, Thomas Gleixner <tglx@linutronix.de>,
        pali@kernel.org
Subject: Re: irqdomain API: how to set affinity of parent irq of chained irqs?
In-Reply-To: <20220502174559.78f5cbc0@dellmb>
References: <20220502102137.764606ee@thinkpad>
        <87mtg0i8m8.wl-maz@kernel.org>
        <20220502174559.78f5cbc0@dellmb>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue)
 FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1
 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO)
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Mon, 02 May 2022 16:45:59 +0100,
Marek Beh=C3=BAn <kabel@kernel.org> wrote:
>=20
> On Mon, 02 May 2022 10:31:11 +0100
> Marc Zyngier <maz@kernel.org> wrote:
>=20
> > On Mon, 02 May 2022 09:21:37 +0100,
> > Marek Beh=C3=BAn <kabel@kernel.org> wrote:
> > >=20
> > > Dear Marc, Thomas,
> > >=20
> > > we have encountered the following problem that can hopefully be put
> > > some light onto: What is the intended way to set affinity (and possib=
ly
> > > other irq attributes) of parent IRQ of chained IRQs, when using the
> > > irqdomain API? =20
> >=20
> > Simples: you can't. What sense does it make to change the affinity of
> > the parent interrupt, given that its fate is tied to *all* of the
> > other interrupts that are muxed to it?
>=20
> Dear Marc,
>=20
> thank you for your answer. Still:
>=20
> What about when we want to set the same affinity for all the chained
> interrupts?
>=20
> Example: on Armada 385 there are 4 PCIe controllers. Each controller
> has one interrupt from which we trigger chained interrupts. We would
> like to configure each controller to trigger interrupt (and thus all
> chained interrupts in the domain) on different CPU core.
>=20
> Moreover we would really like to do this in runtime, through sysfs,
> depending on for example whether there are cards plugged in the PCIe
> ports.
>=20
> Maybe there should be some mechanism to allow to change affinity for
> whole irqdomain, or something?

Should? Maybe. But not for an irqdomain (which really doesn't have
anything to do with interrupt affinity).

What you may want is a new sysfs interface that would allow a parent
interrupt affinity being changed, but also exposing to userspace all
the interrupts this affects *at the same time*. something like:

/sys/kernel/irq/42/smp_affinity_list
/sys/kernel/irq/42/muxed_irqs/
/sys/kernel/irq/42/muxed_irqs/56 -> ../../56
/sys/kernel/irq/42/muxed_irqs/57 -> ../../57

The main issues are that:

- we don't really track the muxing information in any of the data
  structures, so you can't just walk a short list and generate this
  information. You'd need to build the topology information at
  allocation time (or fish it out at runtime, but that's likely a
  pain).

- sysfs doesn't deal with affinities at all. procfs does, but adding
  more crap there is frowned upon.

- it *must* be a new interface. You can't repurpose the existing one,
  as something like irqbalance would be otherwise be massively
  confused by seeing interrupts moving around behind its back.

- conversely, you'll need to teach irqbalance how to deal with this
  new interface.

- this needs to be safe against CPU hotplug. It probably already is,
  but nobody ever tested it, given that userspace can't interact with
  these interrupts at the moment.

	M.

--=20
Without deviation from the norm, progress is not possible.