Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <62f7402d-f0e7-8e8a-e1a4-958ddbcf8d8b@quicinc.com>
Date:   Tue, 1 Nov 2022 17:12:14 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.3.3
Subject: Re: [PATCH v6 02/21] dt-bindings: Add binding for gunyah hypervisor
Content-Language: en-US
To:     Jassi Brar <jassisinghbrar@gmail.com>
CC:     Bjorn Andersson <quic_bjorande@quicinc.com>,
        Rob Herring <robh+dt@kernel.org>,
        Krzysztof Kozlowski <krzysztof.kozlowski+dt@linaro.org>,
        Murali Nalajala <quic_mnalajal@quicinc.com>,
        Trilok Soni <quic_tsoni@quicinc.com>,
        "Srivatsa Vaddagiri" <quic_svaddagi@quicinc.com>,
        Carl van Schaik <quic_cvanscha@quicinc.com>,
        Prakruthi Deepak Heragu <quic_pheragu@quicinc.com>,
        Andy Gross <agross@kernel.org>,
        Dmitry Baryshkov <dmitry.baryshkov@linaro.org>,
        <linux-arm-kernel@lists.infradead.org>,
        "Mark Rutland" <mark.rutland@arm.com>,
        Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
        Sudeep Holla <sudeep.holla@arm.com>,
        "Marc Zyngier" <maz@kernel.org>, Jonathan Corbet <corbet@lwn.net>,
        Will Deacon <will@kernel.org>,
        Catalin Marinas <catalin.marinas@arm.com>,
        Arnd Bergmann <arnd@arndb.de>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        "Srinivas Kandagatla" <srinivas.kandagatla@linaro.org>,
        Amol Maheshwari <amahesh@qti.qualcomm.com>,
        Kalle Valo <kvalo@kernel.org>, <devicetree@vger.kernel.org>,
        <linux-doc@vger.kernel.org>, <linux-arm-msm@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>
References: <20221026185846.3983888-1-quic_eberman@quicinc.com>
 <20221026185846.3983888-3-quic_eberman@quicinc.com>
 <CABb+yY3JVNPG3dcyHNFxEeGEu3MN_pAOh3+cwexPPe2YG6SNUg@mail.gmail.com>
 <fb7e101f-8de0-d77e-30e1-74b882b19583@quicinc.com>
 <CABb+yY08jP+Q5xvzLf=7F1tULP6-eZz5EDiK9mBj2fAv=iZa_A@mail.gmail.com>
 <4cb58489-cd42-1868-9add-0c360065de23@quicinc.com>
 <CABb+yY2GA90RLazHZL7sLtC+ka-P8y6s00V2BVF4OMPTDi-rKg@mail.gmail.com>
From:   Elliot Berman <quic_eberman@quicinc.com>
In-Reply-To: <CABb+yY2GA90RLazHZL7sLtC+ka-P8y6s00V2BVF4OMPTDi-rKg@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk


On 11/1/2022 2:58 PM, Jassi Brar wrote:
> On Tue, Nov 1, 2022 at 3:35 PM Elliot Berman <quic_eberman@quicinc.com> wrote:
>>
>>
>>
>> On 11/1/2022 9:23 AM, Jassi Brar wrote:
>>> On Mon, Oct 31, 2022 at 10:20 PM Elliot Berman <quic_eberman@quicinc.com> wrote:
>>>>
>>>> Hi Jassi,
>>>>
>>>> On 10/27/2022 7:33 PM, Jassi Brar wrote:
>>>>    > On Wed, Oct 26, 2022 at 1:59 PM Elliot Berman
>>>> <quic_eberman@quicinc.com> wrote:
>>>>    > .....
>>>>    >> +
>>>>    >> +        gunyah-resource-mgr@0 {
>>>>    >> +            compatible = "gunyah-resource-manager-1-0",
>>>> "gunyah-resource-manager";
>>>>    >> +            interrupts = <GIC_SPI 3 IRQ_TYPE_EDGE_RISING>, /* TX
>>>> full IRQ */
>>>>    >> +                         <GIC_SPI 4 IRQ_TYPE_EDGE_RISING>; /* RX
>>>> empty IRQ */
>>>>    >> +            reg = <0x00000000 0x00000000>, <0x00000000 0x00000001>;
>>>>    >> +                  /* TX, RX cap ids */
>>>>    >> +        };
>>>>    >>
>>>>    > All these resources are used only by the mailbox controller driver.
>>>>    > So, this should be the mailbox controller node, rather than the
>>>>    > mailbox user.> One option is to load gunyah-resource-manager as a
>>>> module that relies
>>>>    > on the gunyah-mailbox provider. That would also avoid the "Allow
>>>>    > direct registration to a channel" hack patch.
>>>>
>>>> A message queue to another guest VM wouldn't be known at boot time and
>>>> thus couldn't be described on the devicetree.
>>>>
>>> I think you need to implement of_xlate() ... or please tell me what
>>> exactly you need to specify in the dt.
>>
>> Dynamically created virtual machines can't be known on the dt, so there
>> is nothing to specify in the DT. There couldn't be a devicetree node for
>> the message queue client because that client is only exists once the VM
>> is created by userspace.
>>
> The underlying "physical channel" is the synchronous SMC instruction,
> which remains 1 irrespective of the number of mailbox instances
> created.

I disagree that the physical channel is the SMC instruction. Regardless 
though, there are num_online_cpus() "physical channels" with this 
perspective.

> So basically you are sharing one resource among users. Why doesn't the
> RM request the "smc instruction" channel once and share it among
> users?

I suppose in this scenario, a single mailbox channel would represent all 
message queues? This would cause Linux to serialize *all* message queue 
hypercalls. Sorry, I can only think negative implications.

Error handling needs to move into clients: if a TX message queue becomes 
full or an RX message queue becomes empty, then we'll need to return 
error back to the client right away. The clients would need to register 
for the RTS/RTR interrupts to know when to send/receive messages and 
have retry error handling. If the mailbox controller retried for the 
clients as currently proposed, then we could get into a scenario where a 
message queue could never be ready to send/receive and thus stuck 
forever trying to process that message. The effect here would be that 
the mailbox controller becomes a wrapper to some SMC instructions that 
aren't related at the SMC instruction level.

A single channel would limit performance of SMP systems because only one 
core could send/receive a message. There is no such limitation for 
message queues to behave like this.

Thanks,
Elliot