[permalink] [raw]

Subject: Re: [PATCH v2 0/6] Device tree support for Hyper-V VMBus driver

On Wed, Feb 01, 2023 at 06:15:23PM +0100, Krzysztof Kozlowski wrote:
> On 01/02/2023 17:34, Saurabh Singh Sengar wrote:
> >> Also see my comment on v1 about running DT validation on your dtb. I'm
> >> sure running it would point out other issues. Such as the root level
> >> comaptible string(s) need to be documented. You need cpu nodes,
> >> interrupt controller, timers, etc. Those all have to be documented.
> >
> > I will be changing the parent node to soc node as suggested by Krzysztof
> > in other thread.
> >
> > soc {
> > #address-cells = <2>;
> > #size-cells = <2>;
> >
> > vmbus@ff0000000 {
> > #address-cells = <2>;
> > #size-cells = <1>;
> > compatible = "Microsoft,vmbus";
> > ranges = <0x00 0x00 0x0f 0xf0000000 0x10000000>;
> > };
> > };
> >
> > This will be sufficient.
>
> It will be ok for the example, but will not be ok for supporting your
> use case. Please solve all the points from Rob's comment above. Where is
> their documentation?
>
> Best regards,
> Krzysztof

Hi Rob/ Krzysztof,

I am happy to update the documentation as requested. Please note
that, apart from CPUs, there is no other device node in the tree.

Here are some of the info related to our system:

Timers:
VMBus code uses a Hyper-V Synthetic timer and there is no device tree
node or ACPI method required for this. This is implemented as
drivers/clocksource/hyperv_timer.c

Interrupt controller:
The hypervisor virtualizes interrupt delivery to virtual processors.
This is done through the use of a synthetic interrupt controller
(SynIC) which is an extension of a virtualized local APIC. In the cpu
DT nodes we have APIC ids.

Below are the cpu nodes we use, please suggest if I need to update any
document for it.
cpus {
#address-cells = <1>;
#size-cells = <0>;

cpu@0 {
device_type = "cpu";
reg = <0>;
status = "okay";
};

cpu@1 {
device_type = "cpu";
reg = <1>;
status = "okay";
};
};

Regards,
Saurabh

2023-02-07 17:39:15

by Rob Herring (Arm)

[permalink] [raw]

Subject: Re: [PATCH v2 6/6] Driver: VMBus: Add device tree support

On Fri, Feb 3, 2023 at 11:36 AM Saurabh Singh Sengar
<[email protected]> wrote:
>
> On Wed, Feb 01, 2023 at 11:46:38AM -0600, Rob Herring wrote:
> > On Wed, Feb 01, 2023 at 08:51:33AM -0800, Saurabh Singh Sengar wrote:
> > > On Tue, Jan 31, 2023 at 02:12:53PM -0600, Rob Herring wrote:
> > > > On Tue, Jan 31, 2023 at 12:10 PM Saurabh Sengar
> > > > <[email protected]> wrote:
> > > > >
> > > > > Update the driver to support device tree boot as well along with ACPI.
> > > > > At present the device tree parsing only provides the mmio region info
> > > > > and is not the exact copy of ACPI parsing. This is sufficient to cater
> > > > > all the current device tree usecases for VMBus.
> > > > >
> > > > > Signed-off-by: Saurabh Sengar <[email protected]>
> > > > > ---
> > > > > drivers/hv/vmbus_drv.c | 75 ++++++++++++++++++++++++++++++++++++++++--
> > > > > 1 file changed, 73 insertions(+), 2 deletions(-)
> > > > >
> > > > > diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
> > > > > index 49030e756b9f..1741f1348f9f 100644
> > > > > --- a/drivers/hv/vmbus_drv.c
> > > > > +++ b/drivers/hv/vmbus_drv.c
> > > > > @@ -2152,7 +2152,7 @@ void vmbus_device_unregister(struct hv_device *device_obj)
> > > > > device_unregister(&device_obj->device);
> > > > > }
> (...)
> > > > > struct pci_dev *pdev;
> > > > > @@ -2442,6 +2443,7 @@ void vmbus_free_mmio(resource_size_t start, resource_size_t size)
> > > > > }
> > > > > EXPORT_SYMBOL_GPL(vmbus_free_mmio);
> > > > >
> > > > > +#ifdef CONFIG_ACPI
> > > >
> > > > It's better to put C 'if (!IS_ENABLED(CONFIG_ACPI)' code in the
> > >
> > > I wanted to have separate function for ACPI and device tree flow, which
> > > can be easily maintained with #ifdef. Please let me know if its fine.
> >
> > Yes, you can have separate functions:
> >
> > static int vmbus_acpi_add(struct platform_device *pdev)
> > {
> > if (!IS_ENABLED(CONFIG_ACPI))
> > return -ENODEV;
> >
> > ...
> > }
> >
> > The compiler will throw away the function in the end if CONFIG_ACPI is
> > not enabled.
> >
> > That is easier for us to maintain because it reduces the combinations to
> > build.
> >
>
> I tried removing #ifdef CONFIG_ACPI and use C's if(!IS_ENABLED(CONFIG_ACPI)) but looks
> compiler is not optimizing out the rest of function, it still throwing errors
> for acpi functions. This doesn't look 1:1 replacement to me.
> Please let me know if I have missunderstood any of your suggestion.
>
> drivers/hv/vmbus_drv.c:2175:8: error: implicit declaration of function ‘acpi_dev_resource_interrupt’ [-Werror=implicit-function-

That's a failure of the ACPI headers not having empty function
declarations. The DT functions do...

Also, this is just a broken assumption:

#ifdef CONFIG_ACPI

#else
// Assume DT
#endif

Both ACPI and DT can be enabled at the same time. They may be mutually
exclusive for a platform, but not the kernel. For distro kernels, both
will be enabled typically if the arch supports both. On arm64, DT is
never disabled because the boot interface is always DT.

Furthermore, this makes compile testing your code difficult. The arm64
defconfig, allmodconfig and allyesconfig all will not build the DT
code. The same for x86. This means all the CI builds that happen can't
build test this.

Rob

2023-02-07 17:54:24

by Rob Herring (Arm)

[permalink] [raw]

Subject: Re: [PATCH v2 0/6] Device tree support for Hyper-V VMBus driver

On Wed, Feb 1, 2023 at 10:34 AM Saurabh Singh Sengar
<[email protected]> wrote:
>
> On Wed, Feb 01, 2023 at 08:51:46AM -0600, Rob Herring wrote:
> > On Tue, Jan 31, 2023 at 06:04:49PM -0800, Saurabh Singh Sengar wrote:
> > > On Tue, Jan 31, 2023 at 02:27:51PM -0600, Rob Herring wrote:
> > > > On Tue, Jan 31, 2023 at 12:10 PM Saurabh Sengar
> > > > <[email protected]> wrote:
> > > > >
> > > > > This set of patches expands the VMBus driver to include device tree
> > > > > support.
> > > > >
> > > > > The first two patches enable compilation of Hyper-V APIs in a non-ACPI
> > > > > build.
> > > > >
> > > > > The third patch converts the VMBus driver from acpi to more generic
> > > > > platform driver.
> > > > >
> > > > > Further to add device tree documentation for VMBus, it needs to club with
> > > > > other virtualization driver's documentation. For this rename the virtio
> > > > > folder to more generic hypervisor, so that all the hypervisor based
> > > > > devices can co-exist in a single place in device tree documentation. The
> > > > > fourth patch does this renaming.
> > > > >
> > > > > The fifth patch introduces the device tree documentation for VMBus.
> > > > >
> > > > > The sixth patch adds device tree support to the VMBus driver. Currently
> > > > > this is tested only for x86 and it may not work for other archs.
> > > >
> > > > I can read all the patches and see *what* they do. You don't really
> > > > need to list that here. I'm still wondering *why*. That is what the
> > > > cover letter and commit messages should answer. Why do you need DT
> > > > support? How does this even work on x86? FDT is only enabled for
> > > > CE4100 platform.
> > >
> > > HI Rob,
> > >
> > > Thanks for your comments.
> > > We are working on a solution where kernel is booted without ACPI tables to keep
> > > the overall system's memory footprints slim and possibly faster boot time.
> > > We have tested this by enabling CONFIG_OF for x86.
> >
> > It's CONFIG_OF_EARLY_FLATTREE which you would need and that's not user
> > selectable. At a minimum, you need some kconfig changes. Where are
> > those?
>
> You are right we have define a new config flag in Kconfig, and selected CONFIG_OF
> and CONFIG_OF_EARLY_FLATTREE. We are working on upstreaming that patch as well
> however that will be a separate patch series.

Fair enough, but that should come first IMO. Really I just want to see
a complete picture. That can be a reference to a git branch(es) or
other patch series. But again, what I want to see in particular is the
actual DT and validation run on it.

> > Also see my comment on v1 about running DT validation on your dtb. I'm
> > sure running it would point out other issues. Such as the root level
> > comaptible string(s) need to be documented. You need cpu nodes,
> > interrupt controller, timers, etc. Those all have to be documented.
>
> I will be changing the parent node to soc node as suggested by Krzysztof
> in other thread.

Another issue yes, but orthogonal to my comments.

>
> soc {
> #address-cells = <2>;
> #size-cells = <2>;

You are missing 'ranges' here. Without it, addresses aren't translatable.

You are also missing 'compatible = "simple-bus";'. This happens to
work on x86 because of legacy reasons, but we don't want new cases
added.

>
> vmbus@ff0000000 {
> #address-cells = <2>;
> #size-cells = <1>;
> compatible = "Microsoft,vmbus";

'Microsoft' is not a vendor prefix.

> ranges = <0x00 0x00 0x0f 0xf0000000 0x10000000>;
> };
> };
>
> This will be sufficient.

All these comments are unnecessary because the tools will now check
these things and we shouldn't have to.

Rob

2023-02-07 18:37:50

by Sudeep Holla

[permalink] [raw]

Subject: Re: [PATCH v2 0/6] Device tree support for Hyper-V VMBus driver

On Tue, Jan 31, 2023 at 06:04:49PM -0800, Saurabh Singh Sengar wrote:
>
> We are working on a solution where kernel is booted without ACPI tables to keep
> the overall system's memory footprints slim and possibly faster boot time.
> We have tested this by enabling CONFIG_OF for x86.
>

Very interesting. Do you comparison on how slow/fast DT version w.r.t
ACPI and similarly for the memory footprint ? It would be good to add
those information as well. I know some systems just use ACPI static tables
as they don't run ACPICA runtime interpretter, was that experimented as
well or it wasn't used due to some specific(details please) limitations
without it.

> I can add this info in cover letter in next version.

Yes please.

--
Regards,
Sudeep

2023-02-08 02:30:57

by Saurabh Singh Sengar

[permalink] [raw]

Subject: Re: [PATCH v2 6/6] Driver: VMBus: Add device tree support

On Tue, Feb 07, 2023 at 11:38:55AM -0600, Rob Herring wrote:
> On Fri, Feb 3, 2023 at 11:36 AM Saurabh Singh Sengar
> <[email protected]> wrote:
> >
> > On Wed, Feb 01, 2023 at 11:46:38AM -0600, Rob Herring wrote:
> > > On Wed, Feb 01, 2023 at 08:51:33AM -0800, Saurabh Singh Sengar wrote:
> > > > On Tue, Jan 31, 2023 at 02:12:53PM -0600, Rob Herring wrote:
> > > > > On Tue, Jan 31, 2023 at 12:10 PM Saurabh Sengar
> > > > > <[email protected]> wrote:
> > > > I wanted to have separate function for ACPI and device tree flow, which
(...)
> > > > can be easily maintained with #ifdef. Please let me know if its fine.
> > >
> > > Yes, you can have separate functions:
> > >
> > > static int vmbus_acpi_add(struct platform_device *pdev)
> > > {
> > > if (!IS_ENABLED(CONFIG_ACPI))
> > > return -ENODEV;
> > >
> > > ...
> > > }
> > >
> > > The compiler will throw away the function in the end if CONFIG_ACPI is
> > > not enabled.
> > >
> > > That is easier for us to maintain because it reduces the combinations to
> > > build.
> > >
> >
> > I tried removing #ifdef CONFIG_ACPI and use C's if(!IS_ENABLED(CONFIG_ACPI)) but looks
> > compiler is not optimizing out the rest of function, it still throwing errors
> > for acpi functions. This doesn't look 1:1 replacement to me.
> > Please let me know if I have missunderstood any of your suggestion.
> >
> > drivers/hv/vmbus_drv.c:2175:8: error: implicit declaration of function ‘acpi_dev_resource_interrupt’ [-Werror=implicit-function-
>
> That's a failure of the ACPI headers not having empty function
> declarations. The DT functions do...
>
> Also, this is just a broken assumption:
>
> #ifdef CONFIG_ACPI
>
> #else
> // Assume DT
> #endif
>
> Both ACPI and DT can be enabled at the same time. They may be mutually
> exclusive for a platform, but not the kernel. For distro kernels, both
> will be enabled typically if the arch supports both. On arm64, DT is
> never disabled because the boot interface is always DT.
>
> Furthermore, this makes compile testing your code difficult. The arm64
> defconfig, allmodconfig and allyesconfig all will not build the DT
> code. The same for x86. This means all the CI builds that happen can't
> build test this.

Thanks for letting me know the challanges with testing. My intention was to give
ACPI higher priority, in case ACPI is enabled system should go ACPI flow, OF flow
should be used only when ACPI is disabled.

I can replace #else part with #ifdef CONFIG_OF if that helps.

Regards,
Saurabh

>
> Rob

2023-02-08 02:45:02

by Saurabh Singh Sengar

[permalink] [raw]

Subject: Re: [PATCH v2 0/6] Device tree support for Hyper-V VMBus driver

On Tue, Feb 07, 2023 at 11:53:54AM -0600, Rob Herring wrote:
> On Wed, Feb 1, 2023 at 10:34 AM Saurabh Singh Sengar
> <[email protected]> wrote:
> >
> > On Wed, Feb 01, 2023 at 08:51:46AM -0600, Rob Herring wrote:
> > > On Tue, Jan 31, 2023 at 06:04:49PM -0800, Saurabh Singh Sengar wrote:
> > > > On Tue, Jan 31, 2023 at 02:27:51PM -0600, Rob Herring wrote:
> > > > > On Tue, Jan 31, 2023 at 12:10 PM Saurabh Sengar
> > > > > <[email protected]> wrote:
> > > > > >
> > > > > > This set of patches expands the VMBus driver to include device tree
> > > > > > support.
(...)
> > You are right we have define a new config flag in Kconfig, and selected CONFIG_OF
> > and CONFIG_OF_EARLY_FLATTREE. We are working on upstreaming that patch as well
> > however that will be a separate patch series.
>
> Fair enough, but that should come first IMO. Really I just want to see
> a complete picture. That can be a reference to a git branch(es) or
> other patch series. But again, what I want to see in particular is the
> actual DT and validation run on it.

Thank you for explaining the concern. I now understand it fully. I have come to the
realization that enabling the vmbus device tree should not be impacted by any changes.
To address this, I will add the following lines to the HYPERV Kconfig definition I
used for testing:

select OF if !ACPI
select OF_EARLY_FLATTREE if !ACPI"

>
> > > Also see my comment on v1 about running DT validation on your dtb. I'm
> > > sure running it would point out other issues. Such as the root level
> > > comaptible string(s) need to be documented. You need cpu nodes,
> > > interrupt controller, timers, etc. Those all have to be documented.
> >
> > I will be changing the parent node to soc node as suggested by Krzysztof
> > in other thread.
>
> Another issue yes, but orthogonal to my comments.
>
> >
> > soc {
> > #address-cells = <2>;
> > #size-cells = <2>;
>
> You are missing 'ranges' here. Without it, addresses aren't translatable.
>
> You are also missing 'compatible = "simple-bus";'. This happens to
> work on x86 because of legacy reasons, but we don't want new cases
> added.

I am a bit unclear on the reason for adding the ranges property in the root node.
To provide more context, I have included my full device tree below. I believe that
having the ranges property in the VMBus device node is sufficient. Please let me
know if this can be improved.

/dts-v1/;

/ {
#address-cells = <0>;
#size-cells = <0>;
model = "microsoft,test";

cpus {
#address-cells = <0x01>;
#size-cells = <0x00>;

cpu@0 {
device_type = "cpu";
reg = <0x00>;
status = "okay";
};

cpu@1 {
device_type = "cpu";
reg = <0x01>;
status = "okay";
};
};

vmbus@ff0000000 {
#address-cells = <2>;
#size-cells = <2>;
compatible = "microsoft,vmbus";
ranges = <0x0f 0xf0000000 0x0f 0xf0000000 0x0 0x10000000>;
};
};

>
> >
> > vmbus@ff0000000 {
> > #address-cells = <2>;
> > #size-cells = <1>;
> > compatible = "Microsoft,vmbus";
>
> 'Microsoft' is not a vendor prefix.
>
> > ranges = <0x00 0x00 0x0f 0xf0000000 0x10000000>;
> > };
> > };
> >
> > This will be sufficient.
>
> All these comments are unnecessary because the tools will now check
> these things and we shouldn't have to.

Agree, and its fixed in latest version.

Regards,
Saurabh
>
> Rob