Hi,
When building guest kernels for virtualisation we were bringing in a
bunch of stuff from physical hardware which we don't need for our
idealised fixable virtual PCI devices. This series makes some Kconfig
changes to allow the ThunderX and XGene PCI drivers to be compiled
out. It also drops PCI_QUIRKS from the KVM guest build as a virtual
PCI device should be quirk free.
This is my first time hacking around Kconfig so I hope I've got the
balance between depends and selects right but please let be know if it
could be specified in a cleaner way.
Alex Bennée (3):
arm64: allow de-selection of ThunderX PCI controllers
arm64: gate the whole of pci-xgene on CONFIG_PCI_XGENE
kernel/configs: don't include PCI_QUIRKS in KVM guest configs
arch/arm64/Kconfig.platforms | 2 ++
arch/arm64/configs/defconfig | 1 +
drivers/pci/controller/Kconfig | 7 +++++++
drivers/pci/controller/Makefile | 8 +++-----
kernel/configs/kvm_guest.config | 1 +
5 files changed, 14 insertions(+), 5 deletions(-)
--
2.20.1
The VIRTIO_PCI support is an idealised PCI bus, we don't need a bunch
of bloat for real world hardware for a VirtIO guest.
Signed-off-by: Alex Bennée <[email protected]>
---
kernel/configs/kvm_guest.config | 1 +
1 file changed, 1 insertion(+)
diff --git a/kernel/configs/kvm_guest.config b/kernel/configs/kvm_guest.config
index 208481d91090..672863a2fdf1 100644
--- a/kernel/configs/kvm_guest.config
+++ b/kernel/configs/kvm_guest.config
@@ -13,6 +13,7 @@ CONFIG_IP_PNP_DHCP=y
CONFIG_BINFMT_ELF=y
CONFIG_PCI=y
CONFIG_PCI_MSI=y
+CONFIG_PCI_QUIRKS=n
CONFIG_DEBUG_KERNEL=y
CONFIG_VIRTUALIZATION=y
CONFIG_HYPERVISOR_GUEST=y
--
2.20.1
This is a little weirder as bits of the file are already conditioned
on the exiting symbol. Either way they are not actually needed for
non-xgene machines saving another 12k:
-rwxr-xr-x 1 alex alex 86033880 Aug 3 16:39 vmlinux.orig*
-rwxr-xr-x 1 alex alex 85652472 Aug 3 16:54 vmlinux.rm-thunder*
-rwxr-xr-x 1 alex alex 85639808 Aug 3 17:12 vmlinux*
Signed-off-by: Alex Bennée <[email protected]>
---
drivers/pci/controller/Makefile | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/drivers/pci/controller/Makefile b/drivers/pci/controller/Makefile
index 8fad4781a5d3..3b9b72f5773a 100644
--- a/drivers/pci/controller/Makefile
+++ b/drivers/pci/controller/Makefile
@@ -47,6 +47,4 @@ obj-y += mobiveil/
obj-$(CONFIG_PCI_THUNDER) += pci-thunder-ecam.o
obj-$(CONFIG_PCI_THUNDER) += pci-thunder-pem.o
-ifdef CONFIG_PCI
-obj-$(CONFIG_ARM64) += pci-xgene.o
-endif
+obj-$(CONFIG_PCI_XGENE) += pci-xgene.o
--
2.20.1
For a pure VirtIO guest bringing in all the PCI quirk handling adds a
significant amount of bloat to kernel we don't need. Solve this by
adding a CONFIG symbol for the ThunderX PCI devices and allowing it to
be turned off. Saving over 300k from the uncompressed vmlinux:
-rwxr-xr-x 1 alex alex 85652472 Aug 3 16:48 vmlinux*
-rwxr-xr-x 1 alex alex 86033880 Aug 3 16:39 vmlinux.orig*
Signed-off-by: Alex Bennée <[email protected]>
Cc: Robert Richter <[email protected]>
Cc: [email protected]
Cc: [email protected]
---
arch/arm64/Kconfig.platforms | 2 ++
arch/arm64/configs/defconfig | 1 +
drivers/pci/controller/Kconfig | 7 +++++++
drivers/pci/controller/Makefile | 4 ++--
4 files changed, 12 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/Kconfig.platforms b/arch/arm64/Kconfig.platforms
index 8dd05b2a925c..a328eebdaa59 100644
--- a/arch/arm64/Kconfig.platforms
+++ b/arch/arm64/Kconfig.platforms
@@ -253,12 +253,14 @@ config ARCH_SPRD
config ARCH_THUNDER
bool "Cavium Inc. Thunder SoC Family"
+ select PCI_THUNDER
help
This enables support for Cavium's Thunder Family of SoCs.
config ARCH_THUNDER2
bool "Cavium ThunderX2 Server Processors"
select GPIOLIB
+ select PCI_THUNDER
help
This enables support for Cavium's ThunderX2 CN99XX family of
server processors.
diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
index 2ca7ba69c318..d840cba99941 100644
--- a/arch/arm64/configs/defconfig
+++ b/arch/arm64/configs/defconfig
@@ -199,6 +199,7 @@ CONFIG_PCI_HOST_GENERIC=y
CONFIG_PCI_XGENE=y
CONFIG_PCIE_ALTERA=y
CONFIG_PCIE_ALTERA_MSI=y
+CONFIG_PCI_THUNDER=y
CONFIG_PCI_HOST_THUNDER_PEM=y
CONFIG_PCI_HOST_THUNDER_ECAM=y
CONFIG_PCIE_ROCKCHIP_HOST=m
diff --git a/drivers/pci/controller/Kconfig b/drivers/pci/controller/Kconfig
index adddf21fa381..28335ffa5d48 100644
--- a/drivers/pci/controller/Kconfig
+++ b/drivers/pci/controller/Kconfig
@@ -286,6 +286,13 @@ config PCI_LOONGSON
Say Y here if you want to enable PCI controller support on
Loongson systems.
+config PCI_THUNDER
+ bool "Thunder X PCIE controllers"
+ depends on ARM64
+ select PCI_QUIRKS
+ help
+ Say Y here to enable ThunderX ECAM and PEM PCI controllers.
+
source "drivers/pci/controller/dwc/Kconfig"
source "drivers/pci/controller/mobiveil/Kconfig"
source "drivers/pci/controller/cadence/Kconfig"
diff --git a/drivers/pci/controller/Makefile b/drivers/pci/controller/Makefile
index efd9733ead26..8fad4781a5d3 100644
--- a/drivers/pci/controller/Makefile
+++ b/drivers/pci/controller/Makefile
@@ -45,8 +45,8 @@ obj-y += mobiveil/
# ARM64 and use internal ifdefs to only build the pieces we need
# depending on whether ACPI, the DT driver, or both are enabled.
+obj-$(CONFIG_PCI_THUNDER) += pci-thunder-ecam.o
+obj-$(CONFIG_PCI_THUNDER) += pci-thunder-pem.o
ifdef CONFIG_PCI
-obj-$(CONFIG_ARM64) += pci-thunder-ecam.o
-obj-$(CONFIG_ARM64) += pci-thunder-pem.o
obj-$(CONFIG_ARM64) += pci-xgene.o
endif
--
2.20.1
On Tue, 4 Aug 2020 at 14:45, Alex Bennée <[email protected]> wrote:
>
> Hi,
>
> When building guest kernels for virtualisation we were bringing in a
> bunch of stuff from physical hardware which we don't need for our
> idealised fixable virtual PCI devices. This series makes some Kconfig
> changes to allow the ThunderX and XGene PCI drivers to be compiled
> out. It also drops PCI_QUIRKS from the KVM guest build as a virtual
> PCI device should be quirk free.
>
What about PCI passthrough?
> This is my first time hacking around Kconfig so I hope I've got the
> balance between depends and selects right but please let be know if it
> could be specified in a cleaner way.
>
> Alex Bennée (3):
> arm64: allow de-selection of ThunderX PCI controllers
> arm64: gate the whole of pci-xgene on CONFIG_PCI_XGENE
> kernel/configs: don't include PCI_QUIRKS in KVM guest configs
>
> arch/arm64/Kconfig.platforms | 2 ++
> arch/arm64/configs/defconfig | 1 +
> drivers/pci/controller/Kconfig | 7 +++++++
> drivers/pci/controller/Makefile | 8 +++-----
> kernel/configs/kvm_guest.config | 1 +
> 5 files changed, 14 insertions(+), 5 deletions(-)
>
> --
> 2.20.1
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> [email protected]
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On 2020-08-04 13:44, Alex Bennée wrote:
> The VIRTIO_PCI support is an idealised PCI bus, we don't need a bunch
> of bloat for real world hardware for a VirtIO guest.
Who says this guest will only have virtio devices?
Or even, virtio devices without bugs? Given that said device can
come from any VMM, I'm not sure this is the right thing to do.
Thanks,
M.
>
> Signed-off-by: Alex Bennée <[email protected]>
> ---
> kernel/configs/kvm_guest.config | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/kernel/configs/kvm_guest.config
> b/kernel/configs/kvm_guest.config
> index 208481d91090..672863a2fdf1 100644
> --- a/kernel/configs/kvm_guest.config
> +++ b/kernel/configs/kvm_guest.config
> @@ -13,6 +13,7 @@ CONFIG_IP_PNP_DHCP=y
> CONFIG_BINFMT_ELF=y
> CONFIG_PCI=y
> CONFIG_PCI_MSI=y
> +CONFIG_PCI_QUIRKS=n
> CONFIG_DEBUG_KERNEL=y
> CONFIG_VIRTUALIZATION=y
> CONFIG_HYPERVISOR_GUEST=y
--
Jazz is not dead. It just smells funny...
Marc Zyngier <[email protected]> writes:
> On 2020-08-04 13:44, Alex Bennée wrote:
>> The VIRTIO_PCI support is an idealised PCI bus, we don't need a bunch
>> of bloat for real world hardware for a VirtIO guest.
>
> Who says this guest will only have virtio devices?
This is true - although what is the point of kvm_guest.config? We
certainly turn on a whole bunch of virt optimised pathways with PARAVIRT
and HYPERVISOR_GUEST along with the rest of VirtIO.
> Or even, virtio devices without bugs? Given that said device can
> come from any VMM, I'm not sure this is the right thing to do.
Perhaps this patch is one too far. I don't mind dropping it as long as I
can still slim down the kernels I know don't need the extra bloat.
>
> Thanks,
>
> M.
>
>>
>> Signed-off-by: Alex Bennée <[email protected]>
>> ---
>> kernel/configs/kvm_guest.config | 1 +
>> 1 file changed, 1 insertion(+)
>>
>> diff --git a/kernel/configs/kvm_guest.config
>> b/kernel/configs/kvm_guest.config
>> index 208481d91090..672863a2fdf1 100644
>> --- a/kernel/configs/kvm_guest.config
>> +++ b/kernel/configs/kvm_guest.config
>> @@ -13,6 +13,7 @@ CONFIG_IP_PNP_DHCP=y
>> CONFIG_BINFMT_ELF=y
>> CONFIG_PCI=y
>> CONFIG_PCI_MSI=y
>> +CONFIG_PCI_QUIRKS=n
>> CONFIG_DEBUG_KERNEL=y
>> CONFIG_VIRTUALIZATION=y
>> CONFIG_HYPERVISOR_GUEST=y
--
Alex Bennée
Ard Biesheuvel <[email protected]> writes:
> On Tue, 4 Aug 2020 at 14:45, Alex Bennée <[email protected]> wrote:
>>
>> Hi,
>>
>> When building guest kernels for virtualisation we were bringing in a
>> bunch of stuff from physical hardware which we don't need for our
>> idealised fixable virtual PCI devices. This series makes some Kconfig
>> changes to allow the ThunderX and XGene PCI drivers to be compiled
>> out. It also drops PCI_QUIRKS from the KVM guest build as a virtual
>> PCI device should be quirk free.
>>
>
> What about PCI passthrough?
That is a good point - how much of the host PCI controller is visible to
a pass-through guest?
AIUI in passthrough the driver only interacts with the particular cards
IO window. How many quirks are visible just at the device level (rather
than the bus itself)?
That said I think the last patch might get dropped as long as the user
has the option to slim down their kernel with the first two.
>
>> This is my first time hacking around Kconfig so I hope I've got the
>> balance between depends and selects right but please let be know if it
>> could be specified in a cleaner way.
>>
>> Alex Bennée (3):
>> arm64: allow de-selection of ThunderX PCI controllers
>> arm64: gate the whole of pci-xgene on CONFIG_PCI_XGENE
>> kernel/configs: don't include PCI_QUIRKS in KVM guest configs
>>
>> arch/arm64/Kconfig.platforms | 2 ++
>> arch/arm64/configs/defconfig | 1 +
>> drivers/pci/controller/Kconfig | 7 +++++++
>> drivers/pci/controller/Makefile | 8 +++-----
>> kernel/configs/kvm_guest.config | 1 +
>> 5 files changed, 14 insertions(+), 5 deletions(-)
>>
>> --
>> 2.20.1
>>
>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> [email protected]
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
--
Alex Bennée
On 2020-08-04 15:44, Alex Bennée wrote:
> Marc Zyngier <[email protected]> writes:
>
>> On 2020-08-04 13:44, Alex Bennée wrote:
>>> The VIRTIO_PCI support is an idealised PCI bus, we don't need a bunch
>>> of bloat for real world hardware for a VirtIO guest.
>>
>> Who says this guest will only have virtio devices?
>
> This is true - although what is the point of kvm_guest.config? We
> certainly turn on a whole bunch of virt optimised pathways with
> PARAVIRT
> and HYPERVISOR_GUEST along with the rest of VirtIO.
Most of which actually qualifies as bloat itself as far as KVM/arm64
is concerned...
M.
--
Jazz is not dead. It just smells funny...
Marc Zyngier <[email protected]> writes:
> On 2020-08-04 15:44, Alex Bennée wrote:
>> Marc Zyngier <[email protected]> writes:
>>
>>> On 2020-08-04 13:44, Alex Bennée wrote:
>>>> The VIRTIO_PCI support is an idealised PCI bus, we don't need a bunch
>>>> of bloat for real world hardware for a VirtIO guest.
>>>
>>> Who says this guest will only have virtio devices?
>>
>> This is true - although what is the point of kvm_guest.config? We
>> certainly turn on a whole bunch of virt optimised pathways with
>> PARAVIRT
>> and HYPERVISOR_GUEST along with the rest of VirtIO.
>
> Most of which actually qualifies as bloat itself as far as KVM/arm64
> is concerned...
So here is the question - does the kernel care about having a blessed
config for a minimal viable guest? They are certainly used in the cloud
but I understand the kernel is trying to get away from having a zoo of
configs. What is the actual point of kvm_guest.config? Just an easy
enabling for developers?
>
> M.
--
Alex Bennée
On 2020-08-04 16:40, Alex Bennée wrote:
> Marc Zyngier <[email protected]> writes:
>
>> On 2020-08-04 15:44, Alex Bennée wrote:
>>> Marc Zyngier <[email protected]> writes:
>>>
>>>> On 2020-08-04 13:44, Alex Bennée wrote:
>>>>> The VIRTIO_PCI support is an idealised PCI bus, we don't need a
>>>>> bunch
>>>>> of bloat for real world hardware for a VirtIO guest.
>>>>
>>>> Who says this guest will only have virtio devices?
>>>
>>> This is true - although what is the point of kvm_guest.config? We
>>> certainly turn on a whole bunch of virt optimised pathways with
>>> PARAVIRT
>>> and HYPERVISOR_GUEST along with the rest of VirtIO.
>>
>> Most of which actually qualifies as bloat itself as far as KVM/arm64
>> is concerned...
>
> So here is the question - does the kernel care about having a blessed
> config for a minimal viable guest? They are certainly used in the cloud
> but I understand the kernel is trying to get away from having a zoo of
> configs. What is the actual point of kvm_guest.config? Just an easy
> enabling for developers?
The cloud vendor I know certainly doesn't provide a "dumbed down"
kernel configuration. What they run is either a distro kernel
or something that fits their environment (which does include
HW PCI devices, and hardly any virtio device).
My take is that this kvm-special config isn't that useful in
the real world, and I don't believe there is such thing as a
"minimal viable guest" config, certainly not across architectures
and VMMs. Hopefully it fits someone's development workflow, but
that's probably it.
M.
--
Jazz is not dead. It just smells funny...
On Tue, Aug 04, 2020 at 01:44:15PM +0100, Alex Benn?e wrote:
> For a pure VirtIO guest bringing in all the PCI quirk handling adds a
> significant amount of bloat to kernel we don't need. Solve this by
> adding a CONFIG symbol for the ThunderX PCI devices and allowing it to
> be turned off. Saving over 300k from the uncompressed vmlinux:
It *looks* like just turning off CONFIG_PCI_QUIRKS should be
sufficient because pci-thunder-ecam.c and pci-thunder-pem.c are
wrapped with:
#if defined(CONFIG_PCI_HOST_THUNDER_PEM) || (defined(CONFIG_ACPI) && defined(CONFIG_PCI_QUIRKS))
so if you turn off CONFIG_PCI_HOST_THUNDER_ECAM,
CONFIG_PCI_HOST_THUNDER_PEM, and CONFIG_PCI_QUIRKS that should omit
pci-thunder-ecam.o and pci-thunder-pem.o. But I must be missing
something.
> -rwxr-xr-x 1 alex alex 85652472 Aug 3 16:48 vmlinux*
> -rwxr-xr-x 1 alex alex 86033880 Aug 3 16:39 vmlinux.orig*
>
> Signed-off-by: Alex Benn?e <[email protected]>
> Cc: Robert Richter <[email protected]>
> Cc: [email protected]
> Cc: [email protected]
> ---
> arch/arm64/Kconfig.platforms | 2 ++
> arch/arm64/configs/defconfig | 1 +
> drivers/pci/controller/Kconfig | 7 +++++++
> drivers/pci/controller/Makefile | 4 ++--
> 4 files changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm64/Kconfig.platforms b/arch/arm64/Kconfig.platforms
> index 8dd05b2a925c..a328eebdaa59 100644
> --- a/arch/arm64/Kconfig.platforms
> +++ b/arch/arm64/Kconfig.platforms
> @@ -253,12 +253,14 @@ config ARCH_SPRD
>
> config ARCH_THUNDER
> bool "Cavium Inc. Thunder SoC Family"
> + select PCI_THUNDER
> help
> This enables support for Cavium's Thunder Family of SoCs.
>
> config ARCH_THUNDER2
> bool "Cavium ThunderX2 Server Processors"
> select GPIOLIB
> + select PCI_THUNDER
Indent these with tabs, not spaces, to be consistent with other
whitespace usage in this file.
> help
> This enables support for Cavium's ThunderX2 CN99XX family of
> server processors.
> diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
> index 2ca7ba69c318..d840cba99941 100644
> --- a/arch/arm64/configs/defconfig
> +++ b/arch/arm64/configs/defconfig
> @@ -199,6 +199,7 @@ CONFIG_PCI_HOST_GENERIC=y
> CONFIG_PCI_XGENE=y
> CONFIG_PCIE_ALTERA=y
> CONFIG_PCIE_ALTERA_MSI=y
> +CONFIG_PCI_THUNDER=y
> CONFIG_PCI_HOST_THUNDER_PEM=y
> CONFIG_PCI_HOST_THUNDER_ECAM=y
> CONFIG_PCIE_ROCKCHIP_HOST=m
> diff --git a/drivers/pci/controller/Kconfig b/drivers/pci/controller/Kconfig
> index adddf21fa381..28335ffa5d48 100644
> --- a/drivers/pci/controller/Kconfig
> +++ b/drivers/pci/controller/Kconfig
> @@ -286,6 +286,13 @@ config PCI_LOONGSON
> Say Y here if you want to enable PCI controller support on
> Loongson systems.
>
> +config PCI_THUNDER
> + bool "Thunder X PCIE controllers"
> + depends on ARM64
> + select PCI_QUIRKS
> + help
> + Say Y here to enable ThunderX ECAM and PEM PCI controllers.
Indent with tabs, not spaces.
The existing Kconfig help text refers to simply "Thunder", not
"Thunder X", so both of these references should probably follow suit.
s/PCIE controllers/PCIe controllers/
> source "drivers/pci/controller/dwc/Kconfig"
> source "drivers/pci/controller/mobiveil/Kconfig"
> source "drivers/pci/controller/cadence/Kconfig"
> diff --git a/drivers/pci/controller/Makefile b/drivers/pci/controller/Makefile
> index efd9733ead26..8fad4781a5d3 100644
> --- a/drivers/pci/controller/Makefile
> +++ b/drivers/pci/controller/Makefile
> @@ -45,8 +45,8 @@ obj-y += mobiveil/
> # ARM64 and use internal ifdefs to only build the pieces we need
> # depending on whether ACPI, the DT driver, or both are enabled.
>
> +obj-$(CONFIG_PCI_THUNDER) += pci-thunder-ecam.o
> +obj-$(CONFIG_PCI_THUNDER) += pci-thunder-pem.o
> ifdef CONFIG_PCI
> -obj-$(CONFIG_ARM64) += pci-thunder-ecam.o
> -obj-$(CONFIG_ARM64) += pci-thunder-pem.o
> obj-$(CONFIG_ARM64) += pci-xgene.o
> endif
> --
> 2.20.1
>