On Thu, Aug 25, 2016 at 2:19 PM Luis R. Rodriguez <[email protected]> wrote:
>
> On Thu, Aug 25, 2016 at 09:43:13AM +0200, Christoph Hellwig wrote:
> > The idea seems useful, but I reallt don't like the 'reverse-engineering'
> > approach.
> >
> > If we want to this properly from the ground up we should just split out
> > our CONFIG_ SYMBOLS into
> >
> > MODULE_* - builds exactly one module (tristate, or maybe also as a built-in
> > only one, then like a bool)
> >
> > CONFIG_* - just bool, MODULE_ may depend on it, too.
>
> Curious what does the split buy us if the real meaningful input is the value
> assigned to the config ? Ie, MODULE_FOO=m would be the modules we want to
> check for.
Also, unless you have kernel sources I don't think you could still
easily infer which MODULE_* config option enabled the module, could
you? And this was one of the original goals with this patch set. I
still think extending the modinfo section would be needed to allow
udevadm info to easily extract the information, no?
> > The other nice thing is that we could probably fold most of the Makefiles
> > into Kconfig using that methods as well, by listing the objectes required
> > for a module, e.g.
>
> OK If the Kconfig file has the objects listed I can see the gain of using
> Kconfig then to more easily map out to a symbol, given doing this on Makefiles
> is not straight forward.
>
> > module NVME_TARGET
> > tristate "NVMe Target support"
> > depends on BLOCK
> > depends on CONFIGFS_FS
> > name nvmet
> > objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
> > objects discovery.o
> >
> > module NVME_TARGET_LOOP
> > tristate "NVMe loopback device support"
> > depends on BLK_DEV_NVME
> > depends on NVME_TARGET
> > select NVME_FABRICS
> > select SG_POOL
> > name nvme-loop
> > objects loop.o
>
> I can see a huge win of having a direct specification that provides as a
> feature two way mapping from CONFIG <--> module (objects) and backwards again
> easily and clearly without hacks, specially if upon boot then we can then
> provide the precise kernel configuration you need, for both built-in and
> modules. The above could help with modules -- for built-in reverse mapping
> we'd need something else, perhaps a configurable option to keep tabs on
> inits called with associated configs.
>
> The above would be a pretty intrusive change though, in comparison to
> Cristina's original approach.
I think this is the biggest issue so far, and that we have no code to
showcase for it. Meanwhile pegging at least a kconfig symbol to
modinfo for at least a lot of modules would help considerably for
other use cases mentioned.
> The reverse-engineering object --> config
> aspect of her work and of the old scripts/kconfig/streamline_config.pl
> explains why it was hard. I'd be curious to learn of other gains possible
> other than those listed so far, if we had this.
>
> Re-iterating gains of having a simple two way CONFIG <--> module (objects)
> mapping (following the above proposal now):
>
> a) When optimizing build requirements for a kernel for a system.
> That is you boot into a distro kernel and then want to build
> a slim kernel only with sensible kernel configuration options.
>
> b) When you are on a distribution kernel but the distribution
> kernel provided lacks hardware support for your device, you
> may either want to upgrade the full kernel in which case you
> want to do a) or -- you may want to just a backports release
> which provides just the modules you need, you'd use it on top
> of the distribution kernel.
>
> c) Having the mapping in sysfs would allow to simplify
> streamline_config.pl avoid parsing Makefiles in perl. (From Michal)
It took me a while to recall where this was and why after 3 years of
not following up on this thread. So in case others fall into the same
boat, the sysfs gain here is that the modinfo section could be
readable then on any module's:
/sys/module/e1000/section/kconfig_symb
> d) Fold most of the Makefiles into Kconfig
>
> In retrospect c) still seems related to a) as we'd do away with
> the hacks completely needed by streamline_config.pl, a) can be
> augmented if we figure out a built-in solution as well.
>
> d) Just seems like collateral of a more precise mapping than
> what a Makefile provides. Anything else ?
In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
on this, I'm inclined to believe *at least* having some kconfig_symb
exposed for some modules is better than nothing. Christoph are you
totally opposed to this effort until we get a non-reverse engineered
effort in place? It just seems like an extraordinary amount of work
and I'm not quite sure who's volunteering to do it.
Other stakeholders may benefit from at least having some config -->
module mapping for now. Not just backports or building slimmer
kernels.
Luis
On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <[email protected]> wrote:
> In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> on this, I'm inclined to believe *at least* having some kconfig_symb
> exposed for some modules is better than nothing. Christoph are you
> totally opposed to this effort until we get a non-reverse engineered
> effort in place? It just seems like an extraordinary amount of work
> and I'm not quite sure who's volunteering to do it.
>
> Other stakeholders may benefit from at least having some config -->
> module mapping for now. Not just backports or building slimmer
> kernels.
Christoph, *poke*
Luis
On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <[email protected]> wrote:
> > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > on this, I'm inclined to believe *at least* having some kconfig_symb
> > exposed for some modules is better than nothing. Christoph are you
> > totally opposed to this effort until we get a non-reverse engineered
> > effort in place? It just seems like an extraordinary amount of work
> > and I'm not quite sure who's volunteering to do it.
> >
> > Other stakeholders may benefit from at least having some config -->
> > module mapping for now. Not just backports or building slimmer
> > kernels.
>
> Christoph, *poke*
Yes, I'm still totally opposed to a half-backed hack like this.
On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <[email protected]> wrote:
>
> On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <[email protected]> wrote:
> > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > exposed for some modules is better than nothing. Christoph are you
> > > totally opposed to this effort until we get a non-reverse engineered
> > > effort in place? It just seems like an extraordinary amount of work
> > > and I'm not quite sure who's volunteering to do it.
> > >
> > > Other stakeholders may benefit from at least having some config -->
> > > module mapping for now. Not just backports or building slimmer
> > > kernels.
> >
> > Christoph, *poke*
>
> Yes, I'm still totally opposed to a half-backed hack like this.
The solution puts forward a mechanism to add a kconfig_symb where we
are 100% certain we have a direct module --> config mapping.
This is *currently* determined when the streamline_config.pl finds
that an object has only *one* associated config symbol associated. As
Cristina noted, of 62 modules on a running system 58 of them ended up
getting the kconfig_symb assigned, that is 93.5% of all modules on the
system being tested. For the other modules, if they did want this
association, we could allow a way for modules to define their own
KBUILD_KCONF variable so that this could be considered as well, or
they can look at their own kconfig stuff to try to fit the model that
does work. To be clear, the heuristics *can* be updated if there is
confidence in alternative methods for resolution. But since it is
reflective of our current situation, I cannot consider it a hack.
This implementation is a reflection of our reality in the kernel, and
as has been discussed in this thread, if we want to correct the gaps
we need to do a lot of work. And *no one* is working towards these
goals.
That said, even if you go forward with an intrusive solution like the
one you proposed we could still use the same kconfig_symb...
So no, I don't see this as a hack. It's a reflection as to our current
reality. And I cannot see how the kconfig_symb can lie or be
incorrect. So in fact I think that pushing this forward also makes the
problem statement clearer for the future of what semantics needs to be
addressed, and helps us even annotate the problematic areas of the
kernel.
What negative aspects do you see with this being merged in practice?
Luis
On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <[email protected]> wrote:
> >
> > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <[email protected]> wrote:
> > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > exposed for some modules is better than nothing. Christoph are you
> > > > totally opposed to this effort until we get a non-reverse engineered
> > > > effort in place? It just seems like an extraordinary amount of work
> > > > and I'm not quite sure who's volunteering to do it.
> > > >
> > > > Other stakeholders may benefit from at least having some config -->
> > > > module mapping for now. Not just backports or building slimmer
> > > > kernels.
> > >
> > > Christoph, *poke*
> >
> > Yes, I'm still totally opposed to a half-backed hack like this.
>
> The solution puts forward a mechanism to add a kconfig_symb where we
> are 100% certain we have a direct module --> config mapping.
>
> This is *currently* determined when the streamline_config.pl finds
> that an object has only *one* associated config symbol associated. As
> Cristina noted, of 62 modules on a running system 58 of them ended up
> getting the kconfig_symb assigned, that is 93.5% of all modules on the
> system being tested. For the other modules, if they did want this
> association, we could allow a way for modules to define their own
> KBUILD_KCONF variable so that this could be considered as well, or
> they can look at their own kconfig stuff to try to fit the model that
> does work. To be clear, the heuristics *can* be updated if there is
> confidence in alternative methods for resolution. But since it is
> reflective of our current situation, I cannot consider it a hack.
>
> This implementation is a reflection of our reality in the kernel, and
> as has been discussed in this thread, if we want to correct the gaps
> we need to do a lot of work. And *no one* is working towards these
> goals.
>
> That said, even if you go forward with an intrusive solution like the
> one you proposed we could still use the same kconfig_symb...
>
> So no, I don't see this as a hack. It's a reflection as to our current
> reality. And I cannot see how the kconfig_symb can lie or be
> incorrect. So in fact I think that pushing this forward also makes the
> problem statement clearer for the future of what semantics needs to be
> addressed, and helps us even annotate the problematic areas of the
> kernel.
>
> What negative aspects do you see with this being merged in practice?
I'm trying to see what the actual problem that you are wanting to solve
here with this. What exactly is it? Who needs to determine the
"singular" configuration option that caused a kernel module to be built
at the expense of all other options? What can that be used for and who
will use it?
confused,
greg k-h
On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <[email protected]> wrote:
> > >
> > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <[email protected]> wrote:
> > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > and I'm not quite sure who's volunteering to do it.
> > > > >
> > > > > Other stakeholders may benefit from at least having some config -->
> > > > > module mapping for now. Not just backports or building slimmer
> > > > > kernels.
> > > >
> > > > Christoph, *poke*
> > >
> > > Yes, I'm still totally opposed to a half-backed hack like this.
> >
> > The solution puts forward a mechanism to add a kconfig_symb where we
> > are 100% certain we have a direct module --> config mapping.
> >
> > This is *currently* determined when the streamline_config.pl finds
> > that an object has only *one* associated config symbol associated. As
> > Cristina noted, of 62 modules on a running system 58 of them ended up
> > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > system being tested. For the other modules, if they did want this
> > association, we could allow a way for modules to define their own
> > KBUILD_KCONF variable so that this could be considered as well, or
> > they can look at their own kconfig stuff to try to fit the model that
> > does work. To be clear, the heuristics *can* be updated if there is
> > confidence in alternative methods for resolution. But since it is
> > reflective of our current situation, I cannot consider it a hack.
> >
> > This implementation is a reflection of our reality in the kernel, and
> > as has been discussed in this thread, if we want to correct the gaps
> > we need to do a lot of work. And *no one* is working towards these
> > goals.
> >
> > That said, even if you go forward with an intrusive solution like the
> > one you proposed we could still use the same kconfig_symb...
> >
> > So no, I don't see this as a hack. It's a reflection as to our current
> > reality. And I cannot see how the kconfig_symb can lie or be
> > incorrect. So in fact I think that pushing this forward also makes the
> > problem statement clearer for the future of what semantics needs to be
> > addressed, and helps us even annotate the problematic areas of the
> > kernel.
> >
> > What negative aspects do you see with this being merged in practice?
>
> I'm trying to see what the actual problem that you are wanting to solve
> here with this. What exactly is it?
The problem is that there is no current maping of a module to respective
kconfig symbol.
> Who needs to determine the
> "singular" configuration option that caused a kernel module to be built
> at the expense of all other options?
Folks wanting to slim down their kernel build, and users of backports.
> What can that be used for and who will use it?
Without a mapping there is no clean way to let you slim down your kernel
using a distro kernel as a base, enabling only those things you really
need.
Historically those interested would just have streamline_config.pl at
their disposal, but it isn't perfect. Although Cristina extended
streamline_config.pl, the proper approach here would be to see if the
heuristics in streamline_config.pl which she made use of to map a config
symbol for a module could be written in C. The safe heuristic used is
that for modules where there was only one kconfig symbol associated,
we'd be certain that kconfig symbol was the one needed for that module.
A side benefit is also that by having a modinfo section for the kconfib
symbol we'd have access to it via sysfs, an example is
/sys/module/e1000/section/kconfig_symb, and then streamline_config.pl
could be simplified further by not having to parse Makefiles in perl.
The reason for using a modinfo section also is you could get the
information about configs needed for modules without requiring your
kernel sources, and this information then also becomes available to
augment udevadm for its own hardware db info.
Another set of users of this would be users of backports. For instance
users on distro kernels but their current kernel lacks updated drivers
for one of your devices, and you want to use a backports release, but
just want to build / only the drivers you really need for the release.
Backports then could easily scrape for the sysfs attribute / udevadm db
to let you build and install only the modules you need.
Long term users are developers working on trying to clear up kconfig
semantics. Having a clear indicator in our tree about modules where a
direct mapping was not possible (say with a KBUILD_KCONF definition in
their Makefiles to clarify which config symbol enables it) gives us an
idea of kconfig entries which may be contrived and could use some
clarifications.
Luis
On Tue, Jul 02, 2019 at 08:51:06PM +0000, Luis Chamberlain wrote:
> On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> > On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <[email protected]> wrote:
> > > >
> > > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <[email protected]> wrote:
> > > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > > and I'm not quite sure who's volunteering to do it.
> > > > > >
> > > > > > Other stakeholders may benefit from at least having some config -->
> > > > > > module mapping for now. Not just backports or building slimmer
> > > > > > kernels.
> > > > >
> > > > > Christoph, *poke*
> > > >
> > > > Yes, I'm still totally opposed to a half-backed hack like this.
> > >
> > > The solution puts forward a mechanism to add a kconfig_symb where we
> > > are 100% certain we have a direct module --> config mapping.
> > >
> > > This is *currently* determined when the streamline_config.pl finds
> > > that an object has only *one* associated config symbol associated. As
> > > Cristina noted, of 62 modules on a running system 58 of them ended up
> > > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > > system being tested. For the other modules, if they did want this
> > > association, we could allow a way for modules to define their own
> > > KBUILD_KCONF variable so that this could be considered as well, or
> > > they can look at their own kconfig stuff to try to fit the model that
> > > does work. To be clear, the heuristics *can* be updated if there is
> > > confidence in alternative methods for resolution. But since it is
> > > reflective of our current situation, I cannot consider it a hack.
> > >
> > > This implementation is a reflection of our reality in the kernel, and
> > > as has been discussed in this thread, if we want to correct the gaps
> > > we need to do a lot of work. And *no one* is working towards these
> > > goals.
> > >
> > > That said, even if you go forward with an intrusive solution like the
> > > one you proposed we could still use the same kconfig_symb...
> > >
> > > So no, I don't see this as a hack. It's a reflection as to our current
> > > reality. And I cannot see how the kconfig_symb can lie or be
> > > incorrect. So in fact I think that pushing this forward also makes the
> > > problem statement clearer for the future of what semantics needs to be
> > > addressed, and helps us even annotate the problematic areas of the
> > > kernel.
> > >
> > > What negative aspects do you see with this being merged in practice?
> >
> > I'm trying to see what the actual problem that you are wanting to solve
> > here with this. What exactly is it?
>
> The problem is that there is no current maping of a module to respective
> kconfig symbol.
That's because it is not just "one" symbol per module.
If it were, you can just parse the Makefiles and get that single config
option for most modules, right? But even then, multiple options can
influence a single module as to what actually gets built into that
module.
So, I would say, "who really cares"?
> > Who needs to determine the
> > "singular" configuration option that caused a kernel module to be built
> > at the expense of all other options?
>
> Folks wanting to slim down their kernel build, and users of backports.
People who want to "slim" down things are rare, and it's usually worth
it to work backwards anyway (see what functionality is needed and then
go from there, not look at the modules themselves). Or use a tool like
'make localmodconfig' and trim.
> > What can that be used for and who will use it?
>
> Without a mapping there is no clean way to let you slim down your kernel
> using a distro kernel as a base, enabling only those things you really
> need.
It's hard to determine "what you really need" :)
Use localmodconfig and you have a great start, then prune from there.
Trying to put _all_ configuration dependencies in a single module isn't
going to work, our configuration language does not distill down to that.
thanks,
greg k-h
On 28.06.19 20:40, Luis Chamberlain wrote:
Hi folks,
> The solution puts forward a mechanism to add a kconfig_symb where we> are 100% certain we have a direct module --> config mapping.
Okay, but IIRC this will add more boilerplate those modules.
And I wonder whether target binaries are the right place for those
things at all - IMHO that's something one wants to derive from the
source code / .config's. At least in the cases I'm imagining, I don't
even have an actual kernel running on the corresponding target yet.
(eg. in crosscompile situations)
OTOH, a more pressing problem for me is identifying the right drivers
and corresponding config options (usually plural, as certain subsystems
have to be enabled, too) by hardware information like DT, ACPI, DMI,
PCI, etc. For now, I have to do that manually, which is pretty time
consuming.
In embedded world, we often have scenarios where we want a really
minimal kernel, but need to enable/disable certain hi-level peripherals
in the middle of the project (eg. "oh, we also need ethernet, but we
wanna drop usb"). There we'll have to find out what actual chip is,
its corresponding driver, required subsystems, etc, and also kick off
everything we don't need anymore.
I've thought about implementing some actual dependency tracking
(at least recording the auto-enabled symbols), but didn't expect that
to become practically usable anytime soon, so I went for a different
approach: writing a little tool that allows modeling hilevel features
and corresponding (potentially board-specific) config syms, so the whole
.config for certain board and usecase can be autogenerated by just some
small meta-configuration:
https://github.com/metux/kmct
Maybe this could also help for your usecase ?
--mtx
--
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
[email protected] -- +49-151-27565287
On Wed, Jul 03, 2019 at 09:40:48AM +0200, Greg Kroah-Hartman wrote:
> On Tue, Jul 02, 2019 at 08:51:06PM +0000, Luis Chamberlain wrote:
> > On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> > > On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > > > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <[email protected]> wrote:
> > > > >
> > > > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <[email protected]> wrote:
> > > > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > > > and I'm not quite sure who's volunteering to do it.
> > > > > > >
> > > > > > > Other stakeholders may benefit from at least having some config -->
> > > > > > > module mapping for now. Not just backports or building slimmer
> > > > > > > kernels.
> > > > > >
> > > > > > Christoph, *poke*
> > > > >
> > > > > Yes, I'm still totally opposed to a half-backed hack like this.
> > > >
> > > > The solution puts forward a mechanism to add a kconfig_symb where we
> > > > are 100% certain we have a direct module --> config mapping.
> > > >
> > > > This is *currently* determined when the streamline_config.pl finds
> > > > that an object has only *one* associated config symbol associated. As
> > > > Cristina noted, of 62 modules on a running system 58 of them ended up
> > > > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > > > system being tested. For the other modules, if they did want this
> > > > association, we could allow a way for modules to define their own
> > > > KBUILD_KCONF variable so that this could be considered as well, or
> > > > they can look at their own kconfig stuff to try to fit the model that
> > > > does work. To be clear, the heuristics *can* be updated if there is
> > > > confidence in alternative methods for resolution. But since it is
> > > > reflective of our current situation, I cannot consider it a hack.
> > > >
> > > > This implementation is a reflection of our reality in the kernel, and
> > > > as has been discussed in this thread, if we want to correct the gaps
> > > > we need to do a lot of work. And *no one* is working towards these
> > > > goals.
> > > >
> > > > That said, even if you go forward with an intrusive solution like the
> > > > one you proposed we could still use the same kconfig_symb...
> > > >
> > > > So no, I don't see this as a hack. It's a reflection as to our current
> > > > reality. And I cannot see how the kconfig_symb can lie or be
> > > > incorrect. So in fact I think that pushing this forward also makes the
> > > > problem statement clearer for the future of what semantics needs to be
> > > > addressed, and helps us even annotate the problematic areas of the
> > > > kernel.
> > > >
> > > > What negative aspects do you see with this being merged in practice?
> > >
> > > I'm trying to see what the actual problem that you are wanting to solve
> > > here with this. What exactly is it?
> >
> > The problem is that there is no current maping of a module to respective
> > kconfig symbol.
>
> That's because it is not just "one" symbol per module.
This is true. But it is not the case for all modules. In fact it seems
its true that most modules do have *one* main symbol.
On at least Cristina's system of of 62 modules 58 *did* have one symbol.
For the modules evaluated where this was not the case, it did seem wise
to actually revise the symbol definition for the other modules.
> If it were, you can just parse the Makefiles and get that single config
> option for most modules, right?
The heuristic essentially does this and only provides the module
attribute where this was true.
> But even then, multiple options can
> influence a single module as to what actually gets built into that
> module.
Yes. For example one hardware device driver may support different
families of chipsets, and so it could have sub-options for each
family.
> So, I would say, "who really cares"?
For most visible modules it would seem we do have a one config symbol
mapping which could enable it. And I noted who would care. The defaults
of a module, for instance sub-options to enable / disable different
support for different chipsets, *should* suffice for most users.
> > > Who needs to determine the
> > > "singular" configuration option that caused a kernel module to be built
> > > at the expense of all other options?
> >
> > Folks wanting to slim down their kernel build, and users of backports.
>
> People who want to "slim" down things are rare,
It is basic math though:
Users of 'make localmodconfig' + backport users > Users of 'make localmodconfig'
And yet we already support 'make localmodconfig'. So what is being
proposed can help enhance 'make localmodconfig' and yet provides more
users outside of those users, without requiring kernel sources.
> and it's usually worth
> it to work backwards anyway (see what functionality is needed and then
> go from there, not look at the modules themselves). Or use a tool like
> 'make localmodconfig' and trim.
And I am noting we can further enhance a feature which we already
do support, and enable *more* users requiring similar information.
> > > What can that be used for and who will use it?
> >
> > Without a mapping there is no clean way to let you slim down your kernel
> > using a distro kernel as a base, enabling only those things you really
> > need.
>
> It's hard to determine "what you really need" :)
Right, but at least for device functionality, the above would help
significantly. It also poses the question whether or not device drivers
*should* strive towards having one kconfig symbol to help with this.
There is a lot of research over the lack of proper semantics on use of
kconfig, and issues this causes. It is so bad that some researchers have
tried define our semantics through *reverse engineering*. The question
of whether or not we *should* strive to have *one* symbol per a driver
for an actual hardware component is worth evaluating long term, for the
sake of helping with semantics of kconfig use. I see there being gains
with this, and I find it hard to find counters to where quite the
opposite is true. Can you?
> Use localmodconfig and you have a great start, then prune from there.
This thread poses the question if we can do better, and suggests one
small area where we can start.
> Trying to put _all_ configuration dependencies in a single module isn't
> going to work, our configuration language does not distill down to that.
The question we should be evaluating if we *should* strive to buckle up
on this and have at least one config symbol per module associated with
hardware. I'm suggesting there are gains for this, and am providing two
groups of users that would benefit from this clearly. And I'm also
suggesting that it could help with kconfig semantics, long term.
Luis
On Wed, Jul 03, 2019 at 02:16:44PM +0200, Enrico Weigelt, metux IT consult wrote:
> On 28.06.19 20:40, Luis Chamberlain wrote:
>
> Hi folks,
>
> > The solution puts forward a mechanism to add a kconfig_symb where
> > we are 100% certain we have a direct module --> config mapping.
> Okay, but IIRC this will add more boilerplate those modules.
Just one module attribute.
> And I wonder whether target binaries are the right place for those
> things at all - IMHO that's something one wants to derive from the
> source code / .config's.
For the use cases mentioned for why the module attribute is being
suggested it would help to not have to download kernel sources. The
only question we want to answer is: for the hardware components
present on this system, which configs options do I need to enable
to support these components?
> At least in the cases I'm imagining, I don't
> even have an actual kernel running on the corresponding target yet.
> (eg. in crosscompile situations)
Right, so this is a different use case. A *running* kernel provides
*more* information, specially if it had enabled on it modules which
enable hardware.
> OTOH, a more pressing problem for me is identifying the right drivers
> and corresponding config options (usually plural, as certain subsystems
> have to be enabled, too) by hardware information like DT, ACPI, DMI,
> PCI, etc.
Right, I'm familiar with the problem.
At least for virtualization we decided to support at least these two to
help:
* make kvmconfig
* make xenconfig
I'm not suggesting we need a respective target per major component you
have described. I'm mentioning these as examples of ways to also address
similar problems.
> For now, I have to do that manually, which is pretty time
> consuming.
Yes...
Similar problem would be found if one wanted to find a desirable kernel
config for a remote system. One should be able to somehow scrape some
hardware information, dump that to a file, and then somehow generate
a working config for that system.
The module attribute being suggested would enable at least one way
to gather some of the required config symbols: symbols for *hardware*
and where one can run a modern kernel, with many features / hardware
enabled already.
Addressing the problem of obtaining a kernel config for a system where
one cannot run any kernel should be possible, however this *is* a very
different problem.
However, folks producing embedded systems *do* / *should* have a lot of
knowledge of their systems, and so the type of scheme you have devised
seems sensible for it.
> In embedded world, we often have scenarios where we want a really
> minimal kernel, but need to enable/disable certain hi-level peripherals
> in the middle of the project (eg. "oh, we also need ethernet, but we
> wanna drop usb"). There we'll have to find out what actual chip is,
> its corresponding driver, required subsystems, etc, and also kick off
> everything we don't need anymore.
Right. One *should* be able to tell some tool, hey, here is the list of
my desirable .config options. Go and figure out what I need to make that
work and give me a resulting .config. Its not easy.
> I've thought about implementing some actual dependency tracking
> (at least recording the auto-enabled symbols), but didn't expect that
> to become practically usable anytime soon,
The ability to easily ask the kernel to enable the components needed
for a respective config option *is* very useful but indeed not easy.
This is not the only space where this problem exists. Similar problem
exists for distribution packages, and dependencies. Challenges have
been made for proper research towards these problems, and such research
has lead distributions to opt to enable some of these algorithms.
This begs the question if we could learn from similar efforts on Linux
for these sorts of questions. One possibility here is to evaluate the
prospect of using a SAT solver with Minimally Unsatisfiable Subformulas
(MUSes) support, which should be be able to address thir problem. This
prospect is ongoing and currrent R&D is active, for details refer to:
https://kernelnewbies.org/KernelProjects/kconfig-sat
I'd recommend the mailing list, as that is where folks are following up
on, and there seems to be a call scheduled next Wednesday at 8:30am PST
about some new developments.
> so I went for a different
> approach: writing a little tool that allows modeling hilevel features
> and corresponding (potentially board-specific) config syms, so the whole
> .config for certain board and usecase can be autogenerated by just some
> small meta-configuration:
>
> https://github.com/metux/kmct
>
> Maybe this could also help for your usecase ?
It certainly can be useful for components, ie, not hardware. But for
hardware a one-to-one mapping of one driver to one config would be of
much more use.
Based on a quick glance, I'd suggest that it would be great if could
have something similar upstream on linux. It would be unclear yet how to
structure this though. It would be wonderful if for instance kconfig
supported a way to group a major set of components under *one* config
symbol and then say: "I want this major component enabled" and then it'd
go and enable all the defaults which would be required for it. Its not
an easy problem, given though that there may be certain other symbols
which may conflict with your target component. An example is if you
wanted to enable PCI on a system which didn't support it. Because of
this, it seems you'd want *all* desirable configs and let a piece of
software figure out what you need / can enable. And.. this is precisely
where the SAT solver with MUSes could help...
Luis
On Wed, Jul 03, 2019 at 04:50:20PM +0000, Luis Chamberlain wrote:
> On Wed, Jul 03, 2019 at 09:40:48AM +0200, Greg Kroah-Hartman wrote:
> > On Tue, Jul 02, 2019 at 08:51:06PM +0000, Luis Chamberlain wrote:
> > > On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> > > > On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > > > > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <[email protected]> wrote:
> > > > > >
> > > > > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <[email protected]> wrote:
> > > > > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > > > > and I'm not quite sure who's volunteering to do it.
> > > > > > > >
> > > > > > > > Other stakeholders may benefit from at least having some config -->
> > > > > > > > module mapping for now. Not just backports or building slimmer
> > > > > > > > kernels.
> > > > > > >
> > > > > > > Christoph, *poke*
> > > > > >
> > > > > > Yes, I'm still totally opposed to a half-backed hack like this.
> > > > >
> > > > > The solution puts forward a mechanism to add a kconfig_symb where we
> > > > > are 100% certain we have a direct module --> config mapping.
> > > > >
> > > > > This is *currently* determined when the streamline_config.pl finds
> > > > > that an object has only *one* associated config symbol associated. As
> > > > > Cristina noted, of 62 modules on a running system 58 of them ended up
> > > > > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > > > > system being tested. For the other modules, if they did want this
> > > > > association, we could allow a way for modules to define their own
> > > > > KBUILD_KCONF variable so that this could be considered as well, or
> > > > > they can look at their own kconfig stuff to try to fit the model that
> > > > > does work. To be clear, the heuristics *can* be updated if there is
> > > > > confidence in alternative methods for resolution. But since it is
> > > > > reflective of our current situation, I cannot consider it a hack.
> > > > >
> > > > > This implementation is a reflection of our reality in the kernel, and
> > > > > as has been discussed in this thread, if we want to correct the gaps
> > > > > we need to do a lot of work. And *no one* is working towards these
> > > > > goals.
> > > > >
> > > > > That said, even if you go forward with an intrusive solution like the
> > > > > one you proposed we could still use the same kconfig_symb...
> > > > >
> > > > > So no, I don't see this as a hack. It's a reflection as to our current
> > > > > reality. And I cannot see how the kconfig_symb can lie or be
> > > > > incorrect. So in fact I think that pushing this forward also makes the
> > > > > problem statement clearer for the future of what semantics needs to be
> > > > > addressed, and helps us even annotate the problematic areas of the
> > > > > kernel.
> > > > >
> > > > > What negative aspects do you see with this being merged in practice?
> > > >
> > > > I'm trying to see what the actual problem that you are wanting to solve
> > > > here with this. What exactly is it?
> > >
> > > The problem is that there is no current maping of a module to respective
> > > kconfig symbol.
> >
> > That's because it is not just "one" symbol per module.
>
> This is true. But it is not the case for all modules. In fact it seems
> its true that most modules do have *one* main symbol.
You mean "one unique symbol from all other modules", right?
That is much different than just "one" symbol, given that almost
every driver depends on something else being enabled as well (bus type,
platform type, arch, etc.)
And I would argue, that finding that "one" symbol is easy, just parse
the Makefiles. But I would also state that this "one" symbol doesn't
really help you much as those are the "simple" things. It's how to
turn on all of the required symbols to get to that "one" symbol that is
the hard part.
And conversely, if you disable that "one" symbol, does that also mean
you can disable the symbols it depended on? If so, how far back?
And what about functionality? If my usb-storage device is "enabled" in
the build, yet all filesystems are not, or the needed dm module is not,
it is useless. Hardware requires usually more than one real "symbol" in
order to work properly, as you know.
And of course, what does this really matter to anyone? If you build
"all modules" and you only load the modules you actually use for your
hardware (based on auto-loading), then your system uses the same amount
of memory as if you disabled all of the modules you did not need.
Yes, it's faster to build, but is that what you are trying to optimize
for here?
Anyway, if this is just an acidemic thing, have fun, but I would not be
adding anything else to the module image that is not really going to be
useful to anyone.
good luck!
greg k-h
On 03.07.19 19:35, Luis Chamberlain wrote:
Hi,
>> Okay, but IIRC this will add more boilerplate those modules.
>
> Just one module attribute.
Yes, but still one per module. This raises the question whether
maintainers are willing to cope w/ tons of tiny patches for just
one line - for something that will take quite some time to become
actually useful (doesn't help much if only few drivers support it),
and is only helpful in a few use cases.
And to make it really useful, we also need some way to automatically
derive which other symbols to enable (subsystems, etc), w/o auto-
enabling stuff one doesn't need here. (are the defaults sane for
this usecase ?)
The main problem here, IMHO, is that the kconfig system doesn't really
know what makes up a module (it only knows that something w =y cant
depend on something thats =m).
So it smalls like we'd need some config language that really understands
things like modules, subsystems, arches, etc with their properties and
is used by both kconfig and kbuild. Then we could put all metadata there
instead of the current macro calls. At that point we could also put
things like match tables in here, which would solve the problem of
finding the right driver by hardware descriptions.
But that's really a *big* topic, it's not easy.
>> And I wonder whether target binaries are the right place for those
>> things at all - IMHO that's something one wants to derive from the
>> source code / .config's.
>
> For the use cases mentioned for why the module attribute is being
> suggested it would help to not have to download kernel sources.
Are we still talking about compiling custom kernels ?
(how to do that w/o source code ?)
> The only question we want to answer is: for the hardware components
> present on this system, which configs options do I need to enable
> to support these components?
What else would one need that data, if not compiling a custom kernel
(which in turn needs the source) ?
> At least for virtualization we decided to support at least these two to
> help:
>
> * make kvmconfig
> * make xenconfig
These two are rather simple. Most times there isn't much variance in
virtual hardware (unless one starts directly mapping in pci or usb
devices ...)
> Similar problem would be found if one wanted to find a desirable kernel
> config for a remote system. One should be able to somehow scrape some
> hardware information, dump that to a file, and then somehow generate
> a working config for that system.
Yes. That's actually pretty much the same usecase (in my case I'd also
have dts, lspci/lsusb output, etc)
> The module attribute being suggested would enable at least one way
> to gather some of the required config symbols: symbols for *hardware*
> and where one can run a modern kernel, with many features / hardware
> enabled already.
But only for a pretty specific usecase. I'm not opposed to this, but
I wonder whether maintainers are willing to accept that stuff for just
that specific usecase.
> However, folks producing embedded systems *do* / *should* have a lot of
> knowledge of their systems, and so the type of scheme you have devised
> seems sensible for it.
Usually we have (unless we need to do reverse engineering :o). But it's
a pretty time-consuming task. Especially if the requirements change
several times in the development or lifetime of a specific product.
For example "oh, we now need eth", "naah, we don't wanna use usb
anymore", "let's take a different SoM", ... not that have pretty
orthogonal sets of configs we need to maintain: hardware- and non-
hardware-related ones. And hardware-related ones can fall into different
categories like fixed-attached/onboard vs. hotpluggable ones.
Recently I had a case where the customer requested xattr support, so
I had to enable general xattr support as well as per-filesystem.
Pretty simple, but having lots of those cases quickly sums up. One of
the reasons why I've written my own little config generator.
>> In embedded world, we often have scenarios where we want a really
>> minimal kernel, but need to enable/disable certain hi-level peripherals
>> in the middle of the project (eg. "oh, we also need ethernet, but we
>> wanna drop usb"). There we'll have to find out what actual chip is,
>> its corresponding driver, required subsystems, etc, and also kick off
>> everything we don't need anymore.
>
> Right. One *should* be able to tell some tool, hey, here is the list of
> my desirable .config options. Go and figure out what I need to make that
> work and give me a resulting .config. Its not easy.
I think I've already got into a pretty usable state - at least for my
projects. For now only supports a few boards and limited set of
features, but patches are always welcomed :)
>> I've thought about implementing some actual dependency tracking
>> (at least recording the auto-enabled symbols), but didn't expect that
>> to become practically usable anytime soon,
>
> The ability to easily ask the kernel to enable the components needed
> for a respective config option *is* very useful but indeed not easy.
Yes, it would need to understand things like conditional definitions
to deduce that certain things need to be enabled first, before certain
drivers become choosable.
> This is not the only space where this problem exists. Similar problem
> exists for distribution packages, and dependencies. Challenges have
> been made for proper research towards these problems, and such research
> has lead distributions to opt to enable some of these algorithms.
The problem w/ dependencies is that there can be different types of
dependencies, as well as different types of software objects. Just
solving the expressions is only a part of the problem.
> This begs the question if we could learn from similar efforts on Linux
> for these sorts of questions. One possibility here is to evaluate the
> prospect of using a SAT solver with Minimally Unsatisfiable Subformulas
> (MUSes) support, which should be be able to address thir problem. This
> prospect is ongoing and currrent R&D is active, for details refer to:
>
> https://kernelnewbies.org/KernelProjects/kconfig-sat
Good tip, I'll have a look at it.
> It certainly can be useful for components, ie, not hardware. But for
> hardware a one-to-one mapping of one driver to one config would be of
> much more use.
Unfortunately, we don't have this 1:1 mapping. Often drivers support
different sets of devices, depending on other factors, sometimes sub-
options (eg. different hw versions), sometimes depending on other
subsystems, sometimes arch-specific, etc, etc.
I think we should work towards that, but I doubt we'd reach that goal
anytime soon, and begs the question whether it's really worth all the
effort required for that.
> It would be wonderful if for instance kconfig
> supported a way to group a major set of components under *one* config
> symbol and then say: "I want this major component enabled" and then it'd
> go and enable all the defaults which would be required for it.
Yes, thought about that, too. For example have syms for selecting whole
boards and features of them - a bit like this:
--> Preconfigure for specific boards
--> board A
--> board B
...
--> Enable board features
--> Ethernet port
--> Display
--> Touch panel
--> Audio
....
BUT: this would turn into maintenance hell, so I dropped that idea.
> An example is if you
> wanted to enable PCI on a system which didn't support it. Because of
> this, it seems you'd want *all* desirable configs and let a piece of
> software figure out what you need / can enable. And.. this is precisely
> where the SAT solver with MUSes could help...
Yes, but this piece of software first needs to know whether eg. PCI
is available on that HW. Oh, and things like PCI could be a dependency
as well as an feature on its own, depending on how you gonna use it.
(eg. if directly access from userland or VMs).
--mtx
--
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
[email protected] -- +49-151-27565287
On Wed, Jul 03, 2019 at 08:57:58PM +0200, Greg Kroah-Hartman wrote:
> On Wed, Jul 03, 2019 at 04:50:20PM +0000, Luis Chamberlain wrote:
> > On Wed, Jul 03, 2019 at 09:40:48AM +0200, Greg Kroah-Hartman wrote:
> > > On Tue, Jul 02, 2019 at 08:51:06PM +0000, Luis Chamberlain wrote:
> > > > On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> > > > > On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > > > > > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <[email protected]> wrote:
> > > > > > >
> > > > > > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > > > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <[email protected]> wrote:
> > > > > > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > > > > > and I'm not quite sure who's volunteering to do it.
> > > > > > > > >
> > > > > > > > > Other stakeholders may benefit from at least having some config -->
> > > > > > > > > module mapping for now. Not just backports or building slimmer
> > > > > > > > > kernels.
> > > > > > > >
> > > > > > > > Christoph, *poke*
> > > > > > >
> > > > > > > Yes, I'm still totally opposed to a half-backed hack like this.
> > > > > >
> > > > > > The solution puts forward a mechanism to add a kconfig_symb where we
> > > > > > are 100% certain we have a direct module --> config mapping.
> > > > > >
> > > > > > This is *currently* determined when the streamline_config.pl finds
> > > > > > that an object has only *one* associated config symbol associated. As
> > > > > > Cristina noted, of 62 modules on a running system 58 of them ended up
> > > > > > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > > > > > system being tested. For the other modules, if they did want this
> > > > > > association, we could allow a way for modules to define their own
> > > > > > KBUILD_KCONF variable so that this could be considered as well, or
> > > > > > they can look at their own kconfig stuff to try to fit the model that
> > > > > > does work. To be clear, the heuristics *can* be updated if there is
> > > > > > confidence in alternative methods for resolution. But since it is
> > > > > > reflective of our current situation, I cannot consider it a hack.
> > > > > >
> > > > > > This implementation is a reflection of our reality in the kernel, and
> > > > > > as has been discussed in this thread, if we want to correct the gaps
> > > > > > we need to do a lot of work. And *no one* is working towards these
> > > > > > goals.
> > > > > >
> > > > > > That said, even if you go forward with an intrusive solution like the
> > > > > > one you proposed we could still use the same kconfig_symb...
> > > > > >
> > > > > > So no, I don't see this as a hack. It's a reflection as to our current
> > > > > > reality. And I cannot see how the kconfig_symb can lie or be
> > > > > > incorrect. So in fact I think that pushing this forward also makes the
> > > > > > problem statement clearer for the future of what semantics needs to be
> > > > > > addressed, and helps us even annotate the problematic areas of the
> > > > > > kernel.
> > > > > >
> > > > > > What negative aspects do you see with this being merged in practice?
> > > > >
> > > > > I'm trying to see what the actual problem that you are wanting to solve
> > > > > here with this. What exactly is it?
> > > >
> > > > The problem is that there is no current maping of a module to respective
> > > > kconfig symbol.
> > >
> > > That's because it is not just "one" symbol per module.
> >
> > This is true. But it is not the case for all modules. In fact it seems
> > its true that most modules do have *one* main symbol.
>
> You mean "one unique symbol from all other modules", right?
I mean that in the typical case you have *one* main *parent* symbol defining
one device driver for one piece of hardware.
> That is much different than just "one" symbol, given that almost
> every driver depends on something else being enabled as well (bus type,
> platform type, arch, etc.)
Yes but that is different than what is trying to be addressed.
> And I would argue, that finding that "one" symbol is easy, just parse
> the Makefiles. But I would also state that this "one" symbol doesn't
> really help you much as those are the "simple" things. It's how to
> turn on all of the required symbols to get to that "one" symbol that is
> the hard part.
Absolutely, but *that* is a different problem! But I am glad you brought
that up and acknowledge it. Addressing that problem will take time and
folks are working towards it. Addressing that problem will still however
not address the problem being addressed here.
> And conversely, if you disable that "one" symbol, does that also mean
> you can disable the symbols it depended on? If so, how far back?
Right..
> And what about functionality? If my usb-storage device is "enabled" in
> the build, yet all filesystems are not, or the needed dm module is not,
> it is useless.
Yes. The localmodconfig approach is to keep enabled as modules *all* currently
enabled modules, that covers that.
But again, this is a separate problem. The one I am addressing, on
behalf of Cristina, is a subspace, dedicated towards *hardware*
functionality.
> Hardware requires usually more than one real "symbol" in
> order to work properly, as you know.
Right. This does not mean that this information of parent main symbol is
not useful. And having it available on the modules can help with
multiple goals, without requiring kernel sources.
> And of course, what does this really matter to anyone? If you build
> "all modules" and you only load the modules you actually use for your
> hardware (based on auto-loading), then your system uses the same amount
> of memory as if you disabled all of the modules you did not need.
For backports it means you can enable only what you need, or at least
show users what modules in a backports release could be upgraded and let
you enable / disable them. For other users, people can care about build
time. And not everyone has fancy systems to build all modules; and
sometimes you may just want to enable a small qemu system and build
locally on it, enabling all modules on such systems would just be
extremely silly.
> Yes, it's faster to build, but is that what you are trying to optimize
> for here?
Particularly for me, yes. Others have other needs and I have already
stated clearly what the gains are.
> Anyway, if this is just an acidemic thing, have fun, but I would not be
> adding anything else to the module image that is not really going to be
> useful to anyone.
The academic consensus is kconfig semantics are poo and we need to do
slowly start addressing this. I believe that striving towards having
one kconfig symbol per hw component can help long term, and in the
meantime, this also will help with the 'make localmodconfig' and
backport users.
> good luck!
This is not about luck. If you don't want to address these problems,
that's fine, but please provide objective considerations rather than
right out rejection without a reasonable basis. Or even better, provide
alternative recommendations.
The alternative Christoph recommended is hugely instrusive, no one is
working on it, and the simple solution proposed in this thread at least
gets us a small step forward in helping to enable a few more users,
while also postulating if we should strive towards having one main kconfig
for each hw component. Since it seems this is already the case, and
there are only a few outliers, this effort should help spot outliers
and address them to help with our semantics.
Luis
On Wed, Jul 03, 2019 at 09:31:33PM +0200, Enrico Weigelt, metux IT consult wrote:
> On 03.07.19 19:35, Luis Chamberlain wrote:
>
> Hi,
>
> >> Okay, but IIRC this will add more boilerplate those modules.
> >
> > Just one module attribute.
>
> Yes, but still one per module. This raises the question whether
> maintainers are willing to cope w/ tons of tiny patches for just
> one line - for something that will take quite some time to become
> actually useful (doesn't help much if only few drivers support it),
> and is only helpful in a few use cases.
This can be wrapped in kconfig and disabled by default.
> And to make it really useful, we also need some way to automatically
> derive which other symbols to enable (subsystems, etc), w/o auto-
> enabling stuff one doesn't need here. (are the defaults sane for
> this usecase ?)
No, that's a separate problem.
>
> The main problem here, IMHO, is that the kconfig system doesn't really
> know what makes up a module (it only knows that something w =y cant
> depend on something thats =m).
>
> So it smalls like we'd need some config language that really understands
> things like modules, subsystems, arches, etc with their properties and
> is used by both kconfig and kbuild. Then we could put all metadata there
> instead of the current macro calls. At that point we could also put
> things like match tables in here, which would solve the problem of
> finding the right driver by hardware descriptions.
>
> But that's really a *big* topic, it's not easy.
Christoph's sugggestion goes along the lines with addressing some of
this, yet 2 years havegone by and no one is working on it and its hugely
intrusive. I'd welcome the patches.
> >> And I wonder whether target binaries are the right place for those
> >> things at all - IMHO that's something one wants to derive from the
> >> source code / .config's.
> >
> > For the use cases mentioned for why the module attribute is being
> > suggested it would help to not have to download kernel sources.
>
> Are we still talking about compiling custom kernels ?
> (how to do that w/o source code ?)
There is a difference between getting kernel sources for your current
running kernel Vs getting kernel sources from say Linus' tree or the
stable tree, or whatever subsystem you work on. I'm saying that with
a module attribute being present you would *not* have to get the kernel
sources for the current kernel you are running.
> > The only question we want to answer is: for the hardware components
> > present on this system, which configs options do I need to enable
> > to support these components?
>
> What else would one need that data, if not compiling a custom kernel
> (which in turn needs the source) ?
One is compiling a custom kernel.
> > At least for virtualization we decided to support at least these two to
> > help:
> >
> > * make kvmconfig
> > * make xenconfig
>
> These two are rather simple. Most times there isn't much variance in
> virtual hardware (unless one starts directly mapping in pci or usb
> devices ...)
Actually no, there is huge variance on the qemu level and that problem
is not solved by the above.
> > Similar problem would be found if one wanted to find a desirable kernel
> > config for a remote system. One should be able to somehow scrape some
> > hardware information, dump that to a file, and then somehow generate
> > a working config for that system.
>
> Yes. That's actually pretty much the same usecase (in my case I'd also
> have dts, lspci/lsusb output, etc)
>
> > The module attribute being suggested would enable at least one way
> > to gather some of the required config symbols: symbols for *hardware*
> > and where one can run a modern kernel, with many features / hardware
> > enabled already.
>
> But only for a pretty specific usecase.
Two: build time, and backports.
> I'm not opposed to this, but
> I wonder whether maintainers are willing to accept that stuff for just
> that specific usecase.
This is why we are discussing this on this thread.
Since we have 'make localmodconfig' already upstream, and since this
would help both users of that and backports I'd argue it makes sense
upstream. Otherwise I find it that having upstream 'make localmodconfig'
but not wanting to improve that problem space rather odd.
> > However, folks producing embedded systems *do* / *should* have a lot of
> > knowledge of their systems, and so the type of scheme you have devised
> > seems sensible for it.
>
> Usually we have (unless we need to do reverse engineering :o). But it's
> a pretty time-consuming task. Especially if the requirements change
> several times in the development or lifetime of a specific product.
>
> For example "oh, we now need eth", "naah, we don't wanna use usb
> anymore", "let's take a different SoM", ... not that have pretty
> orthogonal sets of configs we need to maintain: hardware- and non-
> hardware-related ones. And hardware-related ones can fall into different
> categories like fixed-attached/onboard vs. hotpluggable ones.
>
> Recently I had a case where the customer requested xattr support, so
> I had to enable general xattr support as well as per-filesystem.
> Pretty simple, but having lots of those cases quickly sums up. One of
> the reasons why I've written my own little config generator.
I agree that the problem space you are dealing with should be made
easier.
> >> In embedded world, we often have scenarios where we want a really
> >> minimal kernel, but need to enable/disable certain hi-level peripherals
> >> in the middle of the project (eg. "oh, we also need ethernet, but we
> >> wanna drop usb"). There we'll have to find out what actual chip is,
> >> its corresponding driver, required subsystems, etc, and also kick off
> >> everything we don't need anymore.
> >
> > Right. One *should* be able to tell some tool, hey, here is the list of
> > my desirable .config options. Go and figure out what I need to make that
> > work and give me a resulting .config. Its not easy.
>
> I think I've already got into a pretty usable state - at least for my
> projects. For now only supports a few boards and limited set of
> features, but patches are always welcomed :)
A solution upstream would be better ;)
> >> I've thought about implementing some actual dependency tracking
> >> (at least recording the auto-enabled symbols), but didn't expect that
> >> to become practically usable anytime soon,
> >
> > The ability to easily ask the kernel to enable the components needed
> > for a respective config option *is* very useful but indeed not easy.
>
> Yes, it would need to understand things like conditional definitions
> to deduce that certain things need to be enabled first, before certain
> drivers become choosable.
>
> > This is not the only space where this problem exists. Similar problem
> > exists for distribution packages, and dependencies. Challenges have
> > been made for proper research towards these problems, and such research
> > has lead distributions to opt to enable some of these algorithms.
>
> The problem w/ dependencies is that there can be different types of
> dependencies, as well as different types of software objects. Just
> solving the expressions is only a part of the problem.
>
> > This begs the question if we could learn from similar efforts on Linux
> > for these sorts of questions. One possibility here is to evaluate the
> > prospect of using a SAT solver with Minimally Unsatisfiable Subformulas
> > (MUSes) support, which should be be able to address thir problem. This
> > prospect is ongoing and currrent R&D is active, for details refer to:
> >
> > https://kernelnewbies.org/KernelProjects/kconfig-sat
>
> Good tip, I'll have a look at it.
>
> > It certainly can be useful for components, ie, not hardware. But for
> > hardware a one-to-one mapping of one driver to one config would be of
> > much more use.
>
> Unfortunately, we don't have this 1:1 mapping. Often drivers support
> different sets of devices, depending on other factors, sometimes sub-
> options (eg. different hw versions), sometimes depending on other
> subsystems, sometimes arch-specific, etc, etc.
>
> I think we should work towards that, but I doubt we'd reach that goal
> anytime soon, and begs the question whether it's really worth all the
> effort required for that.
Code speaks, and fortunately its being worked on.
> > It would be wonderful if for instance kconfig
> > supported a way to group a major set of components under *one* config
> > symbol and then say: "I want this major component enabled" and then it'd
> > go and enable all the defaults which would be required for it.
>
> Yes, thought about that, too. For example have syms for selecting whole
> boards and features of them - a bit like this:
>
> --> Preconfigure for specific boards
> --> board A
> --> board B
> ...
> --> Enable board features
> --> Ethernet port
> --> Display
> --> Touch panel
> --> Audio
> ....
>
> BUT: this would turn into maintenance hell, so I dropped that idea.
>
> > An example is if you
> > wanted to enable PCI on a system which didn't support it. Because of
> > this, it seems you'd want *all* desirable configs and let a piece of
> > software figure out what you need / can enable. And.. this is precisely
> > where the SAT solver with MUSes could help...
>
> Yes, but this piece of software first needs to know whether eg. PCI
> is available on that HW. Oh, and things like PCI could be a dependency
> as well as an feature on its own, depending on how you gonna use it.
> (eg. if directly access from userland or VMs).
Yes, but that is a separate problem. The way 'make localmodconfig'
addresses this is by keeping as modules things which are already
modules. And keeping things it found as =y as buit-in in your kernel as
well.
Luis
On Wed, Jul 3, 2019 at 3:25 PM Luis Chamberlain <[email protected]> wrote:
>
> On Wed, Jul 03, 2019 at 08:57:58PM +0200, Greg Kroah-Hartman wrote:
> > On Wed, Jul 03, 2019 at 04:50:20PM +0000, Luis Chamberlain wrote:
> > > On Wed, Jul 03, 2019 at 09:40:48AM +0200, Greg Kroah-Hartman wrote:
> > > > On Tue, Jul 02, 2019 at 08:51:06PM +0000, Luis Chamberlain wrote:
> > > > > On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> > > > > > On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > > > > > > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <[email protected]> wrote:
> > > > > > > >
> > > > > > > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > > > > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <[email protected]> wrote:
> > > > > > > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > > > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > > > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > > > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > > > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > > > > > > and I'm not quite sure who's volunteering to do it.
> > > > > > > > > >
> > > > > > > > > > Other stakeholders may benefit from at least having some config -->
> > > > > > > > > > module mapping for now. Not just backports or building slimmer
> > > > > > > > > > kernels.
> > > > > > > > >
> > > > > > > > > Christoph, *poke*
> > > > > > > >
> > > > > > > > Yes, I'm still totally opposed to a half-backed hack like this.
> > > > > > >
> > > > > > > The solution puts forward a mechanism to add a kconfig_symb where we
> > > > > > > are 100% certain we have a direct module --> config mapping.
> > > > > > >
> > > > > > > This is *currently* determined when the streamline_config.pl finds
> > > > > > > that an object has only *one* associated config symbol associated. As
> > > > > > > Cristina noted, of 62 modules on a running system 58 of them ended up
> > > > > > > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > > > > > > system being tested. For the other modules, if they did want this
> > > > > > > association, we could allow a way for modules to define their own
> > > > > > > KBUILD_KCONF variable so that this could be considered as well, or
> > > > > > > they can look at their own kconfig stuff to try to fit the model that
> > > > > > > does work. To be clear, the heuristics *can* be updated if there is
> > > > > > > confidence in alternative methods for resolution. But since it is
> > > > > > > reflective of our current situation, I cannot consider it a hack.
> > > > > > >
> > > > > > > This implementation is a reflection of our reality in the kernel, and
> > > > > > > as has been discussed in this thread, if we want to correct the gaps
> > > > > > > we need to do a lot of work. And *no one* is working towards these
> > > > > > > goals.
> > > > > > >
> > > > > > > That said, even if you go forward with an intrusive solution like the
> > > > > > > one you proposed we could still use the same kconfig_symb...
> > > > > > >
> > > > > > > So no, I don't see this as a hack. It's a reflection as to our current
> > > > > > > reality. And I cannot see how the kconfig_symb can lie or be
> > > > > > > incorrect. So in fact I think that pushing this forward also makes the
> > > > > > > problem statement clearer for the future of what semantics needs to be
> > > > > > > addressed, and helps us even annotate the problematic areas of the
> > > > > > > kernel.
> > > > > > >
> > > > > > > What negative aspects do you see with this being merged in practice?
> > > > > >
> > > > > > I'm trying to see what the actual problem that you are wanting to solve
> > > > > > here with this. What exactly is it?
> > > > >
> > > > > The problem is that there is no current maping of a module to respective
> > > > > kconfig symbol.
> > > >
> > > > That's because it is not just "one" symbol per module.
> > >
> > > This is true. But it is not the case for all modules. In fact it seems
> > > its true that most modules do have *one* main symbol.
> >
> > You mean "one unique symbol from all other modules", right?
>
> I mean that in the typical case you have *one* main *parent* symbol defining
> one device driver for one piece of hardware.
>
> > That is much different than just "one" symbol, given that almost
> > every driver depends on something else being enabled as well (bus type,
> > platform type, arch, etc.)
>
> Yes but that is different than what is trying to be addressed.
>
> > And I would argue, that finding that "one" symbol is easy, just parse
> > the Makefiles. But I would also state that this "one" symbol doesn't
> > really help you much as those are the "simple" things. It's how to
> > turn on all of the required symbols to get to that "one" symbol that is
> > the hard part.
>
> Absolutely, but *that* is a different problem! But I am glad you brought
> that up and acknowledge it. Addressing that problem will take time and
> folks are working towards it. Addressing that problem will still however
> not address the problem being addressed here.
>
> > And conversely, if you disable that "one" symbol, does that also mean
> > you can disable the symbols it depended on? If so, how far back?
>
> Right..
>
> > And what about functionality? If my usb-storage device is "enabled" in
> > the build, yet all filesystems are not, or the needed dm module is not,
> > it is useless.
>
> Yes. The localmodconfig approach is to keep enabled as modules *all* currently
> enabled modules, that covers that.
>
> But again, this is a separate problem. The one I am addressing, on
> behalf of Cristina, is a subspace, dedicated towards *hardware*
> functionality.
Hmm...maybe this isn't what I am looking for then. I am interested in
the problem of figuring out what dependencies I need to select to turn
on a desired config symbol (which is obviously a separate issue), and
I am interested in associating symbols with a config symbol and then
ensuring that symbol is exercised. Basically, I want a way to make
sure my tests actually get run without a human looking at them; I feel
like what you are working on might help with this latter issue, but I
am not 100% sure. It sounds like it is not your primary goal in any
case.
> > Hardware requires usually more than one real "symbol" in
> > order to work properly, as you know.
>
> Right. This does not mean that this information of parent main symbol is
> not useful. And having it available on the modules can help with
> multiple goals, without requiring kernel sources.
>
> > And of course, what does this really matter to anyone? If you build
> > "all modules" and you only load the modules you actually use for your
> > hardware (based on auto-loading), then your system uses the same amount
> > of memory as if you disabled all of the modules you did not need.
>
> For backports it means you can enable only what you need, or at least
> show users what modules in a backports release could be upgraded and let
> you enable / disable them. For other users, people can care about build
> time. And not everyone has fancy systems to build all modules; and
> sometimes you may just want to enable a small qemu system and build
> locally on it, enabling all modules on such systems would just be
> extremely silly.
>
> > Yes, it's faster to build, but is that what you are trying to optimize
> > for here?
>
> Particularly for me, yes. Others have other needs and I have already
> stated clearly what the gains are.
>
> > Anyway, if this is just an acidemic thing, have fun, but I would not be
> > adding anything else to the module image that is not really going to be
> > useful to anyone.
>
> The academic consensus is kconfig semantics are poo and we need to do
> slowly start addressing this. I believe that striving towards having
> one kconfig symbol per hw component can help long term, and in the
> meantime, this also will help with the 'make localmodconfig' and
> backport users.
>
> > good luck!
>
> This is not about luck. If you don't want to address these problems,
> that's fine, but please provide objective considerations rather than
> right out rejection without a reasonable basis. Or even better, provide
> alternative recommendations.
>
> The alternative Christoph recommended is hugely instrusive, no one is
> working on it, and the simple solution proposed in this thread at least
> gets us a small step forward in helping to enable a few more users,
> while also postulating if we should strive towards having one main kconfig
> for each hw component. Since it seems this is already the case, and
> there are only a few outliers, this effort should help spot outliers
> and address them to help with our semantics.
Ooo, looks like I should take a look at that.
Cheers!
On Thu, Jul 11, 2019 at 04:07:27PM -0700, Brendan Higgins wrote:
> On Wed, Jul 3, 2019 at 3:25 PM Luis Chamberlain <[email protected]> wrote:
> >
> > On Wed, Jul 03, 2019 at 08:57:58PM +0200, Greg Kroah-Hartman wrote:
> >
> > But again, this is a separate problem. The one I am addressing, on
> > behalf of Cristina, is a subspace, dedicated towards *hardware*
> > functionality.
>
> Hmm...maybe this isn't what I am looking for then. I am interested in
> the problem of figuring out what dependencies I need to select to turn
> on a desired config symbol (which is obviously a separate issue),
It isn't directly. the problem statement is different, and
for that I do recommend checking out the kconfig-sat effort.
It only indirectly relates in that we already know our kconfig
semantics need work, and I do think that the underlying problem
in this thread slowly strives towards addressing kconfig semantics
for modules, associating one main kconfig symbol with one module.
I have no evidence for this though. But since there are users who can
gain from it, I don't see any issues from embracing it.
> and
> I am interested in associating symbols with a config symbol and then
> ensuring that symbol is exercised. Basically, I want a way to make
> sure my tests actually get run without a human looking at them; I feel
> like what you are working on might help with this latter issue, but I
> am not 100% sure. It sounds like it is not your primary goal in any
> case.
Nope, this topic only ralates to yours in the semantics case.
Luis
On Wed, Jul 3, 2019 at 3:42 PM Luis Chamberlain <[email protected]> wrote:
>
> On Wed, Jul 03, 2019 at 09:31:33PM +0200, Enrico Weigelt, metux IT consult wrote:
> > On 03.07.19 19:35, Luis Chamberlain wrote:
> >
> > Hi,
> >
> > >> Okay, but IIRC this will add more boilerplate those modules.
> > >
> > > Just one module attribute.
> >
> > Yes, but still one per module. This raises the question whether
> > maintainers are willing to cope w/ tons of tiny patches for just
> > one line - for something that will take quite some time to become
> > actually useful (doesn't help much if only few drivers support it),
> > and is only helpful in a few use cases.
>
> This can be wrapped in kconfig and disabled by default.
>
> > And to make it really useful, we also need some way to automatically
> > derive which other symbols to enable (subsystems, etc), w/o auto-
> > enabling stuff one doesn't need here. (are the defaults sane for
> > this usecase ?)
>
> No, that's a separate problem.
I am interested in this use case, but I agree that this is a separate issue.
Luis, I saw you cc'ed me on an email with the SAT solver people and
that they are still working on this. Enrico, want me to CC you on that
and we can continue this discussion there?
> > The main problem here, IMHO, is that the kconfig system doesn't really
> > know what makes up a module (it only knows that something w =y cant
> > depend on something thats =m).
> >
> > So it smalls like we'd need some config language that really understands
> > things like modules, subsystems, arches, etc with their properties and
> > is used by both kconfig and kbuild. Then we could put all metadata there
> > instead of the current macro calls. At that point we could also put
> > things like match tables in here, which would solve the problem of
> > finding the right driver by hardware descriptions.
> >
> > But that's really a *big* topic, it's not easy.
>
> Christoph's sugggestion goes along the lines with addressing some of
> this, yet 2 years havegone by and no one is working on it and its hugely
> intrusive. I'd welcome the patches.
I would like to see those. I would also very much like to see code
dependencies, not just kconfig dependencies, tracked in Kbuild
somehow.
> > >> And I wonder whether target binaries are the right place for those
> > >> things at all - IMHO that's something one wants to derive from the
> > >> source code / .config's.
> > >
> > > For the use cases mentioned for why the module attribute is being
> > > suggested it would help to not have to download kernel sources.
> >
> > Are we still talking about compiling custom kernels ?
> > (how to do that w/o source code ?)
>
> There is a difference between getting kernel sources for your current
> running kernel Vs getting kernel sources from say Linus' tree or the
> stable tree, or whatever subsystem you work on. I'm saying that with
> a module attribute being present you would *not* have to get the kernel
> sources for the current kernel you are running.
>
> > > The only question we want to answer is: for the hardware components
> > > present on this system, which configs options do I need to enable
> > > to support these components?
> >
> > What else would one need that data, if not compiling a custom kernel
> > (which in turn needs the source) ?
>
> One is compiling a custom kernel.
>
> > > At least for virtualization we decided to support at least these two to
> > > help:
> > >
> > > * make kvmconfig
> > > * make xenconfig
> >
> > These two are rather simple. Most times there isn't much variance in
> > virtual hardware (unless one starts directly mapping in pci or usb
> > devices ...)
>
> Actually no, there is huge variance on the qemu level and that problem
> is not solved by the above.
>
> > > Similar problem would be found if one wanted to find a desirable kernel
> > > config for a remote system. One should be able to somehow scrape some
> > > hardware information, dump that to a file, and then somehow generate
> > > a working config for that system.
> >
> > Yes. That's actually pretty much the same usecase (in my case I'd also
> > have dts, lspci/lsusb output, etc)
> >
> > > The module attribute being suggested would enable at least one way
> > > to gather some of the required config symbols: symbols for *hardware*
> > > and where one can run a modern kernel, with many features / hardware
> > > enabled already.
> >
> > But only for a pretty specific usecase.
>
> Two: build time, and backports.
I think testing is another potential use case. Admittedly, I might
want something more general, but the general idea of being able to map
a config to code is super useful for automated testing.
For example, I want to run all tests. I enable all configs,cool. But
how do I know that the desired code under test actually ran?
Another example, okay I ran my tests and some of them failed. All my
test debug information is all mixed together and there is a lot of
noise present, also I am afraid that something else not part of the
test influenced the test or code under test behavior. So I want to
rerun the failed tests and the code that they test and nothing else.
> > I'm not opposed to this, but
> > I wonder whether maintainers are willing to accept that stuff for just
> > that specific usecase.
>
> This is why we are discussing this on this thread.
>
> Since we have 'make localmodconfig' already upstream, and since this
> would help both users of that and backports I'd argue it makes sense
> upstream. Otherwise I find it that having upstream 'make localmodconfig'
> but not wanting to improve that problem space rather odd.
>
> > > However, folks producing embedded systems *do* / *should* have a lot of
> > > knowledge of their systems, and so the type of scheme you have devised
> > > seems sensible for it.
> >
> > Usually we have (unless we need to do reverse engineering :o). But it's
> > a pretty time-consuming task. Especially if the requirements change
> > several times in the development or lifetime of a specific product.
> >
> > For example "oh, we now need eth", "naah, we don't wanna use usb
> > anymore", "let's take a different SoM", ... not that have pretty
> > orthogonal sets of configs we need to maintain: hardware- and non-
> > hardware-related ones. And hardware-related ones can fall into different
> > categories like fixed-attached/onboard vs. hotpluggable ones.
> >
> > Recently I had a case where the customer requested xattr support, so
> > I had to enable general xattr support as well as per-filesystem.
> > Pretty simple, but having lots of those cases quickly sums up. One of
> > the reasons why I've written my own little config generator.
>
> I agree that the problem space you are dealing with should be made
> easier.
>
> > >> In embedded world, we often have scenarios where we want a really
> > >> minimal kernel, but need to enable/disable certain hi-level peripherals
> > >> in the middle of the project (eg. "oh, we also need ethernet, but we
> > >> wanna drop usb"). There we'll have to find out what actual chip is,
> > >> its corresponding driver, required subsystems, etc, and also kick off
> > >> everything we don't need anymore.
> > >
> > > Right. One *should* be able to tell some tool, hey, here is the list of
> > > my desirable .config options. Go and figure out what I need to make that
> > > work and give me a resulting .config. Its not easy.
> >
> > I think I've already got into a pretty usable state - at least for my
> > projects. For now only supports a few boards and limited set of
> > features, but patches are always welcomed :)
>
> A solution upstream would be better ;)
>
> > >> I've thought about implementing some actual dependency tracking
> > >> (at least recording the auto-enabled symbols), but didn't expect that
> > >> to become practically usable anytime soon,
> > >
> > > The ability to easily ask the kernel to enable the components needed
> > > for a respective config option *is* very useful but indeed not easy.
> >
> > Yes, it would need to understand things like conditional definitions
> > to deduce that certain things need to be enabled first, before certain
> > drivers become choosable.
> >
> > > This is not the only space where this problem exists. Similar problem
> > > exists for distribution packages, and dependencies. Challenges have
> > > been made for proper research towards these problems, and such research
> > > has lead distributions to opt to enable some of these algorithms.
> >
> > The problem w/ dependencies is that there can be different types of
> > dependencies, as well as different types of software objects. Just
> > solving the expressions is only a part of the problem.
> >
> > > This begs the question if we could learn from similar efforts on Linux
> > > for these sorts of questions. One possibility here is to evaluate the
> > > prospect of using a SAT solver with Minimally Unsatisfiable Subformulas
> > > (MUSes) support, which should be be able to address thir problem. This
> > > prospect is ongoing and currrent R&D is active, for details refer to:
> > >
> > > https://kernelnewbies.org/KernelProjects/kconfig-sat
> >
> > Good tip, I'll have a look at it.
> >
> > > It certainly can be useful for components, ie, not hardware. But for
> > > hardware a one-to-one mapping of one driver to one config would be of
> > > much more use.
> >
> > Unfortunately, we don't have this 1:1 mapping. Often drivers support
> > different sets of devices, depending on other factors, sometimes sub-
> > options (eg. different hw versions), sometimes depending on other
> > subsystems, sometimes arch-specific, etc, etc.
> >
> > I think we should work towards that, but I doubt we'd reach that goal
> > anytime soon, and begs the question whether it's really worth all the
> > effort required for that.
>
> Code speaks, and fortunately its being worked on.
>
> > > It would be wonderful if for instance kconfig
> > > supported a way to group a major set of components under *one* config
> > > symbol and then say: "I want this major component enabled" and then it'd
> > > go and enable all the defaults which would be required for it.
> >
> > Yes, thought about that, too. For example have syms for selecting whole
> > boards and features of them - a bit like this:
> >
> > --> Preconfigure for specific boards
> > --> board A
> > --> board B
> > ...
> > --> Enable board features
> > --> Ethernet port
> > --> Display
> > --> Touch panel
> > --> Audio
> > ....
> >
> > BUT: this would turn into maintenance hell, so I dropped that idea.
I wonder if that would work for the testing scenario? I don't think it
is unreasonable for a test owner to provide a defconfig that makes it
possible to run their test. We could then merge these together to
create a kconfig to run all desired tests. Doesn't address all the
issues I mentioned here, but it's a start.
> > > An example is if you
> > > wanted to enable PCI on a system which didn't support it. Because of
> > > this, it seems you'd want *all* desirable configs and let a piece of
> > > software figure out what you need / can enable. And.. this is precisely
> > > where the SAT solver with MUSes could help...
> >
> > Yes, but this piece of software first needs to know whether eg. PCI
> > is available on that HW. Oh, and things like PCI could be a dependency
> > as well as an feature on its own, depending on how you gonna use it.
> > (eg. if directly access from userland or VMs).
>
> Yes, but that is a separate problem. The way 'make localmodconfig'
> addresses this is by keeping as modules things which are already
> modules. And keeping things it found as =y as buit-in in your kernel as
> well.
Cheers!
On 12.07.19 01:27, Brendan Higgins wrote:
> Enrico, want me to CC you on that> and we can continue this discussion there?
Yes. But I'd prefer having an own list for it - better for sorting and
archiving ;-)
> I wonder if that would work for the testing scenario? I don't think it> is unreasonable for a test owner to provide a defconfig that makes it>
possible to run their test. We could then merge these together to>
create a kconfig to run all desired tests. Doesn't address all the>
issues I mentioned here, but it's a start.
defconfig is a different thing - my idea (that I've dropped) was
actually introducing new config options per board (and sub options
for board features) which switch on all the neccessary things.
The defconfigs are nice thing for starting off with some board, but
they're basically examples, not production configurations. Yet another
point why I've decided to cope all of this in a separate tool.
--mtx
--
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
[email protected] -- +49-151-27565287