Hello,
there is some testsuite included with ndctl, and when following the
instructions to build it most tests fail or are skipped:
[ 95s] Ok: 3
[ 95s] Expected Fail: 0
[ 95s] Fail: 5
[ 95s] Unexpected Pass: 0
[ 95s] Skipped: 15
[ 95s] Timeout: 0
Is this the expected outcome or is this a problem with the ndctl build?
Attaching test run log.
Thanks
Michal
On Tue, Apr 26, 2022 at 9:15 AM Michal Suchánek <[email protected]> wrote:
>
> On Tue, Apr 26, 2022 at 08:51:25AM -0700, Dan Williams wrote:
> > On Tue, Apr 26, 2022 at 5:39 AM Michal Suchánek <[email protected]> wrote:
> > >
> > > Hello,
> > >
> > > there is some testsuite included with ndctl, and when following the
> > > instructions to build it most tests fail or are skipped:
> > >
> > > [ 95s] Ok: 3
> > > [ 95s] Expected Fail: 0
> > > [ 95s] Fail: 5
> > > [ 95s] Unexpected Pass: 0
> > > [ 95s] Skipped: 15
> > > [ 95s] Timeout: 0
> > >
> > > Is this the expected outcome or is this a problem with the ndctl build?
> > >
> > > Attaching test run log.
> >
> > I see a few missing prerequisites:
> >
> > [ 78s] /usr/src/packages/BUILD/ndctl-73/test/pmem-errors.sh: line
> > 64: mkfs.ext4: command not found
> > [ 95s] /usr/src/packages/BUILD/ndctl-73/test/security.sh: line 25:
> > jq: command not found
>
> Indeed, with those installed I get much more tests passing:
>
> [ 148s] Ok: 13
> [ 148s] Expected Fail: 0
> [ 148s] Fail: 4
> [ 148s] Unexpected Pass: 0
> [ 148s] Skipped: 6
> [ 148s] Timeout: 0
>
> >
> > This report:
> >
> > [ 51s] 1/23 ndctl:ndctl / libndctl SKIP
> > 0.02s exit status 77
> >
> > ...seems to indicate that the nfit_test modules did not appear to load
> > correctly. I never expected that the nfit_test modules would be
> > redistributable, so I was surprised to see them being installed by an
> > actual package "nfit_test-kmp-default-0_k5.17.4_1-6.1". The reason
> > they are not redistributable is because they require replacing the
> > production build of the kernel provided modules libnvdimm.ko,
> > nd_pmem.ko, etc... What I expect is happening is that the production
> > version of libnvdimm.ko is already loaded (or is the only one on the
>
> AFAICT neither is the case, that's why I dump the module information in
> the log.
The modinfo just tells you what modules are available, but it does not
necessarily indicate which modules are actively loaded in the system
which is what ndctl_test_init() validates.
On Tue, Apr 26, 2022 at 11:10 AM Michal Suchánek <[email protected]> wrote:
>
> On Tue, Apr 26, 2022 at 09:47:19AM -0700, Dan Williams wrote:
> > On Tue, Apr 26, 2022 at 9:43 AM Michal Suchánek <[email protected]> wrote:
> > >
> > > On Tue, Apr 26, 2022 at 09:32:24AM -0700, Dan Williams wrote:
> > > > On Tue, Apr 26, 2022 at 9:15 AM Michal Suchánek <[email protected]> wrote:
> > > > >
> > > > > On Tue, Apr 26, 2022 at 08:51:25AM -0700, Dan Williams wrote:
> > > > > > On Tue, Apr 26, 2022 at 5:39 AM Michal Suchánek <[email protected]> wrote:
> > > > > > >
> ...
> > > >
> > > > The modinfo just tells you what modules are available, but it does not
> > > > necessarily indicate which modules are actively loaded in the system
> > > > which is what ndctl_test_init() validates.
> > >
> > > Isn't what modinfo lists also what modrobe loads?
> >
> > It shows what modprobe would load on the next invocation, but
> > sometimes when nfit_test fails it's because the initramfs or something
> > else loaded the modules without respecting the extra/ (or updates/ in
> > your case) override modules.
> >
> > > There isn't any pmem so I don't see why production modules would be
> > > loaded before the test modules are installed. Unloading the modules
> > > first does not really make any difference.
> >
> > Ok, my first guess was wrong... would need more debug to see what
> > those other skip tests are complaining about.
>
> There was also missing parted and hostname command.
>
> However, the nfit.ko is detected as production even when I remove all
> the production modules just in case. lsmod confirms that the nvdimm
> modules are not loaded before the test.
>
> Maybe something goes wrong with the test module build?
>
> It is very fragile and requires complete kernel source for each
> configuration built. See below for the package
>
> https://build.opensuse.org/package/show/home:michals/nfit_test
>
> Attaching the log of test run which does not report any missing tools,
> only complains about nfit.ko being production.
Oh... something silly, ndctl_test_init() assumes that the out-of-tree
module directory is always "/lib/modules/$(uname -r)/extra"
if (!strstr(path, "/extra/")) {
log_err(&log_ctx, "%s.ko: appears to be
production version: %s\n",
name, path);
break;
}
Looks like a build configuration variable is needed there to allow for
"updates/".
On Tue, Apr 26, 2022 at 09:47:19AM -0700, Dan Williams wrote:
> On Tue, Apr 26, 2022 at 9:43 AM Michal Such?nek <[email protected]> wrote:
> >
> > On Tue, Apr 26, 2022 at 09:32:24AM -0700, Dan Williams wrote:
> > > On Tue, Apr 26, 2022 at 9:15 AM Michal Such?nek <[email protected]> wrote:
> > > >
> > > > On Tue, Apr 26, 2022 at 08:51:25AM -0700, Dan Williams wrote:
> > > > > On Tue, Apr 26, 2022 at 5:39 AM Michal Such?nek <[email protected]> wrote:
> > > > > >
...
> > >
> > > The modinfo just tells you what modules are available, but it does not
> > > necessarily indicate which modules are actively loaded in the system
> > > which is what ndctl_test_init() validates.
> >
> > Isn't what modinfo lists also what modrobe loads?
>
> It shows what modprobe would load on the next invocation, but
> sometimes when nfit_test fails it's because the initramfs or something
> else loaded the modules without respecting the extra/ (or updates/ in
> your case) override modules.
>
> > There isn't any pmem so I don't see why production modules would be
> > loaded before the test modules are installed. Unloading the modules
> > first does not really make any difference.
>
> Ok, my first guess was wrong... would need more debug to see what
> those other skip tests are complaining about.
There was also missing parted and hostname command.
However, the nfit.ko is detected as production even when I remove all
the production modules just in case. lsmod confirms that the nvdimm
modules are not loaded before the test.
Maybe something goes wrong with the test module build?
It is very fragile and requires complete kernel source for each
configuration built. See below for the package
https://build.opensuse.org/package/show/home:michals/nfit_test
Attaching the log of test run which does not report any missing tools,
only complains about nfit.ko being production.
Thanks
Michal
On Tue, Apr 26, 2022 at 5:39 AM Michal Suchánek <[email protected]> wrote:
>
> Hello,
>
> there is some testsuite included with ndctl, and when following the
> instructions to build it most tests fail or are skipped:
>
> [ 95s] Ok: 3
> [ 95s] Expected Fail: 0
> [ 95s] Fail: 5
> [ 95s] Unexpected Pass: 0
> [ 95s] Skipped: 15
> [ 95s] Timeout: 0
>
> Is this the expected outcome or is this a problem with the ndctl build?
>
> Attaching test run log.
I see a few missing prerequisites:
[ 78s] /usr/src/packages/BUILD/ndctl-73/test/pmem-errors.sh: line
64: mkfs.ext4: command not found
[ 95s] /usr/src/packages/BUILD/ndctl-73/test/security.sh: line 25:
jq: command not found
This report:
[ 51s] 1/23 ndctl:ndctl / libndctl SKIP
0.02s exit status 77
...seems to indicate that the nfit_test modules did not appear to load
correctly. I never expected that the nfit_test modules would be
redistributable, so I was surprised to see them being installed by an
actual package "nfit_test-kmp-default-0_k5.17.4_1-6.1". The reason
they are not redistributable is because they require replacing the
production build of the kernel provided modules libnvdimm.ko,
nd_pmem.ko, etc... What I expect is happening is that the production
version of libnvdimm.ko is already loaded (or is the only one on the
system), and ndctl_test_init()
(https://github.com/pmem/ndctl/blob/main/test/core.c#L110) detects
that case and skips the tests.
This is what I see with my setup that uses the sequence below to
install nfit_test and friends:
Test summary:
Ok: 35
Expected Fail: 0
Fail: 1
Unexpected Pass: 0
Skipped: 1
Timeout: 0
Install procedure from a kernel build directory:
export INSTALL_MOD_PATH=$root
make INSTALL_HDR_PATH=$root/usr headers_install
make M=tools/testing/nvdimm modules_install
make M=tools/testing/cxl modules_install
make modules_install
On Tue, Apr 26, 2022 at 08:51:25AM -0700, Dan Williams wrote:
> On Tue, Apr 26, 2022 at 5:39 AM Michal Such?nek <[email protected]> wrote:
> >
> > Hello,
> >
> > there is some testsuite included with ndctl, and when following the
> > instructions to build it most tests fail or are skipped:
> >
> > [ 95s] Ok: 3
> > [ 95s] Expected Fail: 0
> > [ 95s] Fail: 5
> > [ 95s] Unexpected Pass: 0
> > [ 95s] Skipped: 15
> > [ 95s] Timeout: 0
> >
> > Is this the expected outcome or is this a problem with the ndctl build?
> >
> > Attaching test run log.
>
> I see a few missing prerequisites:
>
> [ 78s] /usr/src/packages/BUILD/ndctl-73/test/pmem-errors.sh: line
> 64: mkfs.ext4: command not found
> [ 95s] /usr/src/packages/BUILD/ndctl-73/test/security.sh: line 25:
> jq: command not found
Indeed, with those installed I get much more tests passing:
[ 148s] Ok: 13
[ 148s] Expected Fail: 0
[ 148s] Fail: 4
[ 148s] Unexpected Pass: 0
[ 148s] Skipped: 6
[ 148s] Timeout: 0
>
> This report:
>
> [ 51s] 1/23 ndctl:ndctl / libndctl SKIP
> 0.02s exit status 77
>
> ...seems to indicate that the nfit_test modules did not appear to load
> correctly. I never expected that the nfit_test modules would be
> redistributable, so I was surprised to see them being installed by an
> actual package "nfit_test-kmp-default-0_k5.17.4_1-6.1". The reason
> they are not redistributable is because they require replacing the
> production build of the kernel provided modules libnvdimm.ko,
> nd_pmem.ko, etc... What I expect is happening is that the production
> version of libnvdimm.ko is already loaded (or is the only one on the
AFAICT neither is the case, that's why I dump the module information in
the log.
Thanks
Michal
On Tue, Apr 26, 2022 at 11:22:55AM -0700, Dan Williams wrote:
> On Tue, Apr 26, 2022 at 11:10 AM Michal Such?nek <[email protected]> wrote:
> >
> > On Tue, Apr 26, 2022 at 09:47:19AM -0700, Dan Williams wrote:
> > > On Tue, Apr 26, 2022 at 9:43 AM Michal Such?nek <[email protected]> wrote:
> > > >
> > > > On Tue, Apr 26, 2022 at 09:32:24AM -0700, Dan Williams wrote:
> > > > > On Tue, Apr 26, 2022 at 9:15 AM Michal Such?nek <[email protected]> wrote:
> > > > > >
> > > > > > On Tue, Apr 26, 2022 at 08:51:25AM -0700, Dan Williams wrote:
> > > > > > > On Tue, Apr 26, 2022 at 5:39 AM Michal Such?nek <[email protected]> wrote:
> > > > > > > >
> > ...
> > > > >
> > > > > The modinfo just tells you what modules are available, but it does not
> > > > > necessarily indicate which modules are actively loaded in the system
> > > > > which is what ndctl_test_init() validates.
> > > >
> > > > Isn't what modinfo lists also what modrobe loads?
> > >
> > > It shows what modprobe would load on the next invocation, but
> > > sometimes when nfit_test fails it's because the initramfs or something
> > > else loaded the modules without respecting the extra/ (or updates/ in
> > > your case) override modules.
> > >
> > > > There isn't any pmem so I don't see why production modules would be
> > > > loaded before the test modules are installed. Unloading the modules
> > > > first does not really make any difference.
> > >
> > > Ok, my first guess was wrong... would need more debug to see what
> > > those other skip tests are complaining about.
> >
> > There was also missing parted and hostname command.
> >
> > However, the nfit.ko is detected as production even when I remove all
> > the production modules just in case. lsmod confirms that the nvdimm
> > modules are not loaded before the test.
> >
> > Maybe something goes wrong with the test module build?
> >
> > It is very fragile and requires complete kernel source for each
> > configuration built. See below for the package
> >
> > https://build.opensuse.org/package/show/home:michals/nfit_test
> >
> > Attaching the log of test run which does not report any missing tools,
> > only complains about nfit.ko being production.
>
> Oh... something silly, ndctl_test_init() assumes that the out-of-tree
> module directory is always "/lib/modules/$(uname -r)/extra"
>
> if (!strstr(path, "/extra/")) {
> log_err(&log_ctx, "%s.ko: appears to be
> production version: %s\n",
> name, path);
> break;
> }
>
> Looks like a build configuration variable is needed there to allow for
> "updates/".
yes, I could also change the path in the package to match this
expactation. Not sure if that would break something else.
There is one last bit: the tests require at least 1GB of ram, or
specifically the default VM with 512MB is not sufficient.
With that only monitor test fails, and some BTT test is skipped because
the 5.17 kernel supposedly does not have the correct support:
[ 182s] 19/23 ndctl:ndctl / monitor.sh FAIL 6.39s exit status 1
[ 182s]
[ 182s]
[ 182s] Ok: 21
[ 182s] Expected Fail: 0
[ 182s] Fail: 1
[ 182s] Unexpected Pass: 0
[ 182s] Skipped: 1
[ 182s] Timeout: 0
Thanks
Michal
On Thu, Apr 28, 2022 at 10:02:47AM +0530, Shivaprasad G Bhat wrote:
> On 4/26/22 23:39, Michal Such?nek wrote:
> > On Tue, Apr 26, 2022 at 09:47:19AM -0700, Dan Williams wrote:
> > > On Tue, Apr 26, 2022 at 9:43 AM Michal Such?nek <[email protected]> wrote:
> > > >
> > > > On Tue, Apr 26, 2022 at 09:32:24AM -0700, Dan Williams wrote:
> > > > > On Tue, Apr 26, 2022 at 9:15 AM Michal Such?nek <[email protected]> wrote:
> > > > > >
> > > > > > On Tue, Apr 26, 2022 at 08:51:25AM -0700, Dan Williams wrote:
> > > > > > > On Tue, Apr 26, 2022 at 5:39 AM Michal Such?nek <[email protected]> wrote:
> > > > > > > >
> > ...
> > > > >
> > > > > The modinfo just tells you what modules are available, but it does not
> > > > > necessarily indicate which modules are actively loaded in the system
> > > > > which is what ndctl_test_init() validates.
> > > >
> > > > Isn't what modinfo lists also what modrobe loads?
>
> <snip>
>
> > modules are not loaded before the test.
> >
> > Maybe something goes wrong with the test module build?
> >
> > It is very fragile and requires complete kernel source for each
> > configuration built. See below for the package
> >
> > https://build.opensuse.org/package/show/home:michals/nfit_test
> >
> > Attaching the log of test run which does not report any missing tools,
> > only complains about nfit.ko being production.
>
> I have attempted to fix few of the known issues in the first 3 patches of
> the series posted here.
>
> https://patchwork.kernel.org/project/linux-nvdimm/list/?series=633103
Thanks for pointing out these series.
It is certainly of interest but given the test VM is x86 the papr fixes
don't really apply to this case.
Michal
On 4/26/22 23:39, Michal Suchánek wrote:
> On Tue, Apr 26, 2022 at 09:47:19AM -0700, Dan Williams wrote:
>> On Tue, Apr 26, 2022 at 9:43 AM Michal Suchánek <[email protected]> wrote:
>>>
>>> On Tue, Apr 26, 2022 at 09:32:24AM -0700, Dan Williams wrote:
>>>> On Tue, Apr 26, 2022 at 9:15 AM Michal Suchánek <[email protected]> wrote:
>>>>>
>>>>> On Tue, Apr 26, 2022 at 08:51:25AM -0700, Dan Williams wrote:
>>>>>> On Tue, Apr 26, 2022 at 5:39 AM Michal Suchánek <[email protected]> wrote:
>>>>>>>
> ...
>>>>
>>>> The modinfo just tells you what modules are available, but it does not
>>>> necessarily indicate which modules are actively loaded in the system
>>>> which is what ndctl_test_init() validates.
>>>
>>> Isn't what modinfo lists also what modrobe loads?
<snip>
> modules are not loaded before the test.
>
> Maybe something goes wrong with the test module build?
>
> It is very fragile and requires complete kernel source for each
> configuration built. See below for the package
>
> https://build.opensuse.org/package/show/home:michals/nfit_test
>
> Attaching the log of test run which does not report any missing tools,
> only complains about nfit.ko being production.
I have attempted to fix few of the known issues in the first 3 patches
of the series posted here.
https://patchwork.kernel.org/project/linux-nvdimm/list/?series=633103
>
> Thanks
>
> Michal