2020-04-22 19:01:50

by Rob Herring (Arm)

Subject: [PATCH v2] dt-bindings: Fix command line length limit calling dt-mk-schema

As the number of schemas has increased, we're starting to hit the error
"execvp: /bin/sh: Argument list too long". This is due to passing all the
schema files on the command line to dt-mk-schema. It currently is only
with out of tree builds and is intermittent depending on the file path
lengths.

Commit 2ba06cd8565b ("kbuild: Always validate DT binding examples") made
hitting this proplem more likely since the example validation now always
gets the full list of schemas.

Fix this by passing the schema file list in a pipe and using xargs. We end
up doing the find twice, but the time is insignificant compared to the
dt-mk-schema time.

Reported-by: Laurent Pinchart <[email protected]>
Cc: Masahiro Yamada <[email protected]>
Signed-off-by: Rob Herring <[email protected]>
---
Documentation/devicetree/bindings/Makefile | 18 ++++++++++--------
1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/Documentation/devicetree/bindings/Makefile b/Documentation/devicetree/bindings/Makefile
index 87c76bdabfe6..7782d9985082 100644
--- a/Documentation/devicetree/bindings/Makefile
+++ b/Documentation/devicetree/bindings/Makefile
@@ -14,16 +14,18 @@ $(obj)/%.example.dts: $(src)/%.yaml FORCE
# Use full schemas when checking %.example.dts
DT_TMP_SCHEMA := $(obj)/processed-schema-examples.yaml

+find_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \
+ -name 'processed-schema*' ! \
+ -name '*.example.dt.yaml' \)
+
quiet_cmd_mk_schema = SCHEMA $@
- cmd_mk_schema = $(DT_MK_SCHEMA) $(DT_MK_SCHEMA_FLAGS) -o $@ $(real-prereqs)
+ cmd_mk_schema = rm -f $@ ; \
+ $(if $(DT_MK_SCHEMA_FLAGS), \
+ echo $(real-prereqs), \
+ $(find_cmd)) | \
+ xargs $(DT_MK_SCHEMA) $(DT_MK_SCHEMA_FLAGS) >> $@

-DT_DOCS = $(addprefix $(src)/, \
- $(shell \
- cd $(srctree)/$(src) && \
- find * \( -name '*.yaml' ! \
- -name 'processed-schema*' ! \
- -name '*.example.dt.yaml' \) \
- ))
+DT_DOCS = $(shell $(find_cmd) | sed -e 's|^$(srctree)/||')

DT_SCHEMA_FILES ?= $(DT_DOCS)

--
2.20.1

2020-04-23 16:11:21

by Laurent Pinchart

Subject: Re: [PATCH v2] dt-bindings: Fix command line length limit calling dt-mk-schema

Hi Rob,

Thank you for the patch.

On Wed, Apr 22, 2020 at 01:57:08PM -0500, Rob Herring wrote:
> As the number of schemas has increased, we're starting to hit the error
> "execvp: /bin/sh: Argument list too long". This is due to passing all the
> schema files on the command line to dt-mk-schema. It currently is only
> with out of tree builds and is intermittent depending on the file path
> lengths.
>
> Commit 2ba06cd8565b ("kbuild: Always validate DT binding examples") made
> hitting this proplem more likely since the example validation now always
> gets the full list of schemas.
>
> Fix this by passing the schema file list in a pipe and using xargs. We end
> up doing the find twice, but the time is insignificant compared to the
> dt-mk-schema time.
>
> Reported-by: Laurent Pinchart <[email protected]>
> Cc: Masahiro Yamada <[email protected]>
> Signed-off-by: Rob Herring <[email protected]>

Tested-by: Laurent Pinchart <[email protected]>

> ---
> Documentation/devicetree/bindings/Makefile | 18 ++++++++++--------
> 1 file changed, 10 insertions(+), 8 deletions(-)
>
> diff --git a/Documentation/devicetree/bindings/Makefile b/Documentation/devicetree/bindings/Makefile
> index 87c76bdabfe6..7782d9985082 100644
> --- a/Documentation/devicetree/bindings/Makefile
> +++ b/Documentation/devicetree/bindings/Makefile
> @@ -14,16 +14,18 @@ $(obj)/%.example.dts: $(src)/%.yaml FORCE
> # Use full schemas when checking %.example.dts
> DT_TMP_SCHEMA := $(obj)/processed-schema-examples.yaml
>
> +find_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \
> + -name 'processed-schema*' ! \
> + -name '*.example.dt.yaml' \)
> +
> quiet_cmd_mk_schema = SCHEMA $@
> - cmd_mk_schema = $(DT_MK_SCHEMA) $(DT_MK_SCHEMA_FLAGS) -o $@ $(real-prereqs)
> + cmd_mk_schema = rm -f $@ ; \
> + $(if $(DT_MK_SCHEMA_FLAGS), \
> + echo $(real-prereqs), \
> + $(find_cmd)) | \
> + xargs $(DT_MK_SCHEMA) $(DT_MK_SCHEMA_FLAGS) >> $@
>
> -DT_DOCS = $(addprefix $(src)/, \
> - $(shell \
> - cd $(srctree)/$(src) && \
> - find * \( -name '*.yaml' ! \
> - -name 'processed-schema*' ! \
> - -name '*.example.dt.yaml' \) \
> - ))
> +DT_DOCS = $(shell $(find_cmd) | sed -e 's|^$(srctree)/||')
>
> DT_SCHEMA_FILES ?= $(DT_DOCS)
>

--
Regards,

Laurent Pinchart

2020-04-23 19:24:33

by Masahiro Yamada

Subject: Re: [PATCH v2] dt-bindings: Fix command line length limit calling dt-mk-schema

On Thu, Apr 23, 2020 at 3:57 AM Rob Herring <[email protected]> wrote:
>
> As the number of schemas has increased, we're starting to hit the error
> "execvp: /bin/sh: Argument list too long". This is due to passing all the
> schema files on the command line to dt-mk-schema. It currently is only
> with out of tree builds and is intermittent depending on the file path
> lengths.
>
> Commit 2ba06cd8565b ("kbuild: Always validate DT binding examples") made
> hitting this proplem more likely since the example validation now always
> gets the full list of schemas.
>
> Fix this by passing the schema file list in a pipe and using xargs. We end
> up doing the find twice, but the time is insignificant compared to the
> dt-mk-schema time.
>
> Reported-by: Laurent Pinchart <[email protected]>
> Cc: Masahiro Yamada <[email protected]>
> Signed-off-by: Rob Herring <[email protected]>


Reviewed-by: Masahiro Yamada <[email protected]>


> ---
> Documentation/devicetree/bindings/Makefile | 18 ++++++++++--------
> 1 file changed, 10 insertions(+), 8 deletions(-)
>
> diff --git a/Documentation/devicetree/bindings/Makefile b/Documentation/devicetree/bindings/Makefile
> index 87c76bdabfe6..7782d9985082 100644
> --- a/Documentation/devicetree/bindings/Makefile
> +++ b/Documentation/devicetree/bindings/Makefile
> @@ -14,16 +14,18 @@ $(obj)/%.example.dts: $(src)/%.yaml FORCE
> # Use full schemas when checking %.example.dts
> DT_TMP_SCHEMA := $(obj)/processed-schema-examples.yaml
>
> +find_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \
> + -name 'processed-schema*' ! \
> + -name '*.example.dt.yaml' \)
> +
> quiet_cmd_mk_schema = SCHEMA $@
> - cmd_mk_schema = $(DT_MK_SCHEMA) $(DT_MK_SCHEMA_FLAGS) -o $@ $(real-prereqs)
> + cmd_mk_schema = rm -f $@ ; \
> + $(if $(DT_MK_SCHEMA_FLAGS), \
> + echo $(real-prereqs), \
> + $(find_cmd)) | \
> + xargs $(DT_MK_SCHEMA) $(DT_MK_SCHEMA_FLAGS) >> $@
>
> -DT_DOCS = $(addprefix $(src)/, \
> - $(shell \
> - cd $(srctree)/$(src) && \
> - find * \( -name '*.yaml' ! \
> - -name 'processed-schema*' ! \
> - -name '*.example.dt.yaml' \) \
> - ))
> +DT_DOCS = $(shell $(find_cmd) | sed -e 's|^$(srctree)/||')
>
> DT_SCHEMA_FILES ?= $(DT_DOCS)
>
> --
> 2.20.1
>


--
Best Regards
Masahiro Yamada