The processing of vmlinux.o with objtool is the most memory-intensive step
of a kernel build. By reducing the maximum memory usage here we can reduce
the maximum memory usage of the whole kernel build.
Therefore memory pressure on memory starved machines is relieved during
kernel builds and the build is faster as less swapping has to occur.
To: Josh Poimboeuf <[email protected]>
To: Peter Zijlstra <[email protected]>
Cc: [email protected]
Signed-off-by: Thomas Weißschuh <[email protected]>
---
Changes in v2:
- Warn on out of range values for reloc->type
- Also reduce size of struct special_alt
- Note: v1 did not make it to the lists, only to individual recipients
---
Thomas Weißschuh (8):
objtool: make struct entries[] static and const
objtool: make struct check_options static
objtool: allocate multiple structures with calloc()
objtool: introduce function elf_reloc_set_type
objtool: reduce memory usage of struct reloc
objtool: optimize layout of struct symbol
objtool: optimize layout of struct special_alt
objtool: explicitly cleanup resources on success
tools/objtool/builtin-check.c | 4 ++-
tools/objtool/check.c | 6 ++--
tools/objtool/elf.c | 56 +++++++++++++++++++--------------
tools/objtool/include/objtool/builtin.h | 2 --
tools/objtool/include/objtool/elf.h | 13 +++++---
tools/objtool/include/objtool/special.h | 2 +-
tools/objtool/special.c | 6 ++--
7 files changed, 51 insertions(+), 38 deletions(-)
---
base-commit: 1b929c02afd37871d5afb9d498426f83432e71c2
change-id: 20221216-objtool-memory-06db3b8bf111
Best regards,
--
Thomas Weißschuh <[email protected]>
An upcoming patch needs to perform validation when setting reloc->type
so introduce a helper to contain this validation.
Signed-off-by: Thomas Weißschuh <[email protected]>
---
tools/objtool/check.c | 6 +++---
tools/objtool/elf.c | 11 ++++++++---
tools/objtool/include/objtool/elf.h | 2 ++
3 files changed, 13 insertions(+), 6 deletions(-)
diff --git a/tools/objtool/check.c b/tools/objtool/check.c
index 4350be739f4f..971ee6826de7 100644
--- a/tools/objtool/check.c
+++ b/tools/objtool/check.c
@@ -1355,7 +1355,7 @@ static void annotate_call_site(struct objtool_file *file,
*/
if (opts.hack_noinstr && insn->sec->noinstr && sym->profiling_func) {
if (reloc) {
- reloc->type = R_NONE;
+ elf_reloc_set_type(reloc, R_NONE);
elf_write_reloc(file->elf, reloc);
}
@@ -1384,7 +1384,7 @@ static void annotate_call_site(struct objtool_file *file,
WARN_FUNC("Tail call to __fentry__ !?!?", insn->sec, insn->offset);
if (opts.mnop) {
if (reloc) {
- reloc->type = R_NONE;
+ elf_reloc_set_type(reloc, R_NONE);
elf_write_reloc(file->elf, reloc);
}
@@ -1863,7 +1863,7 @@ static int handle_jump_alt(struct objtool_file *file,
struct reloc *reloc = insn_reloc(file, orig_insn);
if (reloc) {
- reloc->type = R_NONE;
+ elf_reloc_set_type(reloc, R_NONE);
elf_write_reloc(file->elf, reloc);
}
elf_write_insn(file->elf, orig_insn->sec,
diff --git a/tools/objtool/elf.c b/tools/objtool/elf.c
index 9c326efb8cd9..ee355beb0d82 100644
--- a/tools/objtool/elf.c
+++ b/tools/objtool/elf.c
@@ -555,7 +555,7 @@ int elf_add_reloc(struct elf *elf, struct section *sec, unsigned long offset,
reloc->sec = sec->reloc;
reloc->offset = offset;
- reloc->type = type;
+ elf_reloc_set_type(reloc, type);
reloc->sym = sym;
reloc->addend = addend;
@@ -872,7 +872,7 @@ static int read_rel_reloc(struct section *sec, int i, struct reloc *reloc, unsig
WARN_ELF("gelf_getrel");
return -1;
}
- reloc->type = GELF_R_TYPE(reloc->rel.r_info);
+ elf_reloc_set_type(reloc, GELF_R_TYPE(reloc->rel.r_info));
reloc->addend = 0;
reloc->offset = reloc->rel.r_offset;
*symndx = GELF_R_SYM(reloc->rel.r_info);
@@ -885,7 +885,7 @@ static int read_rela_reloc(struct section *sec, int i, struct reloc *reloc, unsi
WARN_ELF("gelf_getrela");
return -1;
}
- reloc->type = GELF_R_TYPE(reloc->rela.r_info);
+ elf_reloc_set_type(reloc, GELF_R_TYPE(reloc->rela.r_info));
reloc->addend = reloc->rela.r_addend;
reloc->offset = reloc->rela.r_offset;
*symndx = GELF_R_SYM(reloc->rela.r_info);
@@ -1471,3 +1471,8 @@ void elf_close(struct elf *elf)
free(elf->section_data);
free(elf);
}
+
+void elf_reloc_set_type(struct reloc *reloc, int type)
+{
+ reloc->type = type;
+}
diff --git a/tools/objtool/include/objtool/elf.h b/tools/objtool/include/objtool/elf.h
index 1c90f0ac0d53..33ec6cf72325 100644
--- a/tools/objtool/include/objtool/elf.h
+++ b/tools/objtool/include/objtool/elf.h
@@ -83,6 +83,8 @@ struct reloc {
bool jump_table_start;
};
+void elf_reloc_set_type(struct reloc *reloc, int type);
+
#define ELF_HASH_BITS 20
struct elf {
--
2.39.0
On Tue, Dec 27, 2022 at 04:00:57PM +0000, Thomas Wei?schuh wrote:
> The processing of vmlinux.o with objtool is the most memory-intensive step
> of a kernel build. By reducing the maximum memory usage here we can reduce
> the maximum memory usage of the whole kernel build.
> Therefore memory pressure on memory starved machines is relieved during
> kernel builds and the build is faster as less swapping has to occur.
Friendly ping.
These patches can also applied one by one, the only dependency is from
patch 5 to patch 4.
Thanks,
Thomas
> To: Josh Poimboeuf <[email protected]>
> To: Peter Zijlstra <[email protected]>
> Cc: [email protected]
> Signed-off-by: Thomas Wei?schuh <[email protected]>
>
> ---
> Changes in v2:
> - Warn on out of range values for reloc->type
> - Also reduce size of struct special_alt
> - Note: v1 did not make it to the lists, only to individual recipients
>
> ---
> Thomas Wei?schuh (8):
> objtool: make struct entries[] static and const
> objtool: make struct check_options static
> objtool: allocate multiple structures with calloc()
> objtool: introduce function elf_reloc_set_type
> objtool: reduce memory usage of struct reloc
> objtool: optimize layout of struct symbol
> objtool: optimize layout of struct special_alt
> objtool: explicitly cleanup resources on success
>
> tools/objtool/builtin-check.c | 4 ++-
> tools/objtool/check.c | 6 ++--
> tools/objtool/elf.c | 56 +++++++++++++++++++--------------
> tools/objtool/include/objtool/builtin.h | 2 --
> tools/objtool/include/objtool/elf.h | 13 +++++---
> tools/objtool/include/objtool/special.h | 2 +-
> tools/objtool/special.c | 6 ++--
> 7 files changed, 51 insertions(+), 38 deletions(-)
> ---
> base-commit: 1b929c02afd37871d5afb9d498426f83432e71c2
> change-id: 20221216-objtool-memory-06db3b8bf111
>
> Best regards,
> --
> Thomas Wei?schuh <[email protected]>
On Sun, Jan 29, 2023 at 09:43:39PM +0000, Thomas Weißschuh wrote:
> On Tue, Dec 27, 2022 at 04:00:57PM +0000, Thomas Weißschuh wrote:
> > The processing of vmlinux.o with objtool is the most memory-intensive step
> > of a kernel build. By reducing the maximum memory usage here we can reduce
> > the maximum memory usage of the whole kernel build.
> > Therefore memory pressure on memory starved machines is relieved during
> > kernel builds and the build is faster as less swapping has to occur.
>
> Friendly ping.
>
> These patches can also applied one by one, the only dependency is from
> patch 5 to patch 4.
Thanks, I'll go ahead and take five of them now.
--
Josh
On Mon, Jan 30, 2023 at 04:03:56PM -0800, Josh Poimboeuf wrote:
> On Sun, Jan 29, 2023 at 09:43:39PM +0000, Thomas Wei?schuh wrote:
> > On Tue, Dec 27, 2022 at 04:00:57PM +0000, Thomas Wei?schuh wrote:
> > > The processing of vmlinux.o with objtool is the most memory-intensive step
> > > of a kernel build. By reducing the maximum memory usage here we can reduce
> > > the maximum memory usage of the whole kernel build.
> > > Therefore memory pressure on memory starved machines is relieved during
> > > kernel builds and the build is faster as less swapping has to occur.
> >
> > Friendly ping.
> >
> > These patches can also applied one by one, the only dependency is from
> > patch 5 to patch 4.
>
> Thanks, I'll go ahead and take five of them now.
Thanks.
I have another half-finished series that replaces the doubly-linked
list_heads used by elf.h with a custom singly-linked list.
This would save a few pointers per struct.
Do you think this is worth it?
Thomas
On Tue, Jan 31, 2023 at 03:54:42AM +0000, Thomas Weißschuh wrote:
> On Mon, Jan 30, 2023 at 04:03:56PM -0800, Josh Poimboeuf wrote:
> > On Sun, Jan 29, 2023 at 09:43:39PM +0000, Thomas Weißschuh wrote:
> > > On Tue, Dec 27, 2022 at 04:00:57PM +0000, Thomas Weißschuh wrote:
> > > > The processing of vmlinux.o with objtool is the most memory-intensive step
> > > > of a kernel build. By reducing the maximum memory usage here we can reduce
> > > > the maximum memory usage of the whole kernel build.
> > > > Therefore memory pressure on memory starved machines is relieved during
> > > > kernel builds and the build is faster as less swapping has to occur.
> > >
> > > Friendly ping.
> > >
> > > These patches can also applied one by one, the only dependency is from
> > > patch 5 to patch 4.
> >
> > Thanks, I'll go ahead and take five of them now.
>
> Thanks.
>
> I have another half-finished series that replaces the doubly-linked
> list_heads used by elf.h with a custom singly-linked list.
> This would save a few pointers per struct.
>
> Do you think this is worth it?
Maybe, depending on the memory savings.
--
Josh
From: Thomas Weißschuh.
> Sent: 31 January 2023 03:55
...
> I have another half-finished series that replaces the doubly-linked
> list_heads used by elf.h with a custom singly-linked list.
> This would save a few pointers per struct.
>
> Do you think this is worth it?
If you allocate the structures in blocks of (say) 256 you
can use an array of pointers to the blocks and then a
32bit index instead of a 64bit pointer.
For real space-saving you might decide that the index can
never exceed 2^^24 and use a bitfield!
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
On Tue, Jan 31, 2023 at 09:27:15AM -0800, Josh Poimboeuf wrote:
> On Tue, Jan 31, 2023 at 03:54:42AM +0000, Thomas Weißschuh wrote:
> > On Mon, Jan 30, 2023 at 04:03:56PM -0800, Josh Poimboeuf wrote:
> > > On Sun, Jan 29, 2023 at 09:43:39PM +0000, Thomas Weißschuh wrote:
> > > > On Tue, Dec 27, 2022 at 04:00:57PM +0000, Thomas Weißschuh wrote:
> > > > > The processing of vmlinux.o with objtool is the most memory-intensive step
> > > > > of a kernel build. By reducing the maximum memory usage here we can reduce
> > > > > the maximum memory usage of the whole kernel build.
> > > > > Therefore memory pressure on memory starved machines is relieved during
> > > > > kernel builds and the build is faster as less swapping has to occur.
> > > >
> > > > Friendly ping.
> > > >
> > > > These patches can also applied one by one, the only dependency is from
> > > > patch 5 to patch 4.
> > >
> > > Thanks, I'll go ahead and take five of them now.
> >
> > Thanks.
> >
> > I have another half-finished series that replaces the doubly-linked
> > list_heads used by elf.h with a custom singly-linked list.
> > This would save a few pointers per struct.
> >
> > Do you think this is worth it?
>
> Maybe, depending on the memory savings.
FYI, there were more memory usage complaints, so Peter worked up a nice
series to do this and more:
https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/log/?h=objtool/core
--
Josh