Hank Leininger wrote:
> On 2024-04-27, Jacob Bachmeyer wrote:
>
>
>>> - Output is manageable; able to rule out all hits not part of the
>>> actual xz-utils backdoors as false positives.
>>>
>> This is what I would expect: the backdoor dropper appears to have
>> been specifically developed for xz-utils, but could /possibly/ be
>> adaptable to other compression tools.
>>
>
> Indeed, well my thinking was more along the lines of: "This is an
> impressive amount of moving parts to be created new for this project,
> and burned for just this project. What if what we are seeing is the 2nd
> or 3rd gen of such a toolkit, and earlier ones have some similar
> characteristics but maybe fewer layers, etc. so we could spot them?"
I disagree here because, while I agree that there are quite a few moving
parts, I also (having replicated unpacking the dropper shell scripts)
see incremental development paths, and a few major mistakes that a more
polished backdoor could have avoided. If this were a 2nd or 3rd
generation toolkit, those mistakes would have been fixed.
Overall, I think the "Jia Tan" crew went straight for the "Golden Key"
and bit off /way/ more than they could chew.
> [...] while the liblzma decompressor had
> certain things going for it as a target, why not other things? (Nor am I
> restricting to "other things that would get linked into sshd", but
> really more broadly.)
>
I disagree with this assessment: the backdoor blob, according to
reports so far, does not appear to actually affect xz or liblzma /at/
/all/; liblzma is merely used (as a dependency of libsystemd) to smuggle
the blob into the sshd process and pass control into the blob during
process initialization. From the reports that I have seen, the
entrypoint that liblzma is patched to call does very little itself, and
only modifies ld.so's data segment to half-register the blob with the
LD_AUDIT framework so it will be called again later. Nearly all of the
backdoor seems to be independent of liblzma.
> [...] The
> xz-utils stuff is many generations ahead of those; it would be
> interesting to spot some missing links.
>
I suspect that the "missing links" you seek are to be found in various
Windows malware over the years, at least as far as the binary backdoor
blob is involved, but I have not been directly involved in that
analysis. From the reports I have read, it seems to be more-or-less a
piece of Windows malware adapted to an ELF environment. The adaptations
also seem to be exactly the parts that went poorly, causing performance
problems that led to the backdoor's discovery. (Or ld.so really is that
much more efficient than the Windows module loader, and performance
issues that are routinely lost in the noise on Windows got the "Jia Tan"
crew caught.)
The dropper scripts do not appear well-developed to me. Overall, they
/do/ strike me as a first-generation effort, likely the authors' first
shell scripts on top of that, and evidence a poor understanding of the
GNU system. There are worse-than-beginner mistakes in there, like
omitting the spaces around the '=' in a string comparison, and overall
patterns that suggest a poor understanding of shell scripting. For
example, a common idiom in shell programming is to use && and || for
conditional execution of single commands, especially conditional exits;
the dropper uses this idiom once, getting the test wrong (the
aforementioned missing spaces), but then uses if/then/fi later for
conditional exits. The dropper scripts also appear to be the product of
copy-and-paste cargo cult programming: some commands are written as
"[...] || true" even though their exit status will actually be ignored,
suggesting that they were adapted from a Makefile or a script that runs
with "set -e" in effect. Lastly, the outer dropper uses a linear series
of head(1) commands to "skip 1KiB of garbage, extract 2KiB of backdoor,
repeat" but simply reading the manuals would have revealed more
appropriate tools for that---and a way to make the outer dropper
independent of the inner dropper's length.
>> You might get better results by indexing macro definitions found in
>> *.m4 files, instead of trying to fuzzily hash the files. The
>> interesting comparison is then different definitions of macros with
>> the same name.
>>
>
> I like this a lot as a potential next layer for the m4 reconciliation.
> Essentially a field-level matching once things that match at a
> file-level have been eliminated. I don't see why (he says, not actually
> having dug into the m4 format much) we couldn't break apart all the m4s
> we are choosing to consider known-good, and catalog each individual
> macro, and then do the same when bashing project-specific files. Could
> be we can entirely "clear" a file with an unknown checksum because it
> consists 100% of idnividual macros that are known.
More likely, you would be able to "clear" a file that actually differs
in either whitespace or comments---or catch a file that has some
nastiness inserted between macro definitions.
> (Insert weird machine
> here that combined OK macros in surprising ways.)
>
While m4 technically is Turing-complete, I strongly doubt the existence
of m4 weird machines: m4 is a macro processor that performs text
expansion. You should have a manual for it in the Info system.
>> many (most?) modern Linux kernels are compressed using xz, which means
>> that a Thompsonesque attack could binary-patch a freshly built kernel
>> while compressing vmlinux to make vmlinuz.
>>
>
> Good call. This may be far out on the.... "far out" scale, but it'd be
> pretty trivial to harvest distro kernels that used xz to make their
> vmlinuz, and then run those through multiple independent implementations
> like we have done with .tar.xz files. Sold.
>
Make sure that at least one of those implementations is a wrapper around
the xz-embedded decompresser that Linux itself uses. (As in, mmap()
vmlinuz MAP_PRIVATE, mmap an output file at the appropriate virtual
address, patch the jump into the kernel at the end of the decompression
stub with a return, and call the embedded decompresser.) If you are
concerned about weird machines, there is a tiny, paranoid, possibility
that a tampered compressed stream would decompress "clean" with any
other decompresser.
I doubt that you will find anything, though. The "Jia Tan" crew does
not seem to have been that advanced.
>> The IFUNC mechanism is actually a security feature. In "inner-loop"
>> code, having multiple implementations with different optimizations
>> with the preferred implementation for the local processor chosen at
>> runtime is fairly common.
>>
>
> Thanks for this! I've seen that discussed as a (valid, useful) use of
> IFUNC but also AFAIK things like musl don't implement such a thing, so
> either software that wants it just doesn't support musl, or can't pick
> an optimization, or does so in an undesirable way like writable pointers
> in the data segment or... some other option?
When IFUNC usage was added to xz-utils, the older "dispatch through a
function pointer" technique was kept as a fallback when building without
IFUNC.
> [...] I'd be satisfied being able to say "IFUNCs are
> used by 15 out of 10,000 packages; that's a small enough number we can
> a) audit them all b) add alerting to tooling used for builds; when a
> package suddenly starts using them, look into it." If the real number
> turns out to be 1,000 out of 10,000, then that's good to know, and
> probably give up.
>
Alerting should be simple matter of `grep -Ri ifunc .` at the top of the
package tree.
>> I currently suspect that the crackers used IFUNC support as a covert
>> flag. The "jankiness" of the current glibc IFUNC implementation
>> provided a convenient excuse to ask oss-fuzz to --disable-ifunc when
>> building xz-utils, which *also* conveniently inhibited the backdoor
>> dropper and ensured that the fuzzing builds would not contain the
>> backdoor.
>>
>
> Uncynically, I like this conspiracy theory.
>
Then I will go a step farther: I half-suspect that the entire
CLMUL-based CRC calculation may have been added as an excuse to add the
dispatch logic, which in turn was an excuse to use IFUNC. While I have
not run tests, I suspect that the baseline table-driven CRC calculation
is probably already I/O-bound on modern processors, especially x86-64
processors new enough to /have/ CLMUL.
> [...]
> I think Sam looked into existing pkg-config verifiers and found they do
> not complain about things we thought they should complain about (this
> could just mean we misunderstand their purpose). A strict lint-checker
> for such files would be better than just checking for specific
> suspicious patterns. But, I don't yet know how strict a format we could
> insist on (would it turn out 10% of files in fact break what we
> initially think are reasonable rules?). Even still, I think you could
> embed badness in legit variables, although I haven't dug in enough to
> know that for sure.
>
Quoting pkg-config(1):
> Files have two kinds of line: keyword lines start with a keyword plus a
> colon, and variable definitions start with an alphanumeric string plus
> an equals sign. Keywords are defined in advance and have special
> meaning to /pkg-config/; variables do not, you can have any variables
> that you wish (however, users may expect to retrieve the usual
> directory name variables).
The format is clearly defined; any pkg-config file containing nonblank
lines /not/ matching the above description should be rejected, and
pkg-config itself should implement this check. This rule ensures that a
pkg-config file cannot also contain shell commands.
The sample backdoor also uses an *-uninstalled.pc file to override a
legitimate file, but places the "uninstalled" file in a system
directory, which means it is actually installed! The pkg-config program
should also refuse to read *-uninstalled.pc files from system
directories, and emit a warning if such a file is found to exist.
-- Jacob