XZ embedded bug unpacking linux-3.8.tar.xz

[Antonio Diaz Diaz]

Hello Denys et all.

Denys Vlasenko wrote:
> On Mon, Feb 25, 2013 at 7:20 PM, Matias A. Fonzo <selk at dragora.org> wrote:
>>Can be lzip considered for inclusion in busybox?:
[...]
> Matias, sure, this can be done.
> 
> But bbox already has *two* LZMA decompressors.
> Feels wrong, isn't it?

It certainly feels wrong, but those two are in reality the same one, 
which suffered a radical and bumpy transformation. (Do you remember 
lzma-4.42, which had a format incompatible with both lzma-4.32 and xz?)

Some people think that adding lzma support to GNU tools was a mistake. I 
think that adding xz support was simply the continuation of the same 
mistake.

As lzma is legacy software, I guess it will be eventually removed from 
Busybox, just as it is being removed from GNU packages: "The deprecated 
'lzma' compression format for distribution archives has been removed, in 
favor of 'xz' and 'lzip'"[1].

[1] http://lists.gnu.org/archive/html/automake/2012-04/msg00060.html

The xz decompressor included in Busybox is not able to decompress all 
valid xz files because it only understands the xz-embedded subset of the 
xz format. Therefore, any user wanting to decompress or check the 
integrity of "real" xz files needs to install the full xz!

None of the other formats (bzip2, gzip, lzip) have this problem (the 
lunzip proposed for Busybox is able to decompress and check any .lz 
file, even those produced by the parallel version of lzip, plzip). And 
it can only get worse for xz, because "It is possible and even somewhat 
likely that new features will be added in the future which old programs 
won't support"[2].

[2] http://www.mail-archive.com/xz-devel@tukaani.org/msg00059.html


> In the long run it would be a nightmare to have two
> or more LZMA (de)compressors in common use on Linux.

Agreed.


> What happened between lzip and xz? Are they incompatible?
> On what level? File format, or compression stream format too?

The history in a nutshell: "In 2008, Antonio Diaz released lzip, which 
uses a proper container format with checksums and magic numbers instead 
of the raw LZMA data stream, providing a complete Unix-style solution 
for using LZMA. Nevertheless, LZMA Utils was extended to have similar 
features and then renamed to XZ Utils"[3].

[3] http://en.wikipedia.org/wiki/Lzip

Lzip and xz are totally incompatible. Lzip uses the same stream format 
that .lzma files, just with proper header and trailer. Xz is a complex 
container format derived from 7-zip (or at least inspired by it) and 
without any resemblance to the old .lzma format.

Lzip is a compressor, just like gzip and bzip2.

Xz is much more complex than that. Even the stripped-down version of 
unxz included in Busybox is already larger than any of the other 
decompressors.

IMHO all this leaves lzip as the LZMA compressor most suitable for 
Unix-like systems in general, and for Busybox in particular.


Best regards,
Antonio.