[PATCH] Change BINPKG_COMPRESS default from bzip2 to xz

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[PATCH] Change BINPKG_COMPRESS default from bzip2 to xz

Matt Turner-5
Bug: https://bugs.gentoo.org/715108
Signed-off-by: Matt Turner <[hidden email]>
---
Strawman patch. Bikeshed away.

 bin/quickpkg                           | 2 +-
 lib/portage/package/ebuild/doebuild.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/bin/quickpkg b/bin/quickpkg
index df8c1a8e8..1feb01fa1 100755
--- a/bin/quickpkg
+++ b/bin/quickpkg
@@ -115,7 +115,7 @@ def quickpkg_atom(options, infos, arg, eout):
  binpkg_tmpfile = os.path.join(bintree.pkgdir,
  cpv + ".tbz2." + str(os.getpid()))
  ensure_dirs(os.path.dirname(binpkg_tmpfile))
- binpkg_compression = settings.get("BINPKG_COMPRESS", "bzip2")
+ binpkg_compression = settings.get("BINPKG_COMPRESS", "xz")
  try:
  compression = _compressors[binpkg_compression]
  except KeyError as e:
diff --git a/lib/portage/package/ebuild/doebuild.py b/lib/portage/package/ebuild/doebuild.py
index 2bff94cb1..5020b46fe 100644
--- a/lib/portage/package/ebuild/doebuild.py
+++ b/lib/portage/package/ebuild/doebuild.py
@@ -539,7 +539,7 @@ def doebuild_environment(myebuild, mydo, myroot=None, settings=None,
  mysettings["KV"] = ""
  mysettings.backup_changes("KV")
 
- binpkg_compression = mysettings.get("BINPKG_COMPRESS", "bzip2")
+ binpkg_compression = mysettings.get("BINPKG_COMPRESS", "xz")
  try:
  compression = _compressors[binpkg_compression]
  except KeyError as e:
--
2.25.3


Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Change BINPKG_COMPRESS default from bzip2 to xz

Michael Orlitzky
On 4/26/20 12:55 PM, Matt Turner wrote:
> Bug: https://bugs.gentoo.org/715108
> Signed-off-by: Matt Turner <[hidden email]>
> ---
> Strawman patch. Bikeshed away.
>

Fuel for the fire:

 * https://www.nongnu.org/lzip/lzip_benchmark.html
 * https://www.nongnu.org/lzip/xz_inadequate.html

Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Change BINPKG_COMPRESS default from bzip2 to xz

Ulrich Mueller-2
>>>>> On Sun, 26 Apr 2020, Michael Orlitzky wrote:

> Fuel for the fire:

>  * https://www.nongnu.org/lzip/lzip_benchmark.html
>  * https://www.nongnu.org/lzip/xz_inadequate.html

Yep. That's why lzip is the dominant compression format now, and nobody
is using xz any more.

SCNR,
Ulrich

signature.asc (517 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Change BINPKG_COMPRESS default from bzip2 to xz

Michał Górny-5
In reply to this post by Matt Turner-5
On Sun, 2020-04-26 at 09:55 -0700, Matt Turner wrote:
> Bug: https://bugs.gentoo.org/715108
> Signed-off-by: Matt Turner <[hidden email]>
> ---
> Strawman patch. Bikeshed away.
>

xz is generally slow and doesn't do parallel good.  If we want to change
this, we should go for something cool like zstd that scales better.

--
Best regards,
Michał Górny


signature.asc (631 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Change BINPKG_COMPRESS default from bzip2 to xz

Fabian Groffen-2
On 26-04-2020 21:29:42 +0200, Michał Górny wrote:
> On Sun, 2020-04-26 at 09:55 -0700, Matt Turner wrote:
> > Bug: https://bugs.gentoo.org/715108
> > Signed-off-by: Matt Turner <[hidden email]>
> > ---
> > Strawman patch. Bikeshed away.
> >
>
> xz is generally slow and doesn't do parallel good.  If we want to change
> this, we should go for something cool like zstd that scales better.

I'd go for zstd too.  It seems to be the best of both worlds, good
compression at a good speed.

Thanks,
Fabian


--
Fabian Groffen
Gentoo on a different level

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Change BINPKG_COMPRESS default from bzip2 to xz

Matt Turner-5
In reply to this post by Michał Górny-5
On Sun, Apr 26, 2020 at 12:29 PM Michał Górny <[hidden email]> wrote:

>
> On Sun, 2020-04-26 at 09:55 -0700, Matt Turner wrote:
> > Bug: https://bugs.gentoo.org/715108
> > Signed-off-by: Matt Turner <[hidden email]>
> > ---
> > Strawman patch. Bikeshed away.
> >
>
> xz is generally slow and doesn't do parallel good.  If we want to change
> this, we should go for something cool like zstd that scales better.

That works for me.

Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Change BINPKG_COMPRESS default from bzip2 to xz

Zac Medico-2
On 4/26/20 1:09 PM, Matt Turner wrote:

> On Sun, Apr 26, 2020 at 12:29 PM Michał Górny <[hidden email]> wrote:
>>
>> On Sun, 2020-04-26 at 09:55 -0700, Matt Turner wrote:
>>> Bug: https://bugs.gentoo.org/715108
>>> Signed-off-by: Matt Turner <[hidden email]>
>>> ---
>>> Strawman patch. Bikeshed away.
>>>
>>
>> xz is generally slow and doesn't do parallel good.  If we want to change
>> this, we should go for something cool like zstd that scales better.
>
> That works for me.
>
Let's make the default conditional on zstd USE flag as suggested in bug
719456. It will be similar to how the ebuild enables FEATURES=xattr for
USE=xattr.
--
Thanks,
Zac


signature.asc (1000 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Change BINPKG_COMPRESS default from bzip2 to xz

Michael Orlitzky
In reply to this post by Ulrich Mueller-2
On 4/26/20 3:25 PM, Ulrich Mueller wrote:

>>>>>> On Sun, 26 Apr 2020, Michael Orlitzky wrote:
>
>> Fuel for the fire:
>
>>  * https://www.nongnu.org/lzip/lzip_benchmark.html
>>  * https://www.nongnu.org/lzip/xz_inadequate.html
>
> Yep. That's why lzip is the dominant compression format now, and nobody
> is using xz any more.
>

If you believed that argument, you would be using Windows =P

Dude has GRAPHS, it must be true.

Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Change BINPKG_COMPRESS default from bzip2 to xz

Michał Górny-5
On Sun, 2020-04-26 at 18:23 -0400, Michael Orlitzky wrote:

> On 4/26/20 3:25 PM, Ulrich Mueller wrote:
> > > > > > > On Sun, 26 Apr 2020, Michael Orlitzky wrote:
> > > Fuel for the fire:
> > >  * https://www.nongnu.org/lzip/lzip_benchmark.html
> > >  * https://www.nongnu.org/lzip/xz_inadequate.html
> >
> > Yep. That's why lzip is the dominant compression format now, and nobody
> > is using xz any more.
> >
>
> If you believed that argument, you would be using Windows =P
>
> Dude has GRAPHS, it must be true.
>
It is true but nobody cares.  People managed to shove a lot of unused
code into the kernel and nobody cares.

--
Best regards,
Michał Górny


signature.asc (631 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Change BINPKG_COMPRESS default from bzip2 to xz

Zac Medico-2
In reply to this post by Zac Medico-2
On 4/26/20 2:12 PM, Zac Medico wrote:

> On 4/26/20 1:09 PM, Matt Turner wrote:
>> On Sun, Apr 26, 2020 at 12:29 PM Michał Górny <[hidden email]> wrote:
>>>
>>> On Sun, 2020-04-26 at 09:55 -0700, Matt Turner wrote:
>>>> Bug: https://bugs.gentoo.org/715108
>>>> Signed-off-by: Matt Turner <[hidden email]>
>>>> ---
>>>> Strawman patch. Bikeshed away.
>>>>
>>>
>>> xz is generally slow and doesn't do parallel good.  If we want to change
>>> this, we should go for something cool like zstd that scales better.
>>
>> That works for me.
>>
>
> Let's make the default conditional on zstd USE flag as suggested in bug
> 719456. It will be similar to how the ebuild enables FEATURES=xattr for
> USE=xattr.

In https://github.com/gentoo/gentoo/pull/15520 the issue was raised that
it's probably not a good idea to change this default via the zstd USE
flags, since changing the format of binary packages is kind of a big
deal (it affects consumers of any binary packages that you build).

I think it might be a good idea to make this new default apply only to
new installations, like we did when changing default locations for bug
378603.

If we're changing the default, then all architectures need keywords, so
I've opened https://bugs.gentoo.org/721960.
--
Thanks,
Zac