[PATCH] Default BINPKG_COMPRESSION to zstd (bug 715108)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[PATCH] Default BINPKG_COMPRESSION to zstd (bug 715108)

Zac Medico-2
This includes a _compat_upgrade.binpkg_compression script that the
ebuild can call in pkg_preinst in order to maintain a backward-compatible
bzip2 default when appropriate, ensuring that binary package consumers
are not caught off guard.

Bug: https://bugs.gentoo.org/715108
Signed-off-by: Zac Medico <[hidden email]>
---
 .travis.yml                                   |  4 ++
 cnf/make.globals                              |  5 ++-
 .../_compat_upgrade/binpkg_compression.py     | 40 +++++++++++++++++++
 .../tests/resolver/ResolverPlayground.py      |  1 +
 man/make.conf.5                               |  4 +-
 5 files changed, 50 insertions(+), 4 deletions(-)
 create mode 100644 lib/portage/_compat_upgrade/binpkg_compression.py

diff --git a/.travis.yml b/.travis.yml
index 9269d4034..2132c8c87 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,3 +1,4 @@
+dist: bionic
 language: python
 python:
     - 2.7
@@ -8,6 +9,9 @@ python:
     - pypy3
 
 # command to install dependencies
+before_install:
+    # Use "dist: bionic" to get a zstd with --long support.
+    - sudo apt-get -y install zstd
 install:
     - pip install tox
 
diff --git a/cnf/make.globals b/cnf/make.globals
index 4a59dbe3c..dd3f28f70 100644
--- a/cnf/make.globals
+++ b/cnf/make.globals
@@ -34,8 +34,9 @@ RPMDIR="/var/cache/rpm"
 # Temporary build directory
 PORTAGE_TMPDIR="/var/tmp"
 
-# The compression used for binary packages. Defaults to zstd when USE=zstd is enabled.
-BINPKG_COMPRESS="bzip2"
+# The compression used for binary packages. Defaults to zstd except for
+# existing installs where bzip2 is used for backward compatibility.
+BINPKG_COMPRESS="zstd"
 
 # Fetching command (3 tries, passive ftp for firewall compatibility)
 FETCHCOMMAND="wget -t 3 -T 60 --passive-ftp -O \"\${DISTDIR}/\${FILE}\" \"\${URI}\""
diff --git a/lib/portage/_compat_upgrade/binpkg_compression.py b/lib/portage/_compat_upgrade/binpkg_compression.py
new file mode 100644
index 000000000..0f5704733
--- /dev/null
+++ b/lib/portage/_compat_upgrade/binpkg_compression.py
@@ -0,0 +1,40 @@
+# Copyright 2020 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+import re
+
+import portage
+from portage import os
+from portage.const import GLOBAL_CONFIG_PATH
+
+COMPAT_BINPKG_COMPRESS = 'bzip2'
+
+
+def main():
+ """
+ If the current installation is still configured to use the old
+ default BINPKG_COMPRESS=bzip2 setting, then patch make.globals
+ inside ${ED} to maintain backward compatibility, ensuring that
+ binary package consumers are not caught off guard. This is
+ intended to be called from the ebuild as follows:
+
+ pkg_preinst() {
+ python_setup
+ env -u BINPKG_COMPRESS
+ PYTHONPATH="${D%/}$(python_get_sitedir)${PYTHONPATH:+:${PYTHONPATH}}" \
+ "${PYTHON}" -m portage._compat_upgrade.binpkg_compression || die
+ }
+ """
+ if portage.settings.get('BINPKG_COMPRESS', COMPAT_BINPKG_COMPRESS) == COMPAT_BINPKG_COMPRESS:
+ config_path = os.path.join(os.environ['ED'], GLOBAL_CONFIG_PATH.lstrip(os.sep), 'make.globals')
+ with open(config_path) as f:
+ content = f.read()
+ compat_setting = 'BINPKG_COMPRESS="{}"'.format(COMPAT_BINPKG_COMPRESS)
+ portage.output.EOutput().einfo('Setting make.globals default {} for backward compatibility'.format(compat_setting))
+ content = re.sub('^BINPKG_COMPRESS=.*$', compat_setting, content, flags=re.MULTILINE)
+ with open(config_path, 'wt') as f:
+ f.write(content)
+
+
+if __name__ == '__main__':
+ main()
diff --git a/lib/portage/tests/resolver/ResolverPlayground.py b/lib/portage/tests/resolver/ResolverPlayground.py
index 98831e000..de80a0cc1 100644
--- a/lib/portage/tests/resolver/ResolverPlayground.py
+++ b/lib/portage/tests/resolver/ResolverPlayground.py
@@ -112,6 +112,7 @@ class ResolverPlayground(object):
  "uname",
  "uniq",
  "xargs",
+ "zstd",
  )
  # Exclude internal wrappers from PATH lookup.
  orig_path = os.environ['PATH']
diff --git a/man/make.conf.5 b/man/make.conf.5
index f82fed65a..a3bd662ae 100644
--- a/man/make.conf.5
+++ b/man/make.conf.5
@@ -1,4 +1,4 @@
-.TH "MAKE.CONF" "5" "Nov 2019" "Portage VERSION" "Portage"
+.TH "MAKE.CONF" "5" "May 2020" "Portage VERSION" "Portage"
 .SH "NAME"
 make.conf \- custom settings for Portage
 .SH "SYNOPSIS"
@@ -115,7 +115,7 @@ This variable is used to determine the compression used for \fIbinary
 packages\fR. Supported settings and compression algorithms are: bzip2, gzip,
 lz4, lzip, lzop, xz, zstd.
 .br
-Defaults to "bzip2".
+Defaults to "zstd".
 .br
 .I Example:
 .nf
--
2.25.3


Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Default BINPKG_COMPRESSION to zstd (bug 715108)

Brian Dolbec-3
On Sun, 10 May 2020 19:29:34 -0700
Zac Medico <[hidden email]> wrote:

> This includes a _compat_upgrade.binpkg_compression script that the
> ebuild can call in pkg_preinst in order to maintain a
> backward-compatible bzip2 default when appropriate, ensuring that
> binary package consumers are not caught off guard.
>
> Bug: https://bugs.gentoo.org/715108
> Signed-off-by: Zac Medico <[hidden email]>
> ---
>  .travis.yml                                   |  4 ++
>  cnf/make.globals                              |  5 ++-
>  .../_compat_upgrade/binpkg_compression.py     | 40
> +++++++++++++++++++ .../tests/resolver/ResolverPlayground.py      |
> 1 + man/make.conf.5                               |  4 +-
>  5 files changed, 50 insertions(+), 4 deletions(-)
>  create mode 100644 lib/portage/_compat_upgrade/binpkg_compression.py
>
> diff --git a/.travis.yml b/.travis.yml
> index 9269d4034..2132c8c87 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -1,3 +1,4 @@
> +dist: bionic
>  language: python
>  python:
>      - 2.7
> @@ -8,6 +9,9 @@ python:
>      - pypy3
>  
>  # command to install dependencies
> +before_install:
> +    # Use "dist: bionic" to get a zstd with --long support.
> +    - sudo apt-get -y install zstd
>  install:
>      - pip install tox
>  
> diff --git a/cnf/make.globals b/cnf/make.globals
> index 4a59dbe3c..dd3f28f70 100644
> --- a/cnf/make.globals
> +++ b/cnf/make.globals
> @@ -34,8 +34,9 @@ RPMDIR="/var/cache/rpm"
>  # Temporary build directory
>  PORTAGE_TMPDIR="/var/tmp"
>  
> -# The compression used for binary packages. Defaults to zstd when
> USE=zstd is enabled. -BINPKG_COMPRESS="bzip2"
> +# The compression used for binary packages. Defaults to zstd except
> for +# existing installs where bzip2 is used for backward
> compatibility. +BINPKG_COMPRESS="zstd"
>  
>  # Fetching command (3 tries, passive ftp for firewall compatibility)
>  FETCHCOMMAND="wget -t 3 -T 60 --passive-ftp -O
> \"\${DISTDIR}/\${FILE}\" \"\${URI}\"" diff --git
> a/lib/portage/_compat_upgrade/binpkg_compression.py
> b/lib/portage/_compat_upgrade/binpkg_compression.py new file mode
> 100644 index 000000000..0f5704733 --- /dev/null
> +++ b/lib/portage/_compat_upgrade/binpkg_compression.py
> @@ -0,0 +1,40 @@
> +# Copyright 2020 Gentoo Authors
> +# Distributed under the terms of the GNU General Public License v2
> +
> +import re
> +
> +import portage
> +from portage import os
> +from portage.const import GLOBAL_CONFIG_PATH
> +
> +COMPAT_BINPKG_COMPRESS = 'bzip2'
> +
> +
> +def main():
> + """
> + If the current installation is still configured to use the
> old
> + default BINPKG_COMPRESS=bzip2 setting, then patch
> make.globals
> + inside ${ED} to maintain backward compatibility, ensuring
> that
> + binary package consumers are not caught off guard. This is
> + intended to be called from the ebuild as follows:
> +
> + pkg_preinst() {
> + python_setup
> + env -u BINPKG_COMPRESS
> +
> PYTHONPATH="${D%/}$(python_get_sitedir)${PYTHONPATH:+:${PYTHONPATH}}"
> \
> + "${PYTHON}" -m
> portage._compat_upgrade.binpkg_compression || die
> + }
> + """
> + if portage.settings.get('BINPKG_COMPRESS',
> COMPAT_BINPKG_COMPRESS) == COMPAT_BINPKG_COMPRESS:
> + config_path = os.path.join(os.environ['ED'],
> GLOBAL_CONFIG_PATH.lstrip(os.sep), 'make.globals')
> + with open(config_path) as f:
> + content = f.read()
> + compat_setting =
> 'BINPKG_COMPRESS="{}"'.format(COMPAT_BINPKG_COMPRESS)
> + portage.output.EOutput().einfo('Setting
> make.globals default {} for backward
> compatibility'.format(compat_setting))
> + content = re.sub('^BINPKG_COMPRESS=.*$',
> compat_setting, content, flags=re.MULTILINE)
> + with open(config_path, 'wt') as f:
> + f.write(content)
> +
> +
> +if __name__ == '__main__':
> + main()
> diff --git a/lib/portage/tests/resolver/ResolverPlayground.py
> b/lib/portage/tests/resolver/ResolverPlayground.py index
> 98831e000..de80a0cc1 100644 ---
> a/lib/portage/tests/resolver/ResolverPlayground.py +++
> b/lib/portage/tests/resolver/ResolverPlayground.py @@ -112,6 +112,7
> @@ class ResolverPlayground(object): "uname",
>   "uniq",
>   "xargs",
> + "zstd",
>   )
>   # Exclude internal wrappers from PATH lookup.
>   orig_path = os.environ['PATH']
> diff --git a/man/make.conf.5 b/man/make.conf.5
> index f82fed65a..a3bd662ae 100644
> --- a/man/make.conf.5
> +++ b/man/make.conf.5
> @@ -1,4 +1,4 @@
> -.TH "MAKE.CONF" "5" "Nov 2019" "Portage VERSION" "Portage"
> +.TH "MAKE.CONF" "5" "May 2020" "Portage VERSION" "Portage"
>  .SH "NAME"
>  make.conf \- custom settings for Portage
>  .SH "SYNOPSIS"
> @@ -115,7 +115,7 @@ This variable is used to determine the
> compression used for \fIbinary packages\fR. Supported settings and
> compression algorithms are: bzip2, gzip, lz4, lzip, lzop, xz, zstd.
>  .br
> -Defaults to "bzip2".
> +Defaults to "zstd".
>  .br
>  .I Example:
>  .nf
looks good, I've not tested it, but changes are minor

Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Default BINPKG_COMPRESSION to zstd (bug 715108)

Francesco Riosa-3

Il 11/05/20 22:21, Brian Dolbec ha scritto:
> On Sun, 10 May 2020 19:29:34 -0700
> Zac Medico <[hidden email]> wrote:
>
>> This includes a _compat_upgrade.binpkg_compression script that the
>> ebuild can call in pkg_preinst in order to maintain a
>> backward-compatible bzip2 default when appropriate, ensuring that
>> binary package consumers are not caught off guard.

[snip]

For your interest, the binpkg archive for a small LAMP container goes
from 659 to 540 MB, the "--rm" flag isn't really needed for binpkgs but
it help reminder that zstd is the only one that does _not_ remove the
original file after compression

BINPKG_COMPRESS="zstd"
BINPKG_COMPRESS_FLAGS="--rm --long --threads=1 --adapt=min=7,max=19"

tails of the binpkgdir list:

Before 659M total
   3411658 Feb 13 16:03 media-fonts/corefonts-1-r7.tbz2
   3421845 May  5 19:09 dev-python/virtualenv-16.7.9-r1.tbz2
   3450323 May  3 08:54 dev-libs/nss-3.52.tbz2
   3475698 May  5 19:09 app-portage/gentoolkit-0.4.8.tbz2
   3545935 May  8 07:48 app-text/poppler-data-0.4.9.tbz2
   3582965 May 10 23:08 dev-python/cython-0.29.15.tbz2
   3870491 May  5 18:59 dev-libs/glib-2.62.6.tbz2
   3889030 May  9 07:49 sys-apps/groff-1.22.3.tbz2
   4225576 Apr 22 17:07 dev-libs/openssl-1.1.1g.tbz2
   4374231 May  5 20:21 sys-libs/db-5.3.28-r2.tbz2
   4889813 May  5 19:13 sys-devel/gdb-9.1.tbz2
   6418288 Apr  3 11:13 app-editors/vim-core-8.2.0360.tbz2
   7971823 May  5 18:49 dev-scheme/guile-2.2.4.tbz2
   8841520 Apr 30 10:08 sys-devel/binutils-2.33.1-r1.tbz2
   9076674 May  5 19:10 media-gfx/imagemagick-7.0.10.7-r1.tbz2
   9201769 May 11 13:25 net-print/cups-2.3.3-r1.tbz2
   9297311 May  1 08:13 dev-db/mysql-connector-c-8.0.20.tbz2
  12229701 Apr 30 17:10 dev-util/cmake-3.16.5.tbz2
  12977553 Apr  8 01:44 dev-libs/boost-1.72.0-r1.tbz2
  13479702 May  5 18:55 dev-libs/icu-65.1-r1.tbz2
  13788355 May 11 13:27 app-text/ghostscript-gpl-9.50.tbz2
  14028322 Feb 27 01:55 dev-lang/perl-5.30.1.tbz2
  14131663 May  5 19:11 net-dns/bind-tools-9.14.8.tbz2
  14664302 May  5 19:05 sys-libs/glibc-2.30-r8.tbz2
  15235721 Apr 22 17:09 dev-vcs/git-2.26.2.tbz2
  25024737 May  5 18:52 dev-lang/python-3.7.7-r2.tbz2
  26011846 May 10 23:09 app-admin/ansible-2.9.7.tbz2
  26213157 May  5 18:53 dev-lang/python-3.8.2-r2.tbz2
  34949529 May  5 19:15 dev-lang/php-7.3.17.tbz2
  35364894 May  5 18:59 dev-lang/php-7.4.5.tbz2
  48130470 May  5 19:09 dev-db/mariadb-10.4.12.tbz2
123472207 Apr 18 19:32 sys-devel/gcc-9.3.0.tbz2

After 540M total
   3000375 May 12 00:44 media-fonts/corefonts-1-r7.tbz2
   3010790 May 12 00:53 net-analyzer/net-snmp-5.8-r5.tbz2
   3130214 May 12 00:38 dev-libs/nss-3.52.tbz2
   3221436 May 12 00:23 media-fonts/urw-fonts-2.4.9.tbz2
   3386623 May 12 01:04 app-portage/gentoolkit-0.4.8.tbz2
   3541169 May 12 00:33 sys-apps/groff-1.22.3.tbz2
   3675561 May 12 01:06 dev-libs/glib-2.62.6.tbz2
   4305825 May 12 00:49 dev-libs/openssl-1.1.1g.tbz2
   4828074 May 12 00:49 sys-devel/gdb-9.1.tbz2
   6392717 May 12 01:04 app-editors/vim-core-8.2.0360.tbz2
   6883802 May 12 00:54 net-dns/bind-tools-9.14.8.tbz2
   7704310 May 12 00:16 sys-devel/binutils-2.33.1-r1.tbz2
   8044941 May 12 00:43 dev-scheme/guile-2.2.4.tbz2
   8442280 May 12 00:57 dev-vcs/git-2.26.2.tbz2
   8945372 May 12 02:22 media-gfx/imagemagick-7.0.10.7-r1.tbz2
   8989043 May 12 02:05 net-print/cups-2.3.3-r1.tbz2
  10140244 May 12 02:08 dev-util/cmake-3.16.5.tbz2
  12680888 May 12 02:03 sys-libs/glibc-2.30-r8.tbz2
  12991659 May 12 00:56 dev-libs/icu-65.1-r1.tbz2
  13634394 May 12 02:10 dev-libs/boost-1.72.0-r1.tbz2
  13861694 May 12 00:27 dev-lang/perl-5.30.1.tbz2
  13905742 May 12 02:12 app-text/ghostscript-gpl-9.50.tbz2
  16061883 May 12 01:03 app-admin/ansible-2.9.7.tbz2
  18244923 May 12 00:50 dev-lang/python-3.7.7-r2.tbz2
  19061497 May 12 02:15 dev-db/mariadb-10.4.12.tbz2
  19284388 May 12 00:52 dev-lang/python-3.8.2-r2.tbz2
  30342169 May 12 02:22 dev-lang/php-7.3.17.tbz2
  30767719 May 12 02:21 dev-lang/php-7.4.5.tbz2
100930187 May 12 02:01 sys-devel/gcc-9.3.0.tbz2


Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Default BINPKG_COMPRESSION to zstd (bug 715108)

Zac Medico-2
In reply to this post by Brian Dolbec-3
On 5/11/20 1:21 PM, Brian Dolbec wrote:

>> diff --git a/man/make.conf.5 b/man/make.conf.5
>> index f82fed65a..a3bd662ae 100644
>> --- a/man/make.conf.5
>> +++ b/man/make.conf.5
>> @@ -1,4 +1,4 @@
>> -.TH "MAKE.CONF" "5" "Nov 2019" "Portage VERSION" "Portage"
>> +.TH "MAKE.CONF" "5" "May 2020" "Portage VERSION" "Portage"
>>  .SH "NAME"
>>  make.conf \- custom settings for Portage
>>  .SH "SYNOPSIS"
>> @@ -115,7 +115,7 @@ This variable is used to determine the
>> compression used for \fIbinary packages\fR. Supported settings and
>> compression algorithms are: bzip2, gzip, lz4, lzip, lzop, xz, zstd.
>>  .br
>> -Defaults to "bzip2".
>> +Defaults to "zstd".
>>  .br
>>  .I Example:
>>  .nf
> looks good, I've not tested it, but changes are minor
>
I've merged this and released it in portage-2.3.100:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=230595cf600cae6beb6ebf6f817d08ace433c3ea
--
Thanks,
Zac


signature.asc (1000 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Default BINPKG_COMPRESSION to zstd (bug 715108)

Zac Medico-2
In reply to this post by Francesco Riosa-3
On 5/12/20 1:28 AM, Francesco Riosa wrote:

>
> Il 11/05/20 22:21, Brian Dolbec ha scritto:
>> On Sun, 10 May 2020 19:29:34 -0700
>> Zac Medico <[hidden email]> wrote:
>>
>>> This includes a _compat_upgrade.binpkg_compression script that the
>>> ebuild can call in pkg_preinst in order to maintain a
>>> backward-compatible bzip2 default when appropriate, ensuring that
>>> binary package consumers are not caught off guard.
>
> [snip]
>
> For your interest, the binpkg archive for a small LAMP container goes
> from 659 to 540 MB, the "--rm" flag isn't really needed for binpkgs but
> it help reminder that zstd is the only one that does _not_ remove the
> original file after compression
>
> BINPKG_COMPRESS="zstd"
> BINPKG_COMPRESS_FLAGS="--rm --long --threads=1 --adapt=min=7,max=19"
I'm using those flag now. Thanks!
--
Thanks,
Zac


signature.asc (1000 bytes) Download Attachment