[PATCH] per package environment: generalize the mechanism to be profile specific

classic Classic list List threaded Threaded
48 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

Bertrand Simonnet-2
Michal, Zac, did you have time to look at this serie of patches ?

Bertrand

On Tue, Sep 30, 2014 at 3:16 PM, Bertrand Simonnet <[hidden email]> wrote:
Please discard the last serie of patches. git send-email was not configured correctly and some commands  I typed yesterday are being send now.
Sorry about that.

Bertrand

On Mon, Sep 29, 2014 at 3:31 PM, Bertrand SIMONNET <[hidden email]> wrote:
Profiles can define per-package bashrc files to be sourced before emerging.
Each line in package.bashrc must be an atom name then a list of space-delimited
bashrc files (stored in $profile/bashrc/).
---
 bin/ebuild.sh                                      |  6 ++--
 bin/phase-functions.sh                             |  2 +-
 bin/save-ebuild-env.sh                             |  2 +-
 man/portage.5                                      | 30 +++++++++++++++++-
 .../package/ebuild/_config/special_env_vars.py     |  4 +--
 pym/portage/package/ebuild/config.py               | 36 ++++++++++++++++++++++
 pym/portage/package/ebuild/doebuild.py             |  4 ++-
 pym/portage/repository/config.py                   |  2 +-
 8 files changed, 76 insertions(+), 10 deletions(-)

diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index 14cc321..50909e1 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -368,10 +368,10 @@ __source_all_bashrcs() {
                # source the existing profile.bashrcs.
                save_IFS
                IFS=$'\n'
-               local path_array=($PROFILE_PATHS)
+               local bashenv_files=($PORTAGE_BASHRC_FILES)
                restore_IFS
-               for x in "${path_array[@]}" ; do
-                       [ -f "$x/profile.bashrc" ] && __qa_source "$x/profile.bashrc"
+               for x in "${bashenv_files[@]}" ; do
+                       __try_source "${x}"
                done
        fi

diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index f39a024..5dff3bb 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -31,7 +31,7 @@ PORTAGE_READONLY_VARS="D EBUILD EBUILD_PHASE EBUILD_PHASE_FUNC \
        PORTAGE_TMPDIR PORTAGE_UPDATE_ENV PORTAGE_USERNAME \
        PORTAGE_VERBOSE PORTAGE_WORKDIR_MODE PORTAGE_XATTR_EXCLUDE \
        PORTDIR \
-       PROFILE_PATHS REPLACING_VERSIONS REPLACED_BY_VERSION T WORKDIR \
+       REPLACING_VERSIONS REPLACED_BY_VERSION T WORKDIR \
        __PORTAGE_HELPER __PORTAGE_TEST_HARDLINK_LOCKS"

 PORTAGE_SAVED_READONLY_VARS="A CATEGORY P PF PN PR PV PVR"
diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
index f114c48..1a684b9 100644
--- a/bin/save-ebuild-env.sh
+++ b/bin/save-ebuild-env.sh
@@ -97,7 +97,7 @@ __save_ebuild_env() {
                GOOD HILITE HOME \
                LAST_E_CMD LAST_E_LEN LD_PRELOAD MISC_FUNCTIONS_ARGS MOPREFIX \
                NOCOLOR NORMAL PKGDIR PKGUSE PKG_LOGDIR PKG_TMPDIR \
-               PORTAGE_BASHRCS_SOURCED PORTAGE_COMPRESS \
+               PORTAGE_BASHRC_FILES PORTAGE_BASHRCS_SOURCED PORTAGE_COMPRESS \
                PORTAGE_COMPRESS_EXCLUDE_SUFFIXES \
                PORTAGE_DOHTML_UNWARNED_SKIPPED_EXTENSIONS \
                PORTAGE_DOHTML_UNWARNED_SKIPPED_FILES \
diff --git a/man/portage.5 b/man/portage.5
index e399f0f..309e259 100644
--- a/man/portage.5
+++ b/man/portage.5
@@ -24,6 +24,7 @@ make.defaults
 packages
 packages.build
 package.accept_keywords
+package.bashrc
 package.keywords
 package.mask
 package.provided
@@ -358,6 +359,31 @@ a '\-'.
 A list of packages (one per line) that make up a stage1 tarball.  Really only
 useful for stage builders.
 .TP
+.BR package.bashrc
+Per-package bashrc mechanism.  Contains a list of bashrc files to be sourced
+before emerging a given atom.  The bashrc files must be stored in bashrc/, in
+the profile directory.
+
+.I Note:
+.nf
+\- The bashrc files will be sourced after profile.bashrc for the same profile.
+\- profile-formats in metadata/layout.conf must contain profile-bashrcs for this
+to be enabled.
+.fi
+
+.I Format:
+.nf
+\- comments begin with # (no inline comments).
+\- one atom per line with space-delimited list of bashrc files.
+.fi
+
+.I Example:
+.nf
+# By setting INSTALL_MASK in bashrc/nostandardconf.conf, we can avoid installing
+# the standard configuration and enable another package to install it.
+net-misc/dhcp nostardardconf.conf
+.fi
+.TP
 .BR package.provided
 A list of packages (one per line) that portage should assume have been
 provided.  Useful for porting to non-Linux systems. Basically, it's a
@@ -1047,11 +1073,13 @@ The default setting for repoman's --echangelog option.
 The cache formats supported in the metadata tree.  There is the old "pms" format
 and the newer/faster "md5-dict" format.  Default is to detect dirs.
 .TP
-.BR profile\-formats " = [pms|portage-1|portage-2]"
+.BR profile\-formats " = [pms|portage-1|portage-2|profile-bashrcs]"
 Control functionality available to profiles in this repo such as which files
 may be dirs, or the syntax available in parent files.  Use "portage-2" if you're
 unsure.  The default is "portage-1-compat" mode which is meant to be compatible
 with old profiles, but is not allowed to be opted into directly.
+Setting profile-bashrcs will enable the per-profile bashrc mechanism
+\fBpackage.bashrc\fR.
 .RE
 .RE

diff --git a/pym/portage/package/ebuild/_config/special_env_vars.py b/pym/portage/package/ebuild/_config/special_env_vars.py
index 74fedd6..387f4ae 100644
--- a/pym/portage/package/ebuild/_config/special_env_vars.py
+++ b/pym/portage/package/ebuild/_config/special_env_vars.py
@@ -49,7 +49,7 @@ environ_whitelist += [
        "FEATURES", "FILESDIR", "HOME", "MERGE_TYPE", "NOCOLOR", "PATH",
        "PKGDIR",
        "PKGUSE", "PKG_LOGDIR", "PKG_TMPDIR",
-       "PORTAGE_ACTUAL_DISTDIR", "PORTAGE_ARCHLIST",
+       "PORTAGE_ACTUAL_DISTDIR", "PORTAGE_ARCHLIST", "PORTAGE_BASHRC_FILES",
        "PORTAGE_BASHRC", "PM_EBUILD_HOOK_DIR",
        "PORTAGE_BINPKG_FILE", "PORTAGE_BINPKG_TAR_OPTS",
        "PORTAGE_BINPKG_TMPFILE",
@@ -74,7 +74,7 @@ environ_whitelist += [
        "PORTAGE_SIGPIPE_STATUS",
        "PORTAGE_TMPDIR", "PORTAGE_UPDATE_ENV", "PORTAGE_USERNAME",
        "PORTAGE_VERBOSE", "PORTAGE_WORKDIR_MODE", "PORTAGE_XATTR_EXCLUDE",
-       "PORTDIR", "PORTDIR_OVERLAY", "PREROOTPATH", "PROFILE_PATHS",
+       "PORTDIR", "PORTDIR_OVERLAY", "PREROOTPATH",
        "REPLACING_VERSIONS", "REPLACED_BY_VERSION",
        "ROOT", "ROOTPATH", "T", "TMP", "TMPDIR",
        "USE_EXPAND", "USE_ORDER", "WORKDIR",
diff --git a/pym/portage/package/ebuild/config.py b/pym/portage/package/ebuild/config.py
index f639e14..183627f 100644
--- a/pym/portage/package/ebuild/config.py
+++ b/pym/portage/package/ebuild/config.py
@@ -316,6 +316,7 @@ class config(object):
                        self._accept_restrict = copy.deepcopy(clone._accept_restrict)
                        self._paccept_restrict = copy.deepcopy(clone._paccept_restrict)
                        self._penvdict = copy.deepcopy(clone._penvdict)
+                       self._pbashrcdict = copy.deepcopy(clone._pbashrcdict)
                        self._expand_map = copy.deepcopy(clone._expand_map)

                else:
@@ -661,6 +662,7 @@ class config(object):
                        self._ppropertiesdict = portage.dep.ExtendedAtomDict(dict)
                        self._paccept_restrict = portage.dep.ExtendedAtomDict(dict)
                        self._penvdict = portage.dep.ExtendedAtomDict(dict)
+                       self._pbashrcdict = {}

                        self._repo_make_defaults = {}
                        for repo in self.repositories.repos_with_profiles():
@@ -742,6 +744,25 @@ class config(object):
                                for k, v in penvdict.items():
                                        self._penvdict.setdefault(k.cp, {})[k] = v

+                               # package.bashrc
+                               for profile in profiles_complex:
+                                       if not 'profile-bashrcs' in profile.profile_formats:
+                                               continue
+                                       self._pbashrcdict[profile] = \
+                                               portage.dep.ExtendedAtomDict(dict)
+                                       bashrc = grabdict_package(os.path.join(profile.location,
+                                               "package.bashrc"), recursive=1, allow_wildcard=True,
+                                                               allow_repo=True, verify_eapi=False)
+                                       if not bashrc:
+                                               continue
+
+                                       for k, v in bashrc.items():
+                                               envfiles = [os.path.join(profile.location,
+                                                       "bashrc",
+                                                       envname) for envname in v]
+                                               self._pbashrcdict[profile].setdefault(k.cp, {})\
+                                                       .setdefault(k, []).extend(envfiles)
+
                        #getting categories from an external file now
                        self.categories = [grabfile(os.path.join(x, "categories")) \
                                for x in locations_manager.profile_and_user_locations]
@@ -1501,6 +1522,21 @@ class config(object):
                                for x in penv_matches:
                                        self._penv.extend(x)

+               bashrc_files = []
+
+               for profile in self._locations_manager.profiles_complex:
+                       bashrc_files.append(os.path.join(profile.location,
+                               'profile.bashrc'))
+                       if profile in self._pbashrcdict:
+                               cpdict = self._pbashrcdict[profile].get(cp)
+                               if cpdict:
+                                       bashrc_matches = \
+                                               ordered_by_atom_specificity(cpdict, cpv_slot)
+                                       for x in bashrc_matches:
+                                               bashrc_files.extend(x)
+
+               self.configdict["BASHRC_FILES"] = bashrc_files
+
                protected_pkg_keys = set(pkg_configdict)
                protected_pkg_keys.discard('USE')

diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py
index 01707ae..5224775 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -335,7 +335,9 @@ def doebuild_environment(myebuild, mydo, myroot=None, settings=None,
        mysettings["ECLASSDIR"]   = mysettings["PORTDIR"]+"/eclass"
        mysettings["SANDBOX_LOG"] = mycpv.replace("/", "_-_")

-       mysettings["PROFILE_PATHS"] = "\n".join(mysettings.profiles)
+       mysettings["PORTAGE_BASHRC_FILES"] = \
+               "\n".join(mysettings.configdict["BASHRC_FILES"])
+
        mysettings["P"]  = mysplit[0]+"-"+mysplit[1]
        mysettings["PN"] = mysplit[0]
        mysettings["PV"] = mysplit[1]
diff --git a/pym/portage/repository/config.py b/pym/portage/repository/config.py
index 5e0d055..bef643d 100644
--- a/pym/portage/repository/config.py
+++ b/pym/portage/repository/config.py
@@ -40,7 +40,7 @@ if sys.hexversion >= 0x3000000:
 _invalid_path_char_re = re.compile(r'[^a-zA-Z0-9._\-+:/]')

 _valid_profile_formats = frozenset(
-       ['pms', 'portage-1', 'portage-2'])
+       ['pms', 'portage-1', 'portage-2', 'profile-bashrcs'])

 _portage1_profiles_allow_directories = frozenset(
        ["portage-1-compat", "portage-1", 'portage-2'])
--
2.1.0.rc2.206.gedb03e5



Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

Zac Medico-2
On 10/06/2014 10:38 AM, Bertrand Simonnet wrote:
> Michal, Zac, did you have time to look at this series of patches ?

I've reviewed the whole series, and it all looks good to me. I think
it's ready to merge.
--
Thanks,
Zac

Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

Brian Dolbec-3
On Mon, 06 Oct 2014 11:57:45 -0700
Zac Medico <[hidden email]> wrote:

> On 10/06/2014 10:38 AM, Bertrand Simonnet wrote:
> > Michal, Zac, did you have time to look at this series of patches ?
>
> I've reviewed the whole series, and it all looks good to me. I think
> it's ready to merge.

We will be having a team meeting soon to decide recent patches, etc..

Although I think for the 2.2.14 final release we will not include this,
just a few bug fixes.  After 2.2.14 final is released we will likely
merge it so it will be in 9999 for awhile and then the 2.2.15 release.

--
Brian Dolbec <dolsen>


Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

Bertrand Simonnet
It sounds good.
If you think this will not change dramatically between now and the merge, I'll cherry-pick this on our portage version while waiting for the 2.2.15.

Let me know if something goes bad before the merge.

Thanks for the review :)
Bertrand

On Mon, Oct 6, 2014 at 2:01 PM, Brian Dolbec <[hidden email]> wrote:
On Mon, 06 Oct 2014 11:57:45 -0700
Zac Medico <[hidden email]> wrote:

> On 10/06/2014 10:38 AM, Bertrand Simonnet wrote:
> > Michal, Zac, did you have time to look at this series of patches ?
>
> I've reviewed the whole series, and it all looks good to me. I think
> it's ready to merge.

We will be having a team meeting soon to decide recent patches, etc..

Although I think for the 2.2.14 final release we will not include this,
just a few bug fixes.  After 2.2.14 final is released we will likely
merge it so it will be in 9999 for awhile and then the 2.2.15 release.

--
Brian Dolbec <dolsen>



Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

Zac Medico-2
In reply to this post by Brian Dolbec-3
On 10/06/2014 02:01 PM, Brian Dolbec wrote:

> On Mon, 06 Oct 2014 11:57:45 -0700
> Zac Medico <[hidden email]> wrote:
>
>> On 10/06/2014 10:38 AM, Bertrand Simonnet wrote:
>>> Michal, Zac, did you have time to look at this series of patches ?
>>
>> I've reviewed the whole series, and it all looks good to me. I think
>> it's ready to merge.
>
> We will be having a team meeting soon to decide recent patches, etc..
>
> Although I think for the 2.2.14 final release we will not include this,
> just a few bug fixes.  After 2.2.14 final is released we will likely
> merge it so it will be in 9999 for awhile and then the 2.2.15 release.
>

Should we go ahead and merge this now that 2.2.14 is released?
--
Thanks,
Zac

Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

Alexander Berntsen-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 24/10/14 04:00, Zac Medico wrote:
> Should we go ahead and merge this now that 2.2.14 is released?
ACK from me.

- --
Alexander
[hidden email]
https://secure.plaimi.net/~alexander
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlRJ/v0ACgkQRtClrXBQc7XcPAEAqGn2JmtE1NeSlcbYg8xWaRiq
6bnlOF3C+V3lhi6SNFwA/0M1xuuvQL2UtkVGwNev9gCfQjMeigwXXdhMHQSXGhHo
=SbHx
-----END PGP SIGNATURE-----

Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

Zac Medico-2
On 10/24/2014 12:25 AM, Alexander Berntsen wrote:
> On 24/10/14 04:00, Zac Medico wrote:
>> Should we go ahead and merge this now that 2.2.14 is released?
> ACK from me.

Thanks, I've pushed them to git now:

https://github.com/gentoo/portage/commit/60ee4deefb701d532fdd279caa989e7a6f4b8400
https://github.com/gentoo/portage/commit/1d351a59a57e018e9c79a371f0cae21505c2249c
https://github.com/gentoo/portage/commit/803dafc462027d6015721f40513abb5f57dc1178

In the last patch, I made 2 trivial changes:

1) Replace self.configdict["BASHRC_FILES"] with self._pbashrc, since configdict is used
for other unrelated things. Also make it an immutable tuple instead of a list.
2) Test if profile.bashrc exists before adding it to the list, which typically shortens
the list drastically because most profile directories don't have a profile.bashrc.

diff --git a/pym/portage/package/ebuild/config.py b/pym/portage/package/ebuild/config.py
index 33b4753..2ceb122 100644
--- a/pym/portage/package/ebuild/config.py
+++ b/pym/portage/package/ebuild/config.py
@@ -254,6 +254,7 @@ class config(object):
  self._iuse_implicit_match = clone._iuse_implicit_match
  self._non_user_variables = clone._non_user_variables
  self._env_d_blacklist = clone._env_d_blacklist
+ self._pbashrc = clone._pbashrc
  self._repo_make_defaults = clone._repo_make_defaults
  self.usemask = clone.usemask
  self.useforce = clone.useforce
@@ -663,6 +664,7 @@ class config(object):
  self._paccept_restrict = portage.dep.ExtendedAtomDict(dict)
  self._penvdict = portage.dep.ExtendedAtomDict(dict)
  self._pbashrcdict = {}
+ self._pbashrc = ()
 
  self._repo_make_defaults = {}
  for repo in self.repositories.repos_with_profiles():
@@ -1531,8 +1533,10 @@ class config(object):
  bashrc_files = []
 
  for profile in self._locations_manager.profiles_complex:
- bashrc_files.append(os.path.join(profile.location,
- 'profile.bashrc'))
+ profile_bashrc = os.path.join(profile.location,
+ 'profile.bashrc')
+ if os.path.exists(profile_bashrc):
+ bashrc_files.append(profile_bashrc)
  if profile in self._pbashrcdict:
  cpdict = self._pbashrcdict[profile].get(cp)
  if cpdict:
@@ -1541,7 +1545,7 @@ class config(object):
  for x in bashrc_matches:
  bashrc_files.extend(x)
 
- self.configdict["BASHRC_FILES"] = bashrc_files
+ self._pbashrc = tuple(bashrc_files)
 
  protected_pkg_keys = set(pkg_configdict)
  protected_pkg_keys.discard('USE')
diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py
index 20b179e..544d193 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -337,8 +337,7 @@ def doebuild_environment(myebuild, mydo, myroot=None, settings=None,
  mysettings["ECLASSDIR"]   = mysettings["PORTDIR"]+"/eclass"
  mysettings["SANDBOX_LOG"] = mycpv.replace("/", "_-_")
 
- mysettings["PORTAGE_BASHRC_FILES"] = \
- "\n".join(mysettings.configdict["BASHRC_FILES"])
+ mysettings["PORTAGE_BASHRC_FILES"] = "\n".join(mysettings._pbashrc)
 
  mysettings["P"]  = mysplit[0]+"-"+mysplit[1]
  mysettings["PN"] = mysplit[0]

--
Thanks,
Zac

Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

Alexander Berntsen-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 24/10/14 22:57, Zac Medico wrote:
> 1) Replace self.configdict["BASHRC_FILES"] with self._pbashrc,
> since configdict is used for other unrelated things. Also make it
> an immutable tuple instead of a list. 2) Test if profile.bashrc
> exists before adding it to the list, which typically shortens the
> list drastically because most profile directories don't have a
> profile.bashrc.
Great! Thanks for this. And thank you, Bertrand! And Micha?.

- --
Alexander
[hidden email]
https://secure.plaimi.net/~alexander
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlRK1ZQACgkQRtClrXBQc7WeEAEArOF6Oy4PiVZmuox9iXL7Iv4j
orPWnIhM6Xpgk4EXTHMA/jBbJ4BdkM4FMVcTs3AGIUbMCJfLSnj1GnlcQPSnTu7I
=nvms
-----END PGP SIGNATURE-----

123