[RFC] making the tree depend on portage

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

[RFC] making the tree depend on portage

Marius Mauch
Ok, the subject might be confusing, so let me explain this a bit:

Whenever we want/need to make structural changes to the tree that are
going to break backwards compability we have a serious problem (see
GLEP 44 in case you don't know about it). To reduce the impact of that
problem I've got the idea to make the tree itself (so not any
particular ebuild or profile) DEPEND on a minimal portage version,
which the users would be forced to upgrade to (maybe with an override)
before they can do anything else (with the exception of --sync).
Manifest2 is one example for such a situation, another one is the
request to not create manifest entries for ChangeLog and metadata.xml
anymore (needs >=2.0.51.20 on user side).
Don't really like this idea myself, but somthing needs to be done to at
least reduce the problem, having to wait years for old portage versions
to (almost) vanish can't be a permanent solution.

Also not talking about implementation details yet, just after comments
about the general idea of forced portage updates.

And just in case anybody wonders: this cannot be fixed with EAPI or
adding a portage dep on packages as those only take effect when the
ebuild is already parsed while the mentioned problems occur much
earlier.

Marius

--
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] making the tree depend on portage

Patrick Lauer
On Mon, 2005-12-19 at 17:49 +0100, Marius Mauch wrote:
> Ok, the subject might be confusing, so let me explain this a bit:
>
> Whenever we want/need to make structural changes to the tree that are
> going to break backwards compability we have a serious problem (see
> GLEP 44 in case you don't know about it). To reduce the impact of that
> problem I've got the idea to make the tree itself (so not any
> particular ebuild or profile) DEPEND on a minimal portage version,
Makes syncing a bit more complicated.
Downside: potentially more problems
Upside: if there's a disruptive change people will at all times have a
working system (just think about stacked profiles :-) )

> which the users would be forced to upgrade to (maybe with an override)
> before they can do anything else (with the exception of --sync).
Or you have a "version file" transferred by rsync (like timestamp)
When version > portage version: fetch a static portage tree from a known
location (mirror://portage-update-version-N.tbz2 or something)
(emerge-delta-webrsync to the rescue!)
> Manifest2 is one example for such a situation, another one is the
> request to not create manifest entries for ChangeLog and metadata.xml
> anymore (needs >=2.0.51.20 on user side).
> Don't really like this idea myself, but somthing needs to be done to at
> least reduce the problem, having to wait years for old portage versions
> to (almost) vanish can't be a permanent solution.
I don't like it, but unless portage is smart enough to self-update before anything else ...

> Also not talking about implementation details yet, just after comments
> about the general idea of forced portage updates.
If documented properly and guaranteed to work for at least n months after a format change I like it
(But I guess iterative updates should work "forever" as long as all
files still exist)

> And just in case anybody wonders: this cannot be fixed with EAPI or
> adding a portage dep on packages as those only take effect when the
> ebuild is already parsed while the mentioned problems occur much
> earlier.
Thanks for bringing it up for discussion!

Patrick
--
Stand still, and let the rest of the universe move

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] making the tree depend on portage

Ciaran McCreesh
In reply to this post by Marius Mauch
On Mon, 19 Dec 2005 17:49:02 +0100 Marius Mauch <[hidden email]>
wrote:
| And just in case anybody wonders: this cannot be fixed with EAPI or
| adding a portage dep on packages as those only take effect when the
| ebuild is already parsed while the mentioned problems occur much
| earlier.

Is it too late to switch to .ebuild.2, .ebuild.3 etc instead of making
EAPI an internal variable?

--
Ciaran McCreesh : Gentoo Developer (I can kill you with my brain)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] making the tree depend on portage

Brian Harring
On Mon, Dec 19, 2005 at 05:24:06PM +0000, Ciaran McCreesh wrote:
> On Mon, 19 Dec 2005 17:49:02 +0100 Marius Mauch <[hidden email]>
> wrote:
> | And just in case anybody wonders: this cannot be fixed with EAPI or
> | adding a portage dep on packages as those only take effect when the
> | ebuild is already parsed while the mentioned problems occur much
> | earlier.
>
> Is it too late to switch to .ebuild.2, .ebuild.3 etc instead of making
> EAPI an internal variable?
Yes.

EAPI mechanism will work as long as the ebuild is still valid shell
(eclasses complicate this, but the upcoming eclass2 switch over is the
only issue).  EAPI isn't going to be 5 versions in 5 years, I'd expect
2-3 version a year.

Any time we extend the api of provided functions to the ebuild env, we
should be bumping EAPI so the ebuild is known to work.
~harring

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] making the tree depend on portage

Francesco Riosa-2
In reply to this post by Marius Mauch

> Whenever we want/need to make structural changes to the tree that are
> going to break backwards compability we have a serious problem
[...]

Just throwing random thoughts here, but

Would a rescue-portage help on this ?

What I'm meaning is a "binary" portage provided for each ARCH (that
install in /opt), it MUST be easily downloadable from every user.

May be portage in case of hard fail could do that automatically issueing
a message like:

* OOPS, could not include module X this mean portage system is utterly
broken.
* would you try to download a rescue portage?

The rescue-portage does not need to be a fully functional portage, it
need only to be able to install a newer portage and it's direct
dependancies.


Regards,
Francesco
--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: [RFC] making the tree depend on portage

Jason Stubbs
On Tuesday 20 December 2005 21:42, Francesco Riosa wrote:
> > Whenever we want/need to make structural changes to the tree that are
> > going to break backwards compability we have a serious problem
>
> [...]
>
> Just throwing random thoughts here, but
>
> Would a rescue-portage help on this ?

It's easy enough to install portage from sources (for the time being) that I'd
say it's not necessary at this stage. A HOWTO on the matter would be much
quicker and easier to maintain while it remains viable.

--
Jason Stubbs
--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: [RFC] making the tree depend on portage

Francesco Riosa-2
Jason Stubbs wrote:

> On Tuesday 20 December 2005 21:42, Francesco Riosa wrote:
>  
>>> Whenever we want/need to make structural changes to the tree that are
>>> going to break backwards compability we have a serious problem
>>>      
>> [...]
>>
>> Just throwing random thoughts here, but
>>
>> Would a rescue-portage help on this ?
>>    
>
> It's easy enough to install portage from sources (for the time being) that I'd
> say it's not necessary at this stage. A HOWTO on the matter would be much
> quicker and easier to maintain while it remains viable.
>
>  
(-: great :-)

--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: [RFC] making the tree depend on portage

Jason Stubbs
In reply to this post by Marius Mauch
On Tuesday 20 December 2005 01:49, Marius Mauch wrote:
> Also not talking about implementation details yet, just after comments
> about the general idea of forced portage updates.

I gave it a go anyway... ;)

# emerge -up kde-base/kde
Checking for mandatory system updates...  Done.

The following packages can't be merged until your system is updated:

   kde-base/kde

The following important packages are either not up to date or not installed:

   virtual/baselayout sys-apps/coreutils sys-apps/debianutils
   sys-apps/diffutils sys-apps/file sys-apps/findutils sys-apps/gawk
   sys-apps/grep sys-apps/groff sys-apps/kbd sys-apps/man sys-apps/man-pages
   sys-apps/net-tools >=sys-apps/portage-2.0.51.22 sys-apps/sed
   sys-apps/shadow sys-apps/texinfo sys-apps/which virtual/modutils
   virtual/pager sys-apps/busybox sys-apps/hdparm sys-apps/util-linux
   sys-apps/linux32 >=sys-apps/baselayout-1.9.4-r7

Please retry your command after updating your system.


These are the packages that I would merge, in order:

Calculating system dependencies ...done!

[ebuild  N    ] sys-apps/sandbox-1.2.17
[ebuild  N    ] sys-apps/sed-4.1.4  USE="-bootstrap -build -nls -static"
[ebuild  N    ] sys-apps/debianutils-2.15  USE="-build -static"
[ebuild  N    ] sys-apps/portage-2.1_pre1  USE="-build"
*** Portage will stop merging at this point and reload itself,
    recalculate dependencies, and complete the merge.
    You may avoid the remerging of packages by updating portage on its own.
[ebuild  N    ] sys-apps/help2man-1.35.1  USE="-nls"
[ebuild  N    ] sys-apps/coreutils-5.3.0-r2  USE="-acl -build -nls -static"
[ebuild  N    ] sys-apps/sysvinit-2.86-r3  USE="-bootstrap -build -ibm
-static"
[ebuild     U ] sys-libs/readline-5.1 [5.0-r2]
[ebuild  N    ] sys-apps/baselayout-1.12.0_pre11-r3  USE="unicode -bootstrap
-build -static"
[ebuild  N    ] sys-apps/diffutils-2.8.7-r1  USE="-nls -static"
[ebuild  N    ] sys-apps/file-4.16  USE="python -build"
...
...


* Portage is moved to the top of both system and world
* All system atoms are checked that they are matched by installed packages
  and emerge refuses to do anything until unsatisfied atoms are installed
  and/or updated.
* EMERGE_INSANITY_OK is provided for bootstrap.sh (and emerge itself) to
  bypass the checks when the system is known to be in an incomplete state.

Reasoning on checking all system atoms is that other groups are just as likely
to need the functionality as we are. Combining that with how rarely versions
are actually updated for system packages, it shouldn't cause any more bother
to users than it needs to.

--
Jason Stubbs

Index: bin/emerge
===================================================================
--- bin/emerge (revision 2414)
+++ bin/emerge (working copy)
@@ -1418,6 +1418,14 @@
 
  mylist = sysdict.keys()
 
+ newlist = []
+ for atom in mylist:
+ if portage.dep_getkey(atom).split("/")[-1] == "portage":
+ newlist.insert(0, atom)
+ else:
+ newlist.append(atom)
+ mylist = newlist
+
  for mydep in mylist:
  try:
  if not self.select_dep(portage.root, mydep, raise_on_missing=True):
@@ -2501,6 +2509,73 @@
 if "--debug" in myopts:
  edebug=1
 
+
+def check_and_update_system():
+ if os.environ.get("EMERGE_INSANITY_OK") or myaction == "system":
+ return
+ rsync_timestamp = open(portage.settings["PORTDIR"]+"/metadata/timestamp").read().strip()
+ last_check = portage.mtimedb.get("last_system_check", "")
+ if rsync_timestamp == last_check:
+ return
+ print "Checking for mandatory system updates... ",
+ system_atoms = getlist("system")
+ vdb = portage.db["/"]["vartree"].dbapi  # This check only applies to /
+ unsatisfied_atoms = [atom for atom in system_atoms if not vdb.match(atom)]
+ print "Done."
+ if unsatisfied_atoms:
+ if myfiles:
+ allowed_keys = [portage.dep_getkey(atom) for atom in system_atoms]
+ virtuals = portage.settings.virtuals
+ for key in allowed_keys:
+ if key in virtuals:
+ allowed_keys.extend(virtuals[key])
+ allowed_files = []
+ for myfile in myfiles:
+ key = portage.dep_getkey(myfile)
+ if key in allowed_keys:
+ allowed_files.append(myfile)
+ continue
+ if "/" in myfile:
+ continue
+ for category in portage.settings.categories:
+ if category+"/"+key in allowed_keys:
+ allowed_files.append(myfile)
+ break
+ disallowed_files = [myfile for myfile in myfiles if myfile not in allowed_files]
+ if disallowed_files:
+ print red("\nThe following packages can't be merged until your system is updated:")
+ width = 80
+ for myfile in disallowed_files:
+ width += len(myfile) + 1
+ if width >= 76:
+ width = 2
+ print "\n  ",
+ print myfile,
+ myfiles.remove(myfile)
+ print
+ if myfiles:
+ return
+ print red("\nThe following important packages are either not up to date or not installed:")
+ width = 80
+ for atom in unsatisfied_atoms:
+ width += len(atom) + 1
+ if width >= 76:
+ width = 2
+ print "\n  ",
+ print atom,
+ print yellow("\n\nPlease retry your command after updating your system.")
+ print
+ portage.portageexit()
+ os.environ["EMERGE_INSANITY_OK"] = "1"
+ params = ["emerge", "system"]
+ params.append("--ask" in myopts and "--ask" or "--pretend")
+ os.execv("/usr/lib/portage/bin/emerge", params)
+ else:
+ portage.mtimedb["last_system_check"] = rsync_timestamp
+
+check_and_update_system()
+
+
 if myaction in ["sync","rsync","metadata"] and (not "--help" in myopts):
  if "--pretend" in myopts:
  print "emerge: \"sync\" actions do not support \"--pretend.\""
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] making the tree depend on portage

Jason Stubbs
On Tuesday 20 December 2005 23:15, Jason Stubbs wrote:
> On Tuesday 20 December 2005 01:49, Marius Mauch wrote:
> > Also not talking about implementation details yet, just after comments
> > about the general idea of forced portage updates.
>
> I gave it a go anyway... ;)

Also needed:

Index: portage.py
===================================================================
--- portage.py  (revision 2414)
+++ portage.py  (working copy)
@@ -6742,7 +6742,7 @@
 mtimedbkeys=[
 "updates", "info",
 "version", "starttime",
-"resume", "ldpath"
+"resume", "ldpath", "last_system_check"
 ]
 mtimedbfile=root+"var/cache/edb/mtimedb"
 try:

--
Jason Stubbs
--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: [RFC] making the tree depend on portage

Marius Mauch
In reply to this post by Jason Stubbs
Jason Stubbs wrote:
> Reasoning on checking all system atoms is that other groups are just as likely
> to need the functionality as we are. Combining that with how rarely versions
> are actually updated for system packages, it shouldn't cause any more bother
> to users than it needs to.

Well, that approach has one major problem: it won't help for profile
changes (like the mess with the cascaded profiles). Also I don't see
what "system" has to do with handling the tree itself, and for anything
else people can add dependencies, or am I missing something here?

Marius
--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: [RFC] making the tree depend on portage

Jason Stubbs
On Wednesday 21 December 2005 01:57, Marius Mauch wrote:
> Jason Stubbs wrote:
> > Reasoning on checking all system atoms is that other groups are just as
> > likely to need the functionality as we are. Combining that with how
> > rarely versions are actually updated for system packages, it shouldn't
> > cause any more bother to users than it needs to.
>
> Well, that approach has one major problem: it won't help for profile
> changes (like the mess with the cascaded profiles).

Making the tree DEPEND on a specific portage version won't help either. If the
user doesn't have a profile, they have no ARCH or other important things
which means they can't use emerge.

> Also I don't see what "system" has to do with handling the tree itself, and
> for anything else people can add dependencies, or am I missing something
> here?  

Doing it via system is more generic with no loss in functionality that I can
see. As I said above, the only time that not doing it in system would be
useful is if system itself is not available but profile problems imply that
one can't emerge anyway.

--
Jason Stubbs
--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: [RFC] making the tree depend on portage

Marius Mauch
On Wed, 21 Dec 2005 09:53:29 +0900
Jason Stubbs <[hidden email]> wrote:

> On Wednesday 21 December 2005 01:57, Marius Mauch wrote:
> > Jason Stubbs wrote:
> > > Reasoning on checking all system atoms is that other groups are
> > > just as likely to need the functionality as we are. Combining
> > > that with how rarely versions are actually updated for system
> > > packages, it shouldn't cause any more bother to users than it
> > > needs to.
> >
> > Well, that approach has one major problem: it won't help for profile
> > changes (like the mess with the cascaded profiles).
>
> Making the tree DEPEND on a specific portage version won't help
> either. If the user doesn't have a profile, they have no ARCH or
> other important things which means they can't use emerge.
There can be other potential changes that will not break portage
completely.

> > Also I don't see what "system" has to do with handling the tree
> > itself, and for anything else people can add dependencies, or am I
> > missing something here?  
>
> Doing it via system is more generic with no loss in functionality
> that I can see. As I said above, the only time that not doing it in
> system would be useful is if system itself is not available but
> profile problems imply that one can't emerge anyway.

It has other drawbacks though. First "system" is arbitrary, imagine a
"nterprise desktop" profile or so where sytem includes gnome/kde.
The main issue I have with it though is that it spreads the focus too
much, your example had what, 20 packages? What I'm after is a
information when we *know* that the tree won't work anymore, not a "it
may be better to upgrade".

Did I mention that I don't really like this idea in the first place?

Marius

--
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

signature.asc (196 bytes) Download Attachment