Versioning the tree

classic Classic list List threaded Threaded
58 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Versioning the tree

Steven J. Long
Hi,

  There was a discussion a few weeks back about stopping system b0rkage; a
possible sol'n had been previously discussed on the fora, ie having the
tree in svn for easier branching. I understand from the recent ANNOUNCE by
Robin Johnson that svn access is now available, as well as anonymous CVS.
Kudos for all that work, guys.

>> Is it correct that versioning the tree would solve it by allowing various
>> releases to stick to lower versions of packages until they have been QAed
>> by the gentoo community?
Stuart Herbert <stuart.herbert <at> gmail.com> wrote:
> Yes.
>
> The Gentoo package tree is a "live" tree - whatever we commit goes
> straight out to the rsync mirrors for users to download and use.
>
> Live trees are not compatible w/ a high quality product.

Patrick McLean <chutzpah <at> gentoo.org> wrote:
> This is the entire reason for ~arch, packages are all tested in ~ for at
> least 30 days before they are stabilized by arch teams.
Well that's clearly not working is it? Seems that the users are doing the QA
in the wild ATM, and often their systems are borking, even when running
stable (or often enough.)

In any event, what I'd like to raise is the issue of having a
(semi-)official version of gentoo that lags behind the cutting-edge distro
for stability. Is this feasible?

Apologies if this is already being discussed elsewhere.

Regards,
Steve.


--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: Versioning the tree

Steven J. Long
> In any event, what I'd like to raise is the issue of having a
> (semi-)official version of gentoo that lags behind the cutting-edge distro
> for stability. Is this feasible?
>
> Apologies if this is already being discussed elsewhere.
>
I appreciate that there is GLEP 19 according to earlier discussion on this
list (from 2004).

I guess I'm asking whether it's a) more feasible now (I'm guessing yes) and
b) whether it's something that would have any support from current devs.
Not in terms of workload, but culturally.

Sorry for not being clear straight off.

Steve.


--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: Re: Versioning the tree

paul kölle
Steve Long schrieb:
>> In any event, what I'd like to raise is the issue of having a
>> (semi-)official version of gentoo that lags behind the cutting-edge distro
>> for stability. Is this feasible?
>>
>> Apologies if this is already being discussed elsewhere.
>>
> I appreciate that there is GLEP 19 according to earlier discussion on this
> list (from 2004).
The GLEP has been withdrawn by the author.

>
> I guess I'm asking whether it's a) more feasible now (I'm guessing yes) and
> b) whether it's something that would have any support from current devs.
> Not in terms of workload, but culturally.
You can't take workload out of the picture since it's the main issue
here. Stable tree means backport fixes and I don't see this happening as
it can't be automated:
-there is no agreed machine parsable format/central location for CVEs/bugs.
-you can't rely on upstream issuing security patches not tied to new
releases (with new bugs/features).

So who is going to watch all that mailing lists, pulling code from
upstream, patching, testing, etc.?

Don't get me wrong. I'd really love to have a more "stable" base system
at least for core packages and those prone to break remote reboots ;)

cheers
 Paul
--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: Re: Versioning the tree

Stuart Herbert-2
On 11/27/06, paul <[hidden email]> wrote:
> You can't take workload out of the picture since it's the main issue
> here. Stable tree means backport fixes and I don't see this happening as
> it can't be automated:

"Stable tree means backport fixes" is an assumption, not a requirement.

It's one reason why the word "stable" is a poor choice for any such
tree, just as it is a poor choice for the current Portage tree.

I think the original poster hit the nail on the head.  The real
barrier preventing a slower-moving tree is cultural.

Best regards,
Stu
--
--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: Re: Versioning the tree

Kevin F. Quinn (Gentoo)
On Mon, 27 Nov 2006 13:02:17 +0000
"Stuart Herbert" <[hidden email]> wrote:

> On 11/27/06, paul <[hidden email]> wrote:
> > You can't take workload out of the picture since it's the main issue
> > here. Stable tree means backport fixes and I don't see this
> > happening as it can't be automated:
>
> "Stable tree means backport fixes" is an assumption, not a
> requirement.
>
> It's one reason why the word "stable" is a poor choice for any such
> tree, just as it is a poor choice for the current Portage tree.
>
> I think the original poster hit the nail on the head.  The real
> barrier preventing a slower-moving tree is cultural.
One method could be to snapshot all package versions at the time that
Release Engineering make a release, building a package.mask file out of
it masking out all packages of higher revisions (i.e. having '>CPVR'
entry for every package in the tree in stable, and 'CPVR' if no
versions are stable).

Then, rather than back-port security fixes, this list would be updated
with security-fixed version numbers, along with minimum required updates
to dependent packages. The lists could be stored in the tree; for
example as profiles/default-linux/x86/stable.mask.

Obviously maintaining that list is some work; predominantly watching
the announcements from the security team and fixing up dependencies,
and masking out new packages (for what that's worth).  It could be
regenerated on some or all releases, or perhaps on a yearly basis.  It
would also mean that versions listed there cannot be removed from the
tree (until they're bumped in the list).

Some of that maintenance could be tool-assisted; in particular masking
new packages and finding the minimum required bumps to support a
package that was bumped for a security fix.

People who want to use it could then just soft-link
from /etc/portage/package.mask to that list.

It's just a suggestion - I'm not prepared to do the work ;)  However it
might be a simple but effective method to help people maintain secure
but relatively stable systems, without having to upgrade umpteen
packages a week.

--
Kevin F. Quinn

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Versioning the tree

Marius Mauch
In reply to this post by Steven J. Long
On Mon, 27 Nov 2006 11:33:58 +0000
Steve Long <[hidden email]> wrote:

> > In any event, what I'd like to raise is the issue of having a
> > (semi-)official version of gentoo that lags behind the cutting-edge
> > distro for stability. Is this feasible?
> >
> > Apologies if this is already being discussed elsewhere.
> >
> I appreciate that there is GLEP 19 according to earlier discussion on
> this list (from 2004).
>
> I guess I'm asking whether it's a) more feasible now (I'm guessing
> yes) and b) whether it's something that would have any support from
> current devs. Not in terms of workload, but culturally.
>
> Sorry for not being clear straight off.
See http://forums.gentoo.org/viewtopic-p-3730878.html#3730878 for a
related discussion.

Marius

--
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Versioning the tree

Alec Warner-2
In reply to this post by Steven J. Long
Steve Long wrote:
> Hi,
>
>   There was a discussion a few weeks back about stopping system b0rkage; a
> possible sol'n had been previously discussed on the fora, ie having the
> tree in svn for easier branching. I understand from the recent ANNOUNCE by
> Robin Johnson that svn access is now available, as well as anonymous CVS.
> Kudos for all that work, guys.
>

Just a note, that the Tree itself is not in SVN; the SVN repositories
only hold ancillary Gentoo projects, such as portage and gentoolkit and
whatnot.  The tree is still stored in CVS.

--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: Re: Versioning the tree

Chris Gianelloni
In reply to this post by Steven J. Long
On Mon, 2006-11-27 at 11:33 +0000, Steve Long wrote:

> > In any event, what I'd like to raise is the issue of having a
> > (semi-)official version of gentoo that lags behind the cutting-edge distro
> > for stability. Is this feasible?
> >
> > Apologies if this is already being discussed elsewhere.
> >
> I appreciate that there is GLEP 19 according to earlier discussion on this
> list (from 2004).
>
> I guess I'm asking whether it's a) more feasible now (I'm guessing yes) and
It isn't.

> b) whether it's something that would have any support from current devs.

Probably not, considering even the people that were proponents of GLEP19
have dropped support for it.

Personally, I would prefer seeing my "release trees" idea take off.
Essentially, it freezes the tree at a certain point (which I just
coincide with our releases).  Updates are security-only.

Now, this doesn't mean that *everything* must remain this way.  For
example, there could be a 2007.0-r1 "tree" or something which is
2007.0's tree, with some major bug fixes.  The real question is how much
manpower would it take to maintain such a tree and how much use would
"normal" users get out of it, as opposed to "enterprise" users?

I'm thinking of releasing a tarball for a "release tree" next time
around, just to see how it flies.  The main question I have is whether
we'll have time.

--
Chris Gianelloni
Release Engineering Strategic Lead
Alpha/AMD64/x86 Architecture Teams
Games Developer/Council Member/Foundation Trustee
Gentoo Foundation

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Versioning the tree

Chris Gianelloni
In reply to this post by Stuart Herbert-2
On Mon, 2006-11-27 at 13:02 +0000, Stuart Herbert wrote:
> I think the original poster hit the nail on the head.  The real
> barrier preventing a slower-moving tree is cultural.

Somewhat.

As I have said, I've mentioned several times the idea of doing a
"release tree" to go along with each release.  Support on these trees
would be essentially equivalent to our security support model.  We would
support only packages marked stable.  Of course, we also understand that
since we're building off upstream's work, there *will* be times when a
version bump is necessary to reduce our workload.  Basically, our rule
is kinda like this.

No version changes on any packages, except those which are necessary due
to a security violation, or a vulnerable package's dependencies.

Now, I don't know if it is the best approach, but it is the best one
that I could come up with on my own that allows for a slow-moving tree
with higher QA done on it (as the release trees are what we use for
releases, they undergo an enormous amount of testing, in the default
configuration, of course) and minimal changes.

Something that would be useful would be for package maintainers that
wish to participate to perform further testing and validation on
packages in the release snapshot prior to its final freeze.  This would
give us even more eyes on the tree and hopefully provide a much higher
quality snapshot once we're done.

Since it seems that there is still interest in the idea, I'll start
working on it some more.

--
Chris Gianelloni
Release Engineering Strategic Lead
Alpha/AMD64/x86 Architecture Teams
Games Developer/Council Member/Foundation Trustee
Gentoo Foundation

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Versioning the tree

Chris Gianelloni
In reply to this post by Kevin F. Quinn (Gentoo)
On Mon, 2006-11-27 at 16:18 +0100, Kevin F. Quinn wrote:
> One method could be to snapshot all package versions at the time that
> Release Engineering make a release, building a package.mask file out of
> it masking out all packages of higher revisions (i.e. having '>CPVR'
> entry for every package in the tree in stable, and 'CPVR' if no
> versions are stable).

It would be infinitely easier to create a tree just for this.

> Then, rather than back-port security fixes, this list would be updated
> with security-fixed version numbers, along with minimum required updates
> to dependent packages. The lists could be stored in the tree; for
> example as profiles/default-linux/x86/stable.mask.

This is essentially the idea that my release tree uses, except the tree
itself is completely stripped down to stable packages.  I have a script
which already does several things:

#1. grabs "best_visible" for stable on each arch
#2. repeat for each SLOT
#3. purge unnecessary files from FILESDIR
#4. strip to only "stable" profiles from profiles.desc
#5. purge unnecessary USE from use.local.desc and use.desc
#6. strip unused eclasses
#7. regenerate Manifest (via ebuild $foo digest)

I had also planned on it doing the following, but they haven't been
implemented just yet:

- strip all USE flags that aren't used from use.mask (per-profile)
- strip all packages that aren't available from package.mask
(per-profile)

What this gives us is a *much* smaller tree, as it only includes stable
packages/versions.

> Obviously maintaining that list is some work; predominantly watching
> the announcements from the security team and fixing up dependencies,
> and masking out new packages (for what that's worth).  It could be
> regenerated on some or all releases, or perhaps on a yearly basis.  It
> would also mean that versions listed there cannot be removed from the
> tree (until they're bumped in the list).

Well, the simpler approach was to simply copy over the newer, secure
ebuilds, and any required dependencies, into the release tree.  There'd
be no need to update mask files and such this way.  It would also be
generated with the releases, automatically.

> Some of that maintenance could be tool-assisted; in particular masking
> new packages and finding the minimum required bumps to support a
> package that was bumped for a security fix.
>
> People who want to use it could then just soft-link
> from /etc/portage/package.mask to that list.
>
> It's just a suggestion - I'm not prepared to do the work ;)  However it
> might be a simple but effective method to help people maintain secure
> but relatively stable systems, without having to upgrade umpteen
> packages a week.
Well, I am willing, and have been doing, some of the work.  The one
disadvantage to my design is it needs infra.  It needs it's own
repository and rsync.

Basically, it makes the system act a bit more like some other
development models.  We end up with the following:

rsync://rsync.gentoo.org/gentoo-portage (this is equivalent to -current)
rsync://rsync.gentoo.org/2007.0-release (this is the release tree)

Now, the release trees are non-moving.  The 2007.0-release tree is
*always* the 2007.0-release tree for as long as we decide to support it.
Likely, this support period would begin as a single release and get
extended as volunteers came around to support it.  New releases get
their own tree.

I also started writing a tool to "upgrade" the distribution.  The tool
reads pre and post scripts for the upgrade, and performs the necessary
steps.  Basically, a user can "dist-upgrade 2007.1" and the system
would:

- switch to the "2007.1" tree and "emerge --sync"
- perform all "pre" steps from 2007.1
- update world + revdep-rebuild + whatever else is necessary
- perform all "post" steps

With this, there would be a special "version" called "current" which
would put the user on the "gentoo-portage" tree, AKA the tree we all
know and love/loathe.  Release trees wouldn't have "arch" and "~arch" as
everything would be stable there.  Testing would be done in the main
tree.  This, in essence, gives us "testing", "release candidate" and
"stable" environments, with the release trees being stable, and the
current tree becoming test/release candidate.  Anything marked stable in
the current tree would be a candidate for stable in the next release
tree.

Users who want to use the current portage tree get what they want.
Users who want a more stable tree get what they want.  Basically,
everyone (hopefully) is happy, or at least as happy as we can feasibly
make them.

--
Chris Gianelloni
Release Engineering Strategic Lead
Alpha/AMD64/x86 Architecture Teams
Games Developer/Council Member/Foundation Trustee
Gentoo Foundation

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Versioning the tree

James Potts
On 11/28/06, Chris Gianelloni <[hidden email]> wrote:

> On Mon, 2006-11-27 at 16:18 +0100, Kevin F. Quinn wrote:
> > One method could be to snapshot all package versions at the time that
> > Release Engineering make a release, building a package.mask file out of
> > it masking out all packages of higher revisions (i.e. having '>CPVR'
> > entry for every package in the tree in stable, and 'CPVR' if no
> > versions are stable).
>
> It would be infinitely easier to create a tree just for this.
>
> > Then, rather than back-port security fixes, this list would be updated
> > with security-fixed version numbers, along with minimum required updates
> > to dependent packages. The lists could be stored in the tree; for
> > example as profiles/default-linux/x86/stable.mask.
>
> This is essentially the idea that my release tree uses, except the tree
> itself is completely stripped down to stable packages.  I have a script
> which already does several things:
>
> #1. grabs "best_visible" for stable on each arch
> #2. repeat for each SLOT
> #3. purge unnecessary files from FILESDIR
> #4. strip to only "stable" profiles from profiles.desc
> #5. purge unnecessary USE from use.local.desc and use.desc
> #6. strip unused eclasses
> #7. regenerate Manifest (via ebuild $foo digest)
>
> I had also planned on it doing the following, but they haven't been
> implemented just yet:
>
> - strip all USE flags that aren't used from use.mask (per-profile)
> - strip all packages that aren't available from package.mask
> (per-profile)
>
> What this gives us is a *much* smaller tree, as it only includes stable
> packages/versions.
>
> > Obviously maintaining that list is some work; predominantly watching
> > the announcements from the security team and fixing up dependencies,
> > and masking out new packages (for what that's worth).  It could be
> > regenerated on some or all releases, or perhaps on a yearly basis.  It
> > would also mean that versions listed there cannot be removed from the
> > tree (until they're bumped in the list).
>
> Well, the simpler approach was to simply copy over the newer, secure
> ebuilds, and any required dependencies, into the release tree.  There'd
> be no need to update mask files and such this way.  It would also be
> generated with the releases, automatically.
>
> > Some of that maintenance could be tool-assisted; in particular masking
> > new packages and finding the minimum required bumps to support a
> > package that was bumped for a security fix.
> >
> > People who want to use it could then just soft-link
> > from /etc/portage/package.mask to that list.
> >
> > It's just a suggestion - I'm not prepared to do the work ;)  However it
> > might be a simple but effective method to help people maintain secure
> > but relatively stable systems, without having to upgrade umpteen
> > packages a week.
>
> Well, I am willing, and have been doing, some of the work.  The one
> disadvantage to my design is it needs infra.  It needs it's own
> repository and rsync.
>
> Basically, it makes the system act a bit more like some other
> development models.  We end up with the following:
>
> rsync://rsync.gentoo.org/gentoo-portage (this is equivalent to -current)
> rsync://rsync.gentoo.org/2007.0-release (this is the release tree)
>
> Now, the release trees are non-moving.  The 2007.0-release tree is
> *always* the 2007.0-release tree for as long as we decide to support it.
> Likely, this support period would begin as a single release and get
> extended as volunteers came around to support it.  New releases get
> their own tree.
>
> I also started writing a tool to "upgrade" the distribution.  The tool
> reads pre and post scripts for the upgrade, and performs the necessary
> steps.  Basically, a user can "dist-upgrade 2007.1" and the system
> would:
>
> - switch to the "2007.1" tree and "emerge --sync"
> - perform all "pre" steps from 2007.1
> - update world + revdep-rebuild + whatever else is necessary
> - perform all "post" steps
>
> With this, there would be a special "version" called "current" which
> would put the user on the "gentoo-portage" tree, AKA the tree we all
> know and love/loathe.  Release trees wouldn't have "arch" and "~arch" as
> everything would be stable there.  Testing would be done in the main
> tree.  This, in essence, gives us "testing", "release candidate" and
> "stable" environments, with the release trees being stable, and the
> current tree becoming test/release candidate.  Anything marked stable in
> the current tree would be a candidate for stable in the next release
> tree.
>
> Users who want to use the current portage tree get what they want.
> Users who want a more stable tree get what they want.  Basically,
> everyone (hopefully) is happy, or at least as happy as we can feasibly
> make them.
>

This looks good on the surface, Chris, but what happens in the case
where somebody wants to use the Release tree, but also wants (or
needs) one or more packages from the Live tree, and doesn't want to
switch completely over to the live tree?  If I understand what you
want to do correctly, the Release tree would include only stable
packages.  Other packages wouldn't just be masked, they would be
completely unavailable to anybody using that tree.

I like the idea of having a stable p.mask much better, which says
"profile" to me.  Any thoughts on a special profile just for releases?

--Arek (James Potts)
[hidden email]
--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: Re: Versioning the tree

xenoterracide
they could pull the more current ebuilds and put them in an overlay.
also correct me if I'm wrong isn't it possible only to sync certain
parts of the tree using excludes. maybe some additional functionality
saying only sync package X for updates.

On 11/28/06, James Potts <[hidden email]> wrote:

> On 11/28/06, Chris Gianelloni <[hidden email]> wrote:
> > On Mon, 2006-11-27 at 16:18 +0100, Kevin F. Quinn wrote:
> > > One method could be to snapshot all package versions at the time that
> > > Release Engineering make a release, building a package.mask file out of
> > > it masking out all packages of higher revisions (i.e. having '>CPVR'
> > > entry for every package in the tree in stable, and 'CPVR' if no
> > > versions are stable).
> >
> > It would be infinitely easier to create a tree just for this.
> >
> > > Then, rather than back-port security fixes, this list would be updated
> > > with security-fixed version numbers, along with minimum required updates
> > > to dependent packages. The lists could be stored in the tree; for
> > > example as profiles/default-linux/x86/stable.mask.
> >
> > This is essentially the idea that my release tree uses, except the tree
> > itself is completely stripped down to stable packages.  I have a script
> > which already does several things:
> >
> > #1. grabs "best_visible" for stable on each arch
> > #2. repeat for each SLOT
> > #3. purge unnecessary files from FILESDIR
> > #4. strip to only "stable" profiles from profiles.desc
> > #5. purge unnecessary USE from use.local.desc and use.desc
> > #6. strip unused eclasses
> > #7. regenerate Manifest (via ebuild $foo digest)
> >
> > I had also planned on it doing the following, but they haven't been
> > implemented just yet:
> >
> > - strip all USE flags that aren't used from use.mask (per-profile)
> > - strip all packages that aren't available from package.mask
> > (per-profile)
> >
> > What this gives us is a *much* smaller tree, as it only includes stable
> > packages/versions.
> >
> > > Obviously maintaining that list is some work; predominantly watching
> > > the announcements from the security team and fixing up dependencies,
> > > and masking out new packages (for what that's worth).  It could be
> > > regenerated on some or all releases, or perhaps on a yearly basis.  It
> > > would also mean that versions listed there cannot be removed from the
> > > tree (until they're bumped in the list).
> >
> > Well, the simpler approach was to simply copy over the newer, secure
> > ebuilds, and any required dependencies, into the release tree.  There'd
> > be no need to update mask files and such this way.  It would also be
> > generated with the releases, automatically.
> >
> > > Some of that maintenance could be tool-assisted; in particular masking
> > > new packages and finding the minimum required bumps to support a
> > > package that was bumped for a security fix.
> > >
> > > People who want to use it could then just soft-link
> > > from /etc/portage/package.mask to that list.
> > >
> > > It's just a suggestion - I'm not prepared to do the work ;)  However it
> > > might be a simple but effective method to help people maintain secure
> > > but relatively stable systems, without having to upgrade umpteen
> > > packages a week.
> >
> > Well, I am willing, and have been doing, some of the work.  The one
> > disadvantage to my design is it needs infra.  It needs it's own
> > repository and rsync.
> >
> > Basically, it makes the system act a bit more like some other
> > development models.  We end up with the following:
> >
> > rsync://rsync.gentoo.org/gentoo-portage (this is equivalent to -current)
> > rsync://rsync.gentoo.org/2007.0-release (this is the release tree)
> >
> > Now, the release trees are non-moving.  The 2007.0-release tree is
> > *always* the 2007.0-release tree for as long as we decide to support it.
> > Likely, this support period would begin as a single release and get
> > extended as volunteers came around to support it.  New releases get
> > their own tree.
> >
> > I also started writing a tool to "upgrade" the distribution.  The tool
> > reads pre and post scripts for the upgrade, and performs the necessary
> > steps.  Basically, a user can "dist-upgrade 2007.1" and the system
> > would:
> >
> > - switch to the "2007.1" tree and "emerge --sync"
> > - perform all "pre" steps from 2007.1
> > - update world + revdep-rebuild + whatever else is necessary
> > - perform all "post" steps
> >
> > With this, there would be a special "version" called "current" which
> > would put the user on the "gentoo-portage" tree, AKA the tree we all
> > know and love/loathe.  Release trees wouldn't have "arch" and "~arch" as
> > everything would be stable there.  Testing would be done in the main
> > tree.  This, in essence, gives us "testing", "release candidate" and
> > "stable" environments, with the release trees being stable, and the
> > current tree becoming test/release candidate.  Anything marked stable in
> > the current tree would be a candidate for stable in the next release
> > tree.
> >
> > Users who want to use the current portage tree get what they want.
> > Users who want a more stable tree get what they want.  Basically,
> > everyone (hopefully) is happy, or at least as happy as we can feasibly
> > make them.
> >
>
> This looks good on the surface, Chris, but what happens in the case
> where somebody wants to use the Release tree, but also wants (or
> needs) one or more packages from the Live tree, and doesn't want to
> switch completely over to the live tree?  If I understand what you
> want to do correctly, the Release tree would include only stable
> packages.  Other packages wouldn't just be masked, they would be
> completely unavailable to anybody using that tree.
>
> I like the idea of having a stable p.mask much better, which says
> "profile" to me.  Any thoughts on a special profile just for releases?
>
> --Arek (James Potts)
> [hidden email]
> --
> [hidden email] mailing list
>
>
--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: Re: Versioning the tree

Andrew Gaffney
In reply to this post by James Potts
James Potts wrote:
> This looks good on the surface, Chris, but what happens in the case
> where somebody wants to use the Release tree, but also wants (or
> needs) one or more packages from the Live tree, and doesn't want to
> switch completely over to the live tree?  If I understand what you
> want to do correctly, the Release tree would include only stable
> packages.  Other packages wouldn't just be masked, they would be
> completely unavailable to anybody using that tree.

This is what overlays are for :)

--
Andrew Gaffney                            http://dev.gentoo.org/~agaffney/
Gentoo Linux Developer                                   Installer Project
Today's lesson in political correctness:      "Go asphyxiate on a phallus"
--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: Re: Versioning the tree

Chris Gianelloni
In reply to this post by James Potts
On Tue, 2006-11-28 at 14:56 -0600, James Potts wrote:
> This looks good on the surface, Chris, but what happens in the case
> where somebody wants to use the Release tree, but also wants (or
> needs) one or more packages from the Live tree, and doesn't want to
> switch completely over to the live tree?  If I understand what you
> want to do correctly, the Release tree would include only stable
> packages.  Other packages wouldn't just be masked, they would be
> completely unavailable to anybody using that tree.

Correct.  The whole concept of the release tree is it is a complete
package.  All of the parts are supposed to work together.  There's
*nothing* that is not considered stable.  If you need other packages,
you add them to an overlay and support them yourself.

> I like the idea of having a stable p.mask much better, which says
> "profile" to me.  Any thoughts on a special profile just for releases?

It is completely unworkable with the 11,000+ packages we currently have
in the tree, whereas an automated script that parses the current tree is
much simpler.  Quite simply, when GLEP19 was being worked on, it became
painfully obvious almost immediately that even with the 10+ people who
volunteered to work on it, that it would be impossible to maintain a
profile-based system.  We estimated that it would take us more than
twice as many developers working on *only* the stable tree than we had
working on the "live" tree to even keep up.

To put it simply, I have *zero* interest in any profile/mask-based
concepts for providing "stable" as they don't work.  All they do is
create an enormous bottleneck and a massive amount of workload for every
single developer.  With the release trees, essentially only those
interested in supporting the tree are required to work on it.  The tree
is created entirely by scripts and is tested *before* it's released on
the public.

--
Chris Gianelloni
Release Engineering Strategic Lead
Alpha/AMD64/x86 Architecture Teams
Games Developer/Council Member/Foundation Trustee
Gentoo Foundation

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Versioning the tree

Stuart Herbert-2
In reply to this post by Chris Gianelloni
On 11/28/06, Chris Gianelloni <[hidden email]> wrote:
> As I have said, I've mentioned several times the idea of doing a
> "release tree" to go along with each release.

The release tree is not the basis for this.

a) Releases (and the releng work that goes into it) are exclusively
desktop-oriented.
b) Release trees have a nasty habit of picking up last minute changes
(such as gcc 4.1) to suit the release, not stability.

> No version changes on any packages, except those which are necessary due
> to a security violation, or a vulnerable package's dependencies.

Tying a minimal-b0rkage tree to the arbitrary schedule of our releases
does not serve all of our users.  We are back to the same arguments we
had when I said that the Seeds project would have to have its own
independent release schedules :(

Thereś little merit in us creating mostly stagnant trees.  Other Linux
distros are already very good at doing that - far better than we will
be at it - because they have advantages such as a paid workforce and
more upstream developers on their books.

A minimal-b0rkage tree needs to move to reflect the packages that we
believe our users should be using for a stable environment.

Best regards,
Stu

--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: Re: Versioning the tree

Stephen P. Becker
In reply to this post by James Potts
On Tue, 28 Nov 2006 14:56:47 -0600
"James Potts" <[hidden email]> wrote:

> This looks good on the surface, Chris, but what happens in the case
> where somebody wants to use the Release tree, but also wants (or
> needs) one or more packages from the Live tree, and doesn't want to
> switch completely over to the live tree?  If I understand what you
> want to do correctly, the Release tree would include only stable
> packages.  Other packages wouldn't just be masked, they would be
> completely unavailable to anybody using that tree.
>
> I like the idea of having a stable p.mask much better, which says
> "profile" to me.  Any thoughts on a special profile just for releases?
Sounds like a perfect situation to use a package manager with true
multiple repository support.

-Steve

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Versioning the tree

Andrew Gaffney
In reply to this post by Stuart Herbert-2
Stuart Herbert wrote:
> On 11/28/06, Chris Gianelloni <[hidden email]> wrote:
>> As I have said, I've mentioned several times the idea of doing a
>> "release tree" to go along with each release.
>
> The release tree is not the basis for this.
>
> a) Releases (and the releng work that goes into it) are exclusively
> desktop-oriented.

You make it sound like releng doesn't care at all about non-desktop packages.
The reason for the "exclusivity" is that the media that's typically built for
release (GRP, LiveCD) is targetted for the largest audience...desktop users. If
someone wants to volunteer to create a set of server-related GRP and a server
LiveCD (as silly as this is for most things), they wouldn't be blocked outright.

> b) Release trees have a nasty habit of picking up last minute changes
> (such as gcc 4.1) to suit the release, not stability.

Gcc 4.1.1 wasn't a last minute change. The release snapshots are originally
created a month or more before the actual release. As stuff is stabilized in the
tree, it's stabilized in the release snapshot as well. During the release cycle,
we planned for gcc-4.1.1 to be stable by release (at least for x86, amd64, and a
few other arches). We had it stable in the release snapshot a few weeks before
the tree did and were testing the crap out of it.

>> No version changes on any packages, except those which are necessary due
>> to a security violation, or a vulnerable package's dependencies.
>
> Tying a minimal-b0rkage tree to the arbitrary schedule of our releases
> does not serve all of our users.  We are back to the same arguments we
> had when I said that the Seeds project would have to have its own
> independent release schedules :(

The "release tree" isn't really for minimal breakage. The *real* intent (at
least from my POV) is to have a non-moving target for vendors to certify their
software against (wouldn't it be nice for Oracle to be actually supported on
Gentoo 2007.0 or something like that?), and so admins don't have to do the
"upgrade dance" once a week or even every day (like I do).

> Thereś little merit in us creating mostly stagnant trees.  Other Linux
> distros are already very good at doing that - far better than we will
> be at it - because they have advantages such as a paid workforce and
> more upstream developers on their books.

The "non-stagnant" nature of Gentoo isn't the only reason that people use
Gentoo. People use Gentoo for the configurability and customizability. As
someone who admins more than a handful of Gentoo servers, I would absolutely
*love* the combo of Gentoo's flexibility and a non-moving tree to make upgrades
easier to deal with.

--
Andrew Gaffney                            http://dev.gentoo.org/~agaffney/
Gentoo Linux Developer                                   Installer Project
Today's lesson in political correctness:      "Go asphyxiate on a phallus"
--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: Re: Versioning the tree

Steven J. Long
In reply to this post by Chris Gianelloni
Chris Gianelloni wrote:

> I have a script which already does several things:
>
> #1. grabs "best_visible" for stable on each arch
> #2. repeat for each SLOT
> #3. purge unnecessary files from FILESDIR
> #4. strip to only "stable" profiles from profiles.desc
> #5. purge unnecessary USE from use.local.desc and use.desc
> #6. strip unused eclasses
> #7. regenerate Manifest (via ebuild $foo digest)
>
> I had also planned on it doing the following, but they haven't been
> implemented just yet:
>
> - strip all USE flags that aren't used from use.mask (per-profile)
> - strip all packages that aren't available from package.mask
> (per-profile)
>
> What this gives us is a *much* smaller tree, as it only includes stable
> packages/versions.
>
The smaller tree sounds great in terms of maintenance for the volunteers who
maintain the release.
 

>> Obviously maintaining that list is some work; predominantly watching
>> the announcements from the security team and fixing up dependencies,
>> and masking out new packages (for what that's worth).  It could be
>> regenerated on some or all releases, or perhaps on a yearly basis.  It
>> would also mean that versions listed there cannot be removed from the
>> tree (until they're bumped in the list).
>
> Well, the simpler approach was to simply copy over the newer, secure
> ebuilds, and any required dependencies, into the release tree.  There'd
> be no need to update mask files and such this way.  It would also be
> generated with the releases, automatically.
>
<snip> .. The one
> disadvantage to my design is it needs infra.  It needs it's own
> repository and rsync.
>
What does that entail? Would a co-located server suffice?

(If it gets popular, I'd imagine those mirroring current rsyncs etc would
want to mirror the releases as well.)

> Basically, it makes the system act a bit more like some other
> development models.  We end up with the following:
>
> rsync://rsync.gentoo.org/gentoo-portage (this is equivalent to -current)
> rsync://rsync.gentoo.org/2007.0-release (this is the release tree)
>
> Now, the release trees are non-moving.  The 2007.0-release tree is
> *always* the 2007.0-release tree for as long as we decide to support it.
> Likely, this support period would begin as a single release and get
> extended as volunteers came around to support it.  New releases get
> their own tree.
>
Well count me in as a volunteer to help set this up and maintain an x86
release. I'm a pretty good coder if that helps.

> I also started writing a tool to "upgrade" the distribution.  The tool
> reads pre and post scripts for the upgrade, and performs the necessary
> steps.  Basically, a user can "dist-upgrade 2007.1" and the system
> would:
>
> - switch to the "2007.1" tree and "emerge --sync"
> - perform all "pre" steps from 2007.1
> - update world + revdep-rebuild + whatever else is necessary
> - perform all "post" steps
>
> With this, there would be a special "version" called "current" which
> would put the user on the "gentoo-portage" tree, AKA the tree we all
> know and love/loathe.  Release trees wouldn't have "arch" and "~arch" as
> everything would be stable there.  Testing would be done in the main
> tree.  This, in essence, gives us "testing", "release candidate" and
> "stable" environments, with the release trees being stable, and the
> current tree becoming test/release candidate.  Anything marked stable in
> the current tree would be a candidate for stable in the next release
> tree.
>
This sounds like a much more professional proposition to put to an IT exec.

> Users who want to use the current portage tree get what they want.
> Users who want a more stable tree get what they want.  Basically,
> everyone (hopefully) is happy, or at least as happy as we can feasibly
> make them.
>
This all sounds great. Respect for the work you've already put in.

other post:
> With the release trees, essentially only those
> interested in supporting the tree are required to work on it.  The tree
> is created entirely by scripts and is tested before it's released on
> the public.

Having it automated is definitely what I wanted to know about in the
original discussion.

From your post we need to add:
> - strip all USE flags that aren't used from use.mask (per-profile)
> - strip all packages that aren't available from package.mask
> (per-profile)

What language is the script implemented in?

Wrt security updates, is it possible to tie into GLSAs so that we could
automate updating packages that need it? By updating I mean adding the
ebuilds and any dependencies (or dependants that might require updating.)

Caleb Cushing wrote:
> .. isn't it possible only to sync certain
> parts of the tree using excludes. maybe some additional functionality
> saying only sync package X for updates.

Would that help in terms of having say, up to date GUI packages, or is it
just easier to use overlays?

Chris Gianelloni wrote:

> No version changes on any packages, except those which are necessary due
> to a security violation, or a vulnerable package's dependencies.
>
I could imagine a situation where a dependant package (ie one using the
package updated) would also require an update. It'd be rare though, so I
guess it wouldn't need automating.

> Now, I don't know if it is the best approach, but it is the best one
> that I could come up with on my own that allows for a slow-moving tree
> with higher QA done on it (as the release trees are what we use for
> releases, they undergo an enormous amount of testing, in the default
> configuration, of course) and minimal changes.
>
It sounds like the right approach to me. There's all the custom approach of
gentoo, eg overlays, so users can always stay up to date with anything they
particularly want.

> Something that would be useful would be for package maintainers that
> wish to participate to perform further testing and validation on
> packages in the release snapshot prior to its final freeze.  This would
> give us even more eyes on the tree and hopefully provide a much higher
> quality snapshot once we're done.

It sounds like it'd also make the releases in general have better QA. What
do package/ herd maintainers think?

--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: Versioning the tree

Alec Warner-2
Steve Long wrote:

> <snip> .. The one
>> disadvantage to my design is it needs infra.  It needs it's own
>> repository and rsync.
>>
> What does that entail? Would a co-located server suffice?
>
> (If it gets popular, I'd imagine those mirroring current rsyncs etc would
> want to mirror the releases as well.)
>

One of the problems here (IIRC) is that our mirrors are quite full.  Not
only must you mirror the release tree (which is trivial in size) you
also have to mirror the distfiles for that tree.  Right now our
distfiles are maintained by a wonderful python script that checks the
files in SRC_URI and fetches them to the master when needed; when an
ebuild is removed from CVS, the fetcher will not fetch the file and it
will get removed (after two weeks, afaik).

The bonus is that we then need to have a fetcher that runs over the main
tree and all supported release trees, and this of course increases
mirror size as we now have distfiles in the tree from a year ago to
support a year old release.  I am unsure how much space this would take
up; but this is an obstacle that must be overcome (somehow), preferably
by giving infra an estimate on the space reqs.


--
[hidden email] mailing list

Reply | Threaded
Open this post in threaded view
|

Re: Re: Versioning the tree

Stuart Herbert-2
In reply to this post by Andrew Gaffney
On 11/28/06, Andrew Gaffney <[hidden email]> wrote:
> You make it sound like releng doesn't care at all about non-desktop packages.

That wasn't how it was meant.  Was simply meant as a statement of
fact.  Releng activities are currently exclusively desktop-oriented.
Until that changes, releng snapshots aren't fit for the purpose of
being a non-moving tree, as far as servers are concerned.

> The reason for the "exclusivity" is that the media that's typically built for
> release (GRP, LiveCD) is targetted for the largest audience...desktop users. If
> someone wants to volunteer to create a set of server-related GRP and a server
> LiveCD (as silly as this is for most things), they wouldn't be blocked outright.

I'd like to see some figures proving that our largest audience is
desktop users.  I'm not prepared to take that on faith.  (Alas,
producing these figures is non-trivial in the extreme, if not
impossible).

> > b) Release trees have a nasty habit of picking up last minute changes
> > (such as gcc 4.1) to suit the release, not stability.
>
> Gcc 4.1.1 wasn't a last minute change.

I can't agree with you there.  It doesn't matter how many months of
planning and work you guys put into getting gcc-4.1 fit for stable.
If you're doing it off in your own little corner of the world, and
then springing it on the rest of us just days before the release
happens, then to the much larger dev community, it comes as a last
minute change.

If you're "testing the crap" out of something, but only in an
exclusively desktop-oriented way ... well, that can only really be
partial testing, can't it?

> The "release tree" isn't really for minimal breakage.

But that is what Steve (who started this thread) asked for.  And what
he has asked for in his previous thread too.

> The *real* intent (at
> least from my POV) is to have a non-moving target for vendors to certify their
> software against (wouldn't it be nice for Oracle to be actually supported on
> Gentoo 2007.0 or something like that?),

Well, there's a dichotamy here.  Sun were able to certify Gentoo
against their hardware without such a tree.  Has anyone approached
Oracle and asked them what their actual requirements are?  Do Oracle
actually want to certify Oracle on Gentoo at all?

I personally deplore this habit of trying to second guess what someone
else wants.  Assumptions are the mother of all fuckups.  Let's see an
email to -dev from someone at Oracle w/ their shopping list of needs,
and then base the discussion around that.

> and so admins don't have to do the
> "upgrade dance" once a week or even every day (like I do).

A slower-moving tree will substantially reduce this amount of work,
but it isn't going to go away, unless your boxes are on a private
network w/ no local security threats at all.

There'll always be GLSA's to respond to.  That's another issue that
needs to be handled w/ a slow-moving tree.  Are you going to restrict
changes in the slow-moving tree only to changes against a GLSA?

> The "non-stagnant" nature of Gentoo isn't the only reason that people use
> Gentoo. People use Gentoo for the configurability and customizability. As
> someone who admins more than a handful of Gentoo servers, I would absolutely
> *love* the combo of Gentoo's flexibility and a non-moving tree to make upgrades
> easier to deal with.

I honestly don't think you're ever going to get that out of Gentoo,
because of the lack of backporting.  Can you live with a slower-moving
tree?  Or do you personally really need a non-moving tree?

If you really need a non-moving tree, I think you're better off with
RHES or Ubuntu.

Best regards,
Stu
--
[hidden email] mailing list

123