Add bc back to the stage3

classic Classic list List threaded Threaded
50 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Add bc back to the stage3

Tom Wijsman-2
On Tue, 30 Sep 2014 18:58:18 +0100
Ciaran McCreesh <[hidden email]> wrote:

> On Mon, 29 Sep 2014 23:37:20 +0200
> Tom Wijsman <[hidden email]> wrote:
> > On Mon, 29 Sep 2014 04:05:19 +0000 (UTC)
> > "Jorge Manuel B. S. Vicetto" <[hidden email]> wrote:
> > > On Sat, 27 Sep 2014, Tom Wijsman wrote:
> > > > On Sat, 27 Sep 2014 13:22:45 +0100
> > > > Ciaran McCreesh <[hidden email]> wrote:
> > > >> On Sat, 27 Sep 2014 12:47:14 +0200
> > > >> Luca Barbato <[hidden email]> wrote:
> > > >>> Because I'd expect a stage3 to be posix compliant
> > > >>
> > > >> I agree. It's time to replace nano with Vim.
> > >
> > > It seems like everyone needs to "chill" a bit. Ciaran wasn't
> > > trolling, he was making a point. I'm sure everyone around here
> > > understood his point. There were no attacks and no "foul
> > > language", so can we move forward?
> >
> > Constructiveness does not rely on just making points, as replacing
> > nano with Vim is out of the context of adding bc back to stage3.
> > Editors are a world apart from a build tool, even more so from being
> > POSIX. In order to move forward beyond this point, that needs to be
> > recognized.

Ah, quotes are getting cut out, lovely; it focuses on what is crucial.

Reply | Threaded
Open this post in threaded view
|

Re: Add bc back to the stage3

Peter Stuge-4
In reply to this post by Jorge Manuel B. S. Vicetto-2
Hey Jorge,

Jorge Manuel B. S. Vicetto wrote:
> I know that our policies state that technical issues should be raised in
> the dev ml, although they also support doing the discussion in specialized
> mls, but they also mention that one should make an effort to contact those
> involved in the matter.

I completely agree that that is certainly the least someone can do.


Thanks

//Peter

Reply | Threaded
Open this post in threaded view
|

virtual/{posix,stage1,2,3} Was: [gentoo-dev] Add bc back to the stage3

Robin H. Johnson-2
In reply to this post by Richard Yao-2
On Sun, Sep 28, 2014 at 12:31:16PM -0400, Richard Yao wrote:
> May I suggest an alternative? We could implement sys-virtual/posix and
> make it depend on all packages that are not necessary for @system, but
> are necessary for proper POSIX compliance. Then we can tell users who
> need/want an environment containing all tools specified by POSIX, such
> as those not using sys-kernel/*-sources, to `emerge virtual/posix`.
I like this idea, for a virtual/posix; would let us trim a lot of other
things from some environments.

In a similar vein, would releng be open to moving stage1/2/3 package
sets to virtual packages or package sets? Presently they are inside
catalyst, and I think this would clean things up a lot.

--
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : [hidden email]
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Reply | Threaded
Open this post in threaded view
|

Re: virtual/{posix,stage1,2,3} Was: [gentoo-dev] Add bc back to the stage3

W. Trevor King-2
On Fri, Oct 10, 2014 at 09:22:18PM +0000, Robin H. Johnson wrote:
> In a similar vein, would releng be open to moving stage1/2/3 package
> sets to virtual packages or package sets? Presently they are inside
> catalyst, and I think this would clean things up a lot.

They're already in the Portage tree (the profile's packages.build for
stage1, scripts/bootstrap.sh for stage2, and the profile's packages
for stage3) [1].

Cheers,
Trevor

[1]: http://git.tremily.us/?p=catalyst.git;a=blob;f=doc/HOWTO.txt;h=cec22c35484d480fa127beb7be6bbdd9e942dc39;hb=HEAD#l139

     Linking to my public repo, because the gitweb service on
     git.overlays.gentoo.org is still broken.

--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: virtual/{posix,stage1,2,3} Was: [gentoo-dev] Add bc back to the stage3

Rich Freeman
On Fri, Oct 10, 2014 at 5:31 PM, W. Trevor King <[hidden email]> wrote:
> On Fri, Oct 10, 2014 at 09:22:18PM +0000, Robin H. Johnson wrote:
>> In a similar vein, would releng be open to moving stage1/2/3 package
>> sets to virtual packages or package sets? Presently they are inside
>> catalyst, and I think this would clean things up a lot.
>
> They're already in the Portage tree (the profile's packages.build for
> stage1, scripts/bootstrap.sh for stage2, and the profile's packages
> for stage3) [1].

Obviously this entails work on somebody's part, but would it still
make sense to make the stage build process more generic along the
lines Robin suggested?  That is, instead of having 3 specific places
we use to generate a stage1/2/3, we instead just define a stage as a
virtual of some kind that optionally depends on another stage (not
necessarily using the standard DEPEND mechanism) and then pulls in a
list of packages.  The root stage would not depend on another stage,
and therefore would be built from the host system.

The build system would just iteratively start at the bottom and output
a tarball or whatever as each level is completed, then use that as the
basis for bootstrapping the next.

Such a design would also eliminate the need to have the same list of
packages define the contents of @system and a stage3.  It could also
support any number of stages, with any names we wish.

My intent here isn't to get three cheers for making the releng team do
more work.  I'm actually more interested in feedback as to whether
there are any obvious flaws in an approach like this that I'm missing,
or whether something entirely different would be better.

I would also still have stage builds use a profile of some kind - that
could be useful from a standpoint of setting USE flags and so on.
However, if it made sense to build earlier stages with different flags
they could still be specified as USE dependencies (maybe due to
circular deps you have to build a package with different set of USE
flags before you can build the desired USE flags in a later stage).

--
Rich

Reply | Threaded
Open this post in threaded view
|

Re: virtual/{posix,stage1,2,3} Was: [gentoo-dev] Add bc back to the stage3

W. Trevor King-2
On Fri, Oct 10, 2014 at 09:45:37PM -0400, Rich Freeman wrote:
> Obviously this entails work on somebody's part, but would it still
> make sense to make the stage build process more generic along the
> lines Robin suggested?  That is, instead of having 3 specific places
> we use to generate a stage1/2/3, we instead just define a stage as a
> virtual of some kind that optionally depends on another stage (not
> necessarily using the standard DEPEND mechanism) and then pulls in a
> list of packages.  The root stage would not depend on another stage,
> and therefore would be built from the host system.

Why bother with multiple layers and per-profile package lists if a
straight emerge is all you need?  My ideal long-term situation would
be to explicitly list all a package's dependencies (no implicit
@system dependencies).  Then have the stage1 just emerge a
self-hosting virtual/toolchain (which can have per-arch dependencies
if it needs them) from an arbitrary seed, and stage2 rebuild -e using
itself.  Everything after that could be user-generated @world stuff,
and you might want virtual/release with nano and whatever else you
want to put up under
releases/$ARCH/autobuilds/gentoo-$ARCH-$DATE.tar.bz.  That's less
magical, and Catalyst could be a bit simpler, but I don't have the
energy to move things there ;).  It's also pretty much what we have
now, except @system is floating around between virtual/toolchain (our
current stage1 packages) and virtual/release (our current stage3
packages).

Cheers,
Trevor

--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: virtual/{posix,stage1,2,3} Was: Add bc back to the stage3

Steven J. Long
In reply to this post by Robin H. Johnson-2
On Fri, Oct 10, 2014 at 09:22:18PM +0000, Robin H. Johnson wrote:
> On Sun, Sep 28, 2014 at 12:31:16PM -0400, Richard Yao wrote:
> > May I suggest an alternative? We could implement sys-virtual/posix and
> > make it depend on all packages that are not necessary for @system, but
> > are necessary for proper POSIX compliance. Then we can tell users who
> > need/want an environment containing all tools specified by POSIX, such
> > as those not using sys-kernel/*-sources, to `emerge virtual/posix`.

kernel-sources doesn't get you that afaict: it just gets you bc. Certainly
doesn't get you ed, which is tiny and designed to be used in rootfs.

I just don't see the point in not providing a POSIX.2 system by default.
It feels like crippling the distro before it's out the door.

> I like this idea, for a virtual/posix; would let us trim a lot of other
> things from some environments.

Such as?

Not arguing with your use-case. Just wondering why ed and bc are considered
such major burdens, but polkit+systemd+udev+dbug+glib+glibc+godknows are a
minimal base.

I feel like I've gone through the looking-glass. (pam pulling in polkit?)

Anyhow, it seems to that in the minimal embedded env you're talking about,
the user can quite easily remove or package.provide things, and that
can be done in the profile as well as myriad other options for the expert
use-case; none of which speak to the default.

We used to have LDAP enabled by default, for the best part of a decade
to my recollection, just so that people could plug a disk into a Windows
environment, on the grounds that this made us look more "professional."
With respect, nothing looks more amateurish than a *nix that doesn't even
comply with POSIX.2.

It's not exactly hard, yet we don't manage it, but meantime let's add in
massively complex layers. It seems very masochistic sometimes, from an
external perspective. Loads of extra work, discussion and what often
reads like heartache, for want of a better word, in order to achieve what
you could have done really simply.

Still, moar code must be better, right?

--
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

Reply | Threaded
Open this post in threaded view
|

Re: virtual/{posix,stage1,2,3} Was: Add bc back to the stage3

Rich Freeman
On Sat, Oct 11, 2014 at 1:25 AM, Steven J. Long
<[hidden email]> wrote:
> Not arguing with your use-case. Just wondering why ed and bc are considered
> such major burdens, but polkit+systemd+udev+dbug+glib+glibc+godknows are a
> minimal base.

Nobody is talking about adding most of that stuff to the @system set.

The issue is that in general we're trying to reduce @system and list
explicit dependencies.  Honestly, my personal preference would be to
ditch it altogether as an implicit set of dependencies.

I'm all for having virtuals or profiles that make it easy for users to
install commonly-used sets of packages, including convenience ones
that reflect particular configurations.

The whole idea of dependencies, however, is that we don't have to have
endless debates about whether dbus is or isn't a great package to
install by default.  Instead, people just install the packages they
want, and they get the baggage that necessarily comes along with it.

So, this isn't about keeping ed/bc out while we have openssh of all
things in.  The concern is more about not adding more to something
that feels more like a hack at this point, and finding a better way of
letting users easily have sane defaults.  In the end, it is actually
about choice.  If done right it should actually make it easier for you
to have a system free of udev+glibc (I don't think the rest of what
you put there is part of the system set today).

--
Rich

Reply | Threaded
Open this post in threaded view
|

Re: virtual/{posix,stage1,2,3} Was: [gentoo-dev] Add bc back to the stage3

Jorge Manuel B. S. Vicetto-2
In reply to this post by Rich Freeman
On Fri, 10 Oct 2014, Rich Freeman wrote:

> On Fri, Oct 10, 2014 at 5:31 PM, W. Trevor King <[hidden email]> wrote:
>> On Fri, Oct 10, 2014 at 09:22:18PM +0000, Robin H. Johnson wrote:
>>> In a similar vein, would releng be open to moving stage1/2/3 package
>>> sets to virtual packages or package sets? Presently they are inside
>>> catalyst, and I think this would clean things up a lot.
>>
>> They're already in the Portage tree (the profile's packages.build for
>> stage1, scripts/bootstrap.sh for stage2, and the profile's packages
>> for stage3) [1].
>
> Obviously this entails work on somebody's part, but would it still
> make sense to make the stage build process more generic along the
> lines Robin suggested?  That is, instead of having 3 specific places
> we use to generate a stage1/2/3, we instead just define a stage as a
> virtual of some kind that optionally depends on another stage (not
> necessarily using the standard DEPEND mechanism) and then pulls in a
> list of packages.  The root stage would not depend on another stage,
> and therefore would be built from the host system.

All of the above suggestions would require changes to catalyst.
In any case, given the way we build a stage1 and bootstrap stage2, that
isn't possible. For stage1 and stage2 the *order* we build packages is
relevant. We rely on portage following that list in sequence.
Furthermore, because of the implicit dependencies and issues with circular
dependencies, it would likely be impossible for portage to resolve the
list of packages to build for stage1 and stage2 from a virtual.

> The build system would just iteratively start at the bottom and output
> a tarball or whatever as each level is completed, then use that as the
> basis for bootstrapping the next.

That's how catalyst already works.
To address one point in your last sentence in the above quote:

"The root stage would not depend on another stage, and therefore would be
built from the host system."

You almost always don't want to build with the state of a live system, but
of a clean seed - that's why releng tells catalyst to use a stage3 as the
seed for the stage1.

> Such a design would also eliminate the need to have the same list of
> packages define the contents of @system and a stage3.  It could also
> support any number of stages, with any names we wish.

No, not really. You're "over-thinking" this. To be able to "split" the
system set and the stages releng provides, all we need is to fix the bug
that was already mentioned before and have releng provide stages built
from a stage3 (while removing some packages from the system set) and a
list of packages that we want to provide (the ones dropped from system and
a select "few" others).

> I would also still have stage builds use a profile of some kind - that
> could be useful from a standpoint of setting USE flags and so on.
> However, if it made sense to build earlier stages with different flags
> they could still be specified as USE dependencies (maybe due to
> circular deps you have to build a package with different set of USE
> flags before you can build the desired USE flags in a later stage).

We build stages using a profile. One of the variables set on stage specs
is "profile" to list the profile to be used in the stage.

> --
> Rich

Regards,
Jorge Manuel B. S. Vicetto
Gentoo Developer


PS - As I've warned before, I'm not following closely the dev ml. So you
got this reply now, because I just happened to look at the emails in this
ml. If you want me to comment further, your best option is to poke me
about it.

Reply | Threaded
Open this post in threaded view
|

Re: virtual/{posix,stage1,2,3} Was: [gentoo-dev] Add bc back to the stage3

W. Trevor King-2
On Thu, Oct 16, 2014 at 12:13:45AM +0000, Jorge Manuel B. S. Vicetto wrote:
> For stage1 and stage2 the *order* we build packages is relevant.

Is this really true?  The stage1 is being built with ROOT, so it's
only using the seed stage3 packages.  It's hard to have cyclic
dependencies when you're using packages from one root (the seed
stage3) but installing into another (/tmp/stage1root).  Looking
through a stage1 log I see:

  emerge --quiet --oneshot --nodeps --update sys-apps/portage
  …
  emerge --quiet --update --deep --newuse --complete-graph --rebuild-if-new-ver gcc
  …
  emerge --quiet --usepkg --buildpkg --newuse --oneshot --nodeps sys-apps/baselayout
  …
  emerge --quiet --usepkg --buildpkg --newuse --oneshot sys-devel/gnuconfig sys-devel/bison …

The first two are just covering us against serious missmatches between
the package tree and the seed stage3 (and are installed with ROOT=/).
I expect we could handle shoving baselayout into the final emerge
along with the other packages.build stuff.

The logs for stage2 aren't as clear, but looking through the script I
only see:

* A Portage-updating emerge
* The main GCC, binutils, … emerge
* A possible 'emerge --prune sys-devel/gcc'

I'm not sure this is needed at all.  I'll try and find time to build a
stage3 directly from a stage1, and we'll see if it blows up ;).

Cheers,
Trevor

--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy


signature.asc (836 bytes) Download Attachment
123