Stable masks on multilib packages

classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Stable masks on multilib packages

Michał Górny-5
Hello, all.

The late multilib ppc issues made me re-check our stable masks on
abi_x86_* flags and, honestly, I'm not sure if we're doing things
the right way. First, a quick summary.


Let's consider dev-libs/elfutils that has a dependency
on app-arch/bzip2. If we're building 32-bit elfutils, it needs 32-bit
libbz2. This is enforced through a dep in form of:

  app-arch/bzip2[${MULTILIB_USEDEP}]

that gets expanded into:

  app-arch/bzip2[abi_x86_32(-)?,abi_x86_64(-)?,...,abi_mips_o32(-)?,...]

which means that any of the ABI_* flags that gets enabled on elfutils,
needs to be enabled on bzip2 as well. Of course, some of use.forcing
and masking gets applied here but that doesn't really matter.


Now, since we're pretty much converting a lot of different packages,
some of them are eligible for stabilization earlier than the others.
However, the extra MULTILIB_USEDEPs enforce stabilizing multilib
dependencies before the actual packages which made a few developers
unhappy.

That's why we decided to stable-mask the flags on affected packages.
Since the flags are masked on stable, people can stabilize packages
independently of whether their converted deps are stable or not. We
will be worrying about that consistency once we decide to unmask
the flags.

The extra advantage of that is that we can avoid pushing stable users
into the mess involved with partial conversion of emul-linux. The idea
is pretty simple: we keep emul-linux for stable, and once everything is
ready, we stable-unmask the flags and let stable users grok multilib.


Now, to the problem. Currently we're just stable-masking abi_x86_32
on amd64. This serves the latter purpose well and seems to work for the
former. This is probably because the remaining flag is use.forced
(abi_x86_64 on amd64, and abi_x86_32 on x86) which seems to add it to
implicit IUSE somehow.

floppym has done some research w/ stable elfutils and no stable
converted bzip2. It seems that if abi_x86_32 is stable.use.masked
and abi_x86_64 is use.forced, repoman groks it. However, if we remove
abi_x86_64 from use.force, it properly reports flag mismatch error
(since no stable bzip2 has IUSE=abi_x86_64).

Now, I honestly have no idea if this implicit use.force behavior is
PMS-y or not, and how other PMs handle it. I can't find something like
this in the PMS but that doc is horribly hard to cross-reference, so I
might be missing something. I'd appreciate if someone could help me
with that.


That said, I have an alternate idea inspired by the ppc breakage. I'm
thinking of replacing the amd64 abi_x86_32 mask with a global stable
mask of all abi_*_* flags on the relevant packages.

Differences:

1) old solution: native flag is forced, other flags are masked.
   new solution: all flags are masked.

2) old solution: we need to replicate the masks properly for different
                 arches/profiles.
   new solution: we can keep a single mask for all arches.

3) old solution: MULTILIB_USEDEP magically works (w/ portage at least).
   new solution: since all flags are disabled, MULTILIB_USEDEP
                 is a no-op and old packages match correctly.

4) old solution: forced native flag runs the native build.
   new solution: fallback code runs the native build (since no flags
                 are enabled).


Your thoughts?

--
Best regards,
Michał Górny

signature.asc (985 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Anthony G. Basile
On 03/31/2014 06:16 PM, Michał Górny wrote:

> Hello, all.
>
> The late multilib ppc issues made me re-check our stable masks on
> abi_x86_* flags and, honestly, I'm not sure if we're doing things
> the right way. First, a quick summary.
>
>
> Let's consider dev-libs/elfutils that has a dependency
> on app-arch/bzip2. If we're building 32-bit elfutils, it needs 32-bit
> libbz2. This is enforced through a dep in form of:
>
>    app-arch/bzip2[${MULTILIB_USEDEP}]
>
> that gets expanded into:
>
>    app-arch/bzip2[abi_x86_32(-)?,abi_x86_64(-)?,...,abi_mips_o32(-)?,...]
>
> which means that any of the ABI_* flags that gets enabled on elfutils,
> needs to be enabled on bzip2 as well. Of course, some of use.forcing
> and masking gets applied here but that doesn't really matter.
>
>
> Now, since we're pretty much converting a lot of different packages,
> some of them are eligible for stabilization earlier than the others.
> However, the extra MULTILIB_USEDEPs enforce stabilizing multilib
> dependencies before the actual packages which made a few developers
> unhappy.
>
> That's why we decided to stable-mask the flags on affected packages.
> Since the flags are masked on stable, people can stabilize packages
> independently of whether their converted deps are stable or not. We
> will be worrying about that consistency once we decide to unmask
> the flags.
>
> The extra advantage of that is that we can avoid pushing stable users
> into the mess involved with partial conversion of emul-linux. The idea
> is pretty simple: we keep emul-linux for stable, and once everything is
> ready, we stable-unmask the flags and let stable users grok multilib.
>
>
> Now, to the problem. Currently we're just stable-masking abi_x86_32
> on amd64. This serves the latter purpose well and seems to work for the
> former. This is probably because the remaining flag is use.forced
> (abi_x86_64 on amd64, and abi_x86_32 on x86) which seems to add it to
> implicit IUSE somehow.
>
> floppym has done some research w/ stable elfutils and no stable
> converted bzip2. It seems that if abi_x86_32 is stable.use.masked
> and abi_x86_64 is use.forced, repoman groks it. However, if we remove
> abi_x86_64 from use.force, it properly reports flag mismatch error
> (since no stable bzip2 has IUSE=abi_x86_64).
>
> Now, I honestly have no idea if this implicit use.force behavior is
> PMS-y or not, and how other PMs handle it. I can't find something like
> this in the PMS but that doc is horribly hard to cross-reference, so I
> might be missing something. I'd appreciate if someone could help me
> with that.
>
>
> That said, I have an alternate idea inspired by the ppc breakage. I'm
> thinking of replacing the amd64 abi_x86_32 mask with a global stable
> mask of all abi_*_* flags on the relevant packages.
I'm not sure exactly what you mean here --- where/how does this global
stable masking happen?  Right now you have a bunch of maskings of
abi_x86_32 on various packages in arch/amd64/package.use.stable.mask but
not on any other arches where you just have the use.mask/use.force
combination.  Are you going to migrate the amd64 file to other multilib
arches and mask non-native abis?  Like on mips64el/multilib/n32 would
you be masking abi_mips_o32 and abi_mips_n64 for all those packages?

>
> Differences:
>
> 1) old solution: native flag is forced, other flags are masked.
>     new solution: all flags are masked.
>
> 2) old solution: we need to replicate the masks properly for different
>                   arches/profiles.
>     new solution: we can keep a single mask for all arches.

This sounds like a partial answer to my above question.

>
> 3) old solution: MULTILIB_USEDEP magically works (w/ portage at least).
>     new solution: since all flags are disabled, MULTILIB_USEDEP
>                   is a no-op and old packages match correctly.
>
> 4) old solution: forced native flag runs the native build.
>     new solution: fallback code runs the native build (since no flags
>                   are enabled).
>
>
> Your thoughts?
>

How does this "go live" once these flags are enabled?  Do you just
remove the global mask all at once?  That sounds a bit scarier than just
removing the masks one package + deps at a time.  (Maybe a non-question
because I'm still now sure how this masking works.)

Also, I don't see that it should be an issue, but do you think this
might affect catalyst runs --- I have to ask because
repairing/reconfiguring seeds is a lot of work.

--
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : [hidden email]
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA


Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Ben de Groot-2
In reply to this post by Michał Górny-5
On 1 April 2014 06:16, Michał Górny <[hidden email]> wrote:
> Hello, all.
>
> The late multilib ppc issues made me re-check our stable masks on
> abi_x86_* flags and, honestly, I'm not sure if we're doing things
> the right way.
>
> That said, I have an alternate idea inspired by the ppc breakage.
>
> Your thoughts?

In my opinion your multilib approach introduces an unnecessary degree
of complexity, which --as has been shown here again-- is prone to
breakage.

It would be best for our beloved distro to revert all the multilib
changes, and try a different approach, or leave this prone-to-breakage
implementation to an overlay for the few people who would actually
benefit from it.

--
Cheers,

Ben | yngwin
Gentoo developer

Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Michał Górny-5
In reply to this post by Anthony G. Basile
Dnia 2014-03-31, o godz. 19:09:44
"Anthony G. Basile" <[hidden email]> napisał(a):

> On 03/31/2014 06:16 PM, Michał Górny wrote:
> > That said, I have an alternate idea inspired by the ppc breakage. I'm
> > thinking of replacing the amd64 abi_x86_32 mask with a global stable
> > mask of all abi_*_* flags on the relevant packages.
> I'm not sure exactly what you mean here --- where/how does this global
> stable masking happen?  Right now you have a bunch of maskings of
> abi_x86_32 on various packages in arch/amd64/package.use.stable.mask but
> not on any other arches where you just have the use.mask/use.force
> combination.  Are you going to migrate the amd64 file to other multilib
> arches and mask non-native abis?  Like on mips64el/multilib/n32 would
> you be masking abi_mips_o32 and abi_mips_n64 for all those packages?
The new solution would be base/package.use.stable.mask that would mask
all of multilib on all stable arches, on this long list of packages.
So, not only the two ABIs but all of them. Which would effectively make
ebuild behave like it had no multilib at all.

> > Your thoughts?
>
> How does this "go live" once these flags are enabled?  Do you just
> remove the global mask all at once?  That sounds a bit scarier than just
> removing the masks one package + deps at a time.  (Maybe a non-question
> because I'm still now sure how this masking works.)

Something like this, yes. Once all packages are migrated and some time
passes, we unmask all the flags locally and do a repoman run. We find
out what needs to go stable, report bugs, wait and repeat.

Then we do a second stabilization request, this time for testing
the tree (mostly emul-linux replacement) with multilib enabled. Once
arch teams are done testing, they remove the stable masks for
particular ABI.

When all reverse dependencies are fixed to use || () deps instead of
emul-linux (and rev-bumped) we can move away from emul-linux through
the usual procedure of last rites with a proper announcement. This is
likely the most fragile step of all but we will do our best to make it
as simple as possible, and I think our users can handle that.

> Also, I don't see that it should be an issue, but do you think this
> might affect catalyst runs --- I have to ask because
> repairing/reconfiguring seeds is a lot of work.

Well, I think this mostly depends on whether you want multiple multilib
ABIs in stages -- or if you assume that the toolchain is enough,
and people will build other ABIs as they need.

Though I think that once toolchain switches to abi_* flags (vapier
seems to show interest in that), we will use.force the necessary ABIs
on it and its dependencies, so there should be no explicit need for
change.

However, I guess the toolchain changes will be sent out for testing
anyway, so releng can check first hand.

--
Best regards,
Michał Górny

signature.asc (985 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Duncan-42
Michał Górny posted on Tue, 01 Apr 2014 07:54:11 +0200 as excerpted:

> Something like this, yes. Once all packages are migrated and some time
> passes, we unmask all the flags locally and do a repoman run. We find
> out what needs to go stable, report bugs, wait and repeat.

The big question on the user side is how many times we're going to have
to rebuild the same code, just because people are messing with multilib
stuff.  Masking all the now unmasked is going to force yet another
rebuild on stable users, who presumably are stable in part because they
/don't/ want all those "pointless" rebuilds.

Altho I'm ~arch so stable masks won't directly affect me, but at the same
time, I'm no-multilib, so all those --newuse and -rX-bump rebuilds really
ARE useless rebuilds of the /exact/ same thing!  While my system is fast
enough it's not a big deal here, it's still a pain tracking all those -rX
bumps to see what justifies the bump (at least I can tell simple USE flag
changes on sight), and I hurt for those with slower systems!

And wow, if you do go the global masking route, I seriously hope you have
plenty of announcement before the big unmasking, with at probably 30 days
warning and coverage on the web site, GMN, and news items, and possibly
even a second news-item warning 3-5 days before it happens, since the
global unmask will trigger about as big a universal rebuild as it gets,
and those always bring quite a few user complaints.  (Tho at least this
one won't be breaking much of the system until it's done, unlike some of
the big library upgrades of yore.)

IOW, while that very likely would have been the better way to do it
originally, I really don't know whether changing things up in mid-stream
like this is going to result in more or less pain for users.  Not being a
stable-arch user nor being multilib, I've been spared much of the worry
since on my systems it should "just work" either way, but honest
question, how far /are/ we into this, and will the change now result in
more work for more people (given that there's way more users than devs)
than just continuing the current path, now that we're in the middle of
things?

--
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman


Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Anthony G. Basile
In reply to this post by Michał Górny-5
On 04/01/2014 01:54 AM, Michał Górny wrote:

>> >Also, I don't see that it should be an issue, but do you think this
>> >might affect catalyst runs --- I have to ask because
>> >repairing/reconfiguring seeds is a lot of work.
> Well, I think this mostly depends on whether you want multiple multilib
> ABIs in stages -- or if you assume that the toolchain is enough,
> and people will build other ABIs as they need.
>
> Though I think that once toolchain switches to abi_* flags (vapier
> seems to show interest in that), we will use.force the necessary ABIs
> on it and its dependencies, so there should be no explicit need for
> change.
>
> However, I guess the toolchain changes will be sent out for testing
> anyway, so releng can check first hand.
I have taken a non-standard approach to the hardened/linux/uclibc
profiles where I do no inherit from arch.  There was too much cruft in
arch, so I just mirror the relevant stuff including the abi flag stuff.  
I understand my approach has pros/cons ... but give out the should when
the change is being tested and I'll see if I'm about to have any
breakage.   I currently seed at least 12 different
arches/isa/abi/hard-soft float combinations which would be a pita to fix
manually on native (read slow) hardware.

--
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : [hidden email]
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA


Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Anthony G. Basile
On 04/01/2014 07:40 AM, Anthony G. Basile wrote:

> On 04/01/2014 01:54 AM, Michał Górny wrote:
>>> >Also, I don't see that it should be an issue, but do you think this
>>> >might affect catalyst runs --- I have to ask because
>>> >repairing/reconfiguring seeds is a lot of work.
>> Well, I think this mostly depends on whether you want multiple multilib
>> ABIs in stages -- or if you assume that the toolchain is enough,
>> and people will build other ABIs as they need.
>>
>> Though I think that once toolchain switches to abi_* flags (vapier
>> seems to show interest in that), we will use.force the necessary ABIs
>> on it and its dependencies, so there should be no explicit need for
>> change.
>>
>> However, I guess the toolchain changes will be sent out for testing
>> anyway, so releng can check first hand.
> I have taken a non-standard approach to the hardened/linux/uclibc
> profiles where I do no inherit from arch.  There was too much cruft in
> arch, so I just mirror the relevant stuff including the abi flag
> stuff.  I understand my approach has pros/cons ...


> but give out the should when


Sorry, my fingers and brains are not in sync.  ... give out the SHOUT
when ...


> the change is being tested and I'll see if I'm about to have any
> breakage.   I currently seed at least 12 different
> arches/isa/abi/hard-soft float combinations which would be a pita to
> fix manually on native (read slow) hardware.
>


--
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : [hidden email]
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA


Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Alexandre Rostovtsev-2
In reply to this post by Ben de Groot-2
On Tue, 2014-04-01 at 13:13 +0800, Ben de Groot wrote:

> On 1 April 2014 06:16, Michał Górny <[hidden email]> wrote:
> > Hello, all.
> >
> > The late multilib ppc issues made me re-check our stable masks on
> > abi_x86_* flags and, honestly, I'm not sure if we're doing things
> > the right way.
> >
> > That said, I have an alternate idea inspired by the ppc breakage.
> >
> > Your thoughts?
>
> In my opinion your multilib approach introduces an unnecessary degree
> of complexity, which --as has been shown here again-- is prone to
> breakage.
>
> It would be best for our beloved distro to revert all the multilib
> changes, and try a different approach, or leave this prone-to-breakage
> implementation to an overlay for the few people who would actually
> benefit from it.
Speaking as a wine maintainer, the emul-linux-x86-* approach has many
times been proven to be an embarrassing failure and the main source of
pain and frustration for wine users. The sooner emul-linux-x86-* can be
removed from the tree, the better for Gentoo.

I am aware of only two solutions to the emul-linux-x86-* problems :
multilib-portage and multilib-build.eclass. The first requires everybody
to switch to a new package manager. The second allows us to keep using
portage, but requires library maintainers to add some simple boilerplate
to their ebuilds for multilib support.

Do you have yet another alternative in mind?

signature.asc (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Rich Freeman
On Tue, Apr 1, 2014 at 9:58 AM, Alexandre Rostovtsev
<[hidden email]> wrote:

> On Tue, 2014-04-01 at 13:13 +0800, Ben de Groot wrote:
>>
>> In my opinion your multilib approach introduces an unnecessary degree
>> of complexity, which --as has been shown here again-- is prone to
>> breakage.
>>
>> It would be best for our beloved distro to revert all the multilib
>> changes, and try a different approach, or leave this prone-to-breakage
>> implementation to an overlay for the few people who would actually
>> benefit from it.
>
> I am aware of only two solutions to the emul-linux-x86-* problems :
> multilib-portage and multilib-build.eclass. The first requires everybody
> to switch to a new package manager. The second allows us to keep using
> portage, but requires library maintainers to add some simple boilerplate
> to their ebuilds for multilib support.
>
> Do you have yet another alternative in mind?

++

I'm all for better solutions.  I'm not in favor of abandoning
solutions that work moderately well in favor of talking about maybe
coming up with something better sometime down the road.

The multilib eclass isn't a perfect solution.  It will have issues in
concept, design, and implementation.  These will come up from time to
time.  I don't think it is productive that anytime any of these pop up
that we end up having a discussion about just reverting it entirely.

By all means work on a competing solution.  Get it working and
supported by portage.  When the day comes that we want to endorse one
solution or the other as the preferred solution we can have that
discussion.

Dirty hack or not the multilib eclass seems to be working fairly well
in practice, especially considering just how big a change it is.  The
emul-linux packages were an even dirtier hack and those worked well
for a decade.

Progress can be incremental.

Rich

Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Ian Stakenvicius-2
In reply to this post by Michał Górny-5
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 31/03/14 06:16 PM, Michał Górny wrote:

> Hello, all.
>
> The late multilib ppc issues made me re-check our stable masks on
> abi_x86_* flags and, honestly, I'm not sure if we're doing things
> the right way. First, a quick summary.
>
>
> Let's consider dev-libs/elfutils that has a dependency on
> app-arch/bzip2. If we're building 32-bit elfutils, it needs 32-bit
> libbz2. This is enforced through a dep in form of:
>
> app-arch/bzip2[${MULTILIB_USEDEP}]
>
> that gets expanded into:
>
> app-arch/bzip2[abi_x86_32(-)?,abi_x86_64(-)?,...,abi_mips_o32(-)?,...]
>
>  which means that any of the ABI_* flags that gets enabled on
> elfutils, needs to be enabled on bzip2 as well. Of course, some of
> use.forcing and masking gets applied here but that doesn't really
> matter.
>
>
> Now, since we're pretty much converting a lot of different
> packages, some of them are eligible for stabilization earlier than
> the others. However, the extra MULTILIB_USEDEPs enforce stabilizing
> multilib dependencies before the actual packages which made a few
> developers unhappy.
>
> That's why we decided to stable-mask the flags on affected
> packages. Since the flags are masked on stable, people can
> stabilize packages independently of whether their converted deps
> are stable or not. We will be worrying about that consistency once
> we decide to unmask the flags.
>
> The extra advantage of that is that we can avoid pushing stable
> users into the mess involved with partial conversion of emul-linux.
> The idea is pretty simple: we keep emul-linux for stable, and once
> everything is ready, we stable-unmask the flags and let stable
> users grok multilib.
>
>
> Now, to the problem. Currently we're just stable-masking
> abi_x86_32 on amd64. This serves the latter purpose well and seems
> to work for the former. This is probably because the remaining flag
> is use.forced (abi_x86_64 on amd64, and abi_x86_32 on x86) which
> seems to add it to implicit IUSE somehow.
>
> floppym has done some research w/ stable elfutils and no stable
> converted bzip2. It seems that if abi_x86_32 is stable.use.masked
> and abi_x86_64 is use.forced, repoman groks it. However, if we
> remove abi_x86_64 from use.force, it properly reports flag mismatch
> error (since no stable bzip2 has IUSE=abi_x86_64).
>
> Now, I honestly have no idea if this implicit use.force behavior
> is PMS-y or not, and how other PMs handle it. I can't find
> something like this in the PMS but that doc is horribly hard to
> cross-reference, so I might be missing something. I'd appreciate if
> someone could help me with that.
>
>
> That said, I have an alternate idea inspired by the ppc breakage.
> I'm thinking of replacing the amd64 abi_x86_32 mask with a global
> stable mask of all abi_*_* flags on the relevant packages.
>
> Differences:
>
> 1) old solution: native flag is forced, other flags are masked. new
> solution: all flags are masked.
>
> 2) old solution: we need to replicate the masks properly for
> different arches/profiles. new solution: we can keep a single mask
> for all arches.
>
> 3) old solution: MULTILIB_USEDEP magically works (w/ portage at
> least). new solution: since all flags are disabled,
> MULTILIB_USEDEP is a no-op and old packages match correctly.
>
> 4) old solution: forced native flag runs the native build. new
> solution: fallback code runs the native build (since no flags are
> enabled).
>
>
> Your thoughts?
>


... how messy is this going to be on current systems, if implemented?
 This -is- going to cause use flag changes on a whackload of packages,
which at minimum I expect means a bunch of needless rebuilds.

I think it's probably best to keep with the current path, and just
hurry up the migration on x86.  If we switch to masking all abi_*_*'s,
then not only do we rebuild a whackload of packages now, but we also
end up rebuilding them again when we remove the abi_*_* mask, -even
if- the alternative abi's aren't needed.

A note on #3: if MULTILIB_USEDEP -isnt- working on other PMs, i hope
we have bug reports and we should work to resolve those ASAP or
abandon the project/find a different solution.  We're over a year into
it though (i think?) so i expect it's fine since I don't remember
hearing of any issues yet...


on a side note, to follow up -- when we do remove the masks, what're
the thoughts on perhaps changing the multilib profile on amd64 to
auto-enable abi_x86_32 flags on some packages?  Indeed, if the
toolchain is converted to multilib-build, would this be enough to
differentiate between amd64-multilib and pure64?  (bootstrapping may
of course still be a problem if users are trying to switch, but
probably not any more of a problem than it is right now, right?)



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iF4EAREIAAYFAlM610sACgkQ2ugaI38ACPAPoQD/TpzYLaAwLiRBSVLRII/lI1OK
fN9hOQ8wvMaHoKfuHK4A/RmUtt1G7k8HEsAhsMSWAYFUd0KdlbEd4snhcxaR6W+M
=n7yQ
-----END PGP SIGNATURE-----

Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Michał Górny-5
Dnia 2014-04-01, o godz. 11:12:11
Ian Stakenvicius <[hidden email]> napisał(a):

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 31/03/14 06:16 PM, Michał Górny wrote:
> > Hello, all.
> >
> > The late multilib ppc issues made me re-check our stable masks on
> > abi_x86_* flags and, honestly, I'm not sure if we're doing things
> > the right way. First, a quick summary.
> >
> >
> > Let's consider dev-libs/elfutils that has a dependency on
> > app-arch/bzip2. If we're building 32-bit elfutils, it needs 32-bit
> > libbz2. This is enforced through a dep in form of:
> >
> > app-arch/bzip2[${MULTILIB_USEDEP}]
> >
> > that gets expanded into:
> >
> > app-arch/bzip2[abi_x86_32(-)?,abi_x86_64(-)?,...,abi_mips_o32(-)?,...]
> >
> >  which means that any of the ABI_* flags that gets enabled on
> > elfutils, needs to be enabled on bzip2 as well. Of course, some of
> > use.forcing and masking gets applied here but that doesn't really
> > matter.
> >
> >
> > Now, since we're pretty much converting a lot of different
> > packages, some of them are eligible for stabilization earlier than
> > the others. However, the extra MULTILIB_USEDEPs enforce stabilizing
> > multilib dependencies before the actual packages which made a few
> > developers unhappy.
> >
> > That's why we decided to stable-mask the flags on affected
> > packages. Since the flags are masked on stable, people can
> > stabilize packages independently of whether their converted deps
> > are stable or not. We will be worrying about that consistency once
> > we decide to unmask the flags.
> >
> > The extra advantage of that is that we can avoid pushing stable
> > users into the mess involved with partial conversion of emul-linux.
> > The idea is pretty simple: we keep emul-linux for stable, and once
> > everything is ready, we stable-unmask the flags and let stable
> > users grok multilib.
> >
> >
> > Now, to the problem. Currently we're just stable-masking
> > abi_x86_32 on amd64. This serves the latter purpose well and seems
> > to work for the former. This is probably because the remaining flag
> > is use.forced (abi_x86_64 on amd64, and abi_x86_32 on x86) which
> > seems to add it to implicit IUSE somehow.
> >
> > floppym has done some research w/ stable elfutils and no stable
> > converted bzip2. It seems that if abi_x86_32 is stable.use.masked
> > and abi_x86_64 is use.forced, repoman groks it. However, if we
> > remove abi_x86_64 from use.force, it properly reports flag mismatch
> > error (since no stable bzip2 has IUSE=abi_x86_64).
> >
> > Now, I honestly have no idea if this implicit use.force behavior
> > is PMS-y or not, and how other PMs handle it. I can't find
> > something like this in the PMS but that doc is horribly hard to
> > cross-reference, so I might be missing something. I'd appreciate if
> > someone could help me with that.
> >
> >
> > That said, I have an alternate idea inspired by the ppc breakage.
> > I'm thinking of replacing the amd64 abi_x86_32 mask with a global
> > stable mask of all abi_*_* flags on the relevant packages.
> >
> > Differences:
> >
> > 1) old solution: native flag is forced, other flags are masked. new
> > solution: all flags are masked.
> >
> > 2) old solution: we need to replicate the masks properly for
> > different arches/profiles. new solution: we can keep a single mask
> > for all arches.
> >
> > 3) old solution: MULTILIB_USEDEP magically works (w/ portage at
> > least). new solution: since all flags are disabled,
> > MULTILIB_USEDEP is a no-op and old packages match correctly.
> >
> > 4) old solution: forced native flag runs the native build. new
> > solution: fallback code runs the native build (since no flags are
> > enabled).
> >
> >
> > Your thoughts?
> >
>
>
> ... how messy is this going to be on current systems, if implemented?
>  This -is- going to cause use flag changes on a whackload of packages,
> which at minimum I expect means a bunch of needless rebuilds.
>
> I think it's probably best to keep with the current path, and just
> hurry up the migration on x86.  If we switch to masking all abi_*_*'s,
> then not only do we rebuild a whackload of packages now, but we also
> end up rebuilding them again when we remove the abi_*_* mask, -even
> if- the alternative abi's aren't needed.
Note that the same issue is spilling into ppc now that vapier wants to
introduce multilib there. If we keep going the current way, we need to
mask abi_ppc_32 in the ppc profiles, and that means yet another copy of
the same mask to maintain in sync.

> A note on #3: if MULTILIB_USEDEP -isnt- working on other PMs, i hope
> we have bug reports and we should work to resolve those ASAP or
> abandon the project/find a different solution.  We're over a year into
> it though (i think?) so i expect it's fine since I don't remember
> hearing of any issues yet...

I don't know how many people are using Gentoo with alternative PMs
and stable keywords. I think Paludis could be a case but then, I don't
know how Exherbo incorporates our multilib into their package system.

> on a side note, to follow up -- when we do remove the masks, what're
> the thoughts on perhaps changing the multilib profile on amd64 to
> auto-enable abi_x86_32 flags on some packages?  Indeed, if the
> toolchain is converted to multilib-build, would this be enough to
> differentiate between amd64-multilib and pure64?  (bootstrapping may
> of course still be a problem if users are trying to switch, but
> probably not any more of a problem than it is right now, right?)

Could you be more specific on which packages? I used to think about
enabling it globally but now I see that autounmask would do a pretty
good work of unmasking the flags whenever necessary.

--
Best regards,
Michał Górny

signature.asc (985 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Ian Stakenvicius-2
In reply to this post by Duncan-42
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 01/04/14 07:26 AM, Duncan wrote:
> Altho I'm ~arch so stable masks won't directly affect me, but at
> the same time, I'm no-multilib, so all those --newuse and -rX-bump
> rebuilds really ARE useless rebuilds of the /exact/ same thing!
> While my system is fast enough it's not a big deal here, it's still
> a pain tracking all those -rX bumps to see what justifies the bump
> (at least I can tell simple USE flag changes on sight), and I hurt
> for those with slower systems!

OK, on no-multilib profiles, all of this stuff should be USE-masked
anyhow -- so far as I know, you can't even build the abi_x86_32 side
pf a package if your toolchain wasn't USE=multilib to begin with, can you?


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iF4EAREIAAYFAlM64BEACgkQ2ugaI38ACPACKAD9Ftshw+pl2RPunBcC3ZTDfhqi
y/Ertrl546FHUPzZ1e4A/1/bm7dGP894d+Rc2XfczJk3QoM/y1OcZ6y9ta3z7gGw
=n8me
-----END PGP SIGNATURE-----

Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Michał Górny-5
Dnia 2014-04-01, o godz. 11:49:37
Ian Stakenvicius <[hidden email]> napisał(a):

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 01/04/14 07:26 AM, Duncan wrote:
> > Altho I'm ~arch so stable masks won't directly affect me, but at
> > the same time, I'm no-multilib, so all those --newuse and -rX-bump
> > rebuilds really ARE useless rebuilds of the /exact/ same thing!
> > While my system is fast enough it's not a big deal here, it's still
> > a pain tracking all those -rX bumps to see what justifies the bump
> > (at least I can tell simple USE flag changes on sight), and I hurt
> > for those with slower systems!
>
> OK, on no-multilib profiles, all of this stuff should be USE-masked
> anyhow -- so far as I know, you can't even build the abi_x86_32 side
> pf a package if your toolchain wasn't USE=multilib to begin with, can you?
No-no-no.

abi_x86_64 designating the native ABI should be there and binary
packages can use it in dependencies. Which pretty much makes me think
we oughtn't mask all the flags.

--
Best regards,
Michał Górny

signature.asc (985 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Michał Górny-5
In reply to this post by Michał Górny-5
Dnia 2014-04-01, o godz. 00:16:17
Michał Górny <[hidden email]> napisał(a):

> Your thoughts?

Following the feedback on the mailing list, IRC and bugzilla, and some
of my own thoughts, I'd like to put a few important extra points I've
missed in the original mail:

1. playing with the masks will cause a large unnecessary --changed-use
rebuild for stable users, including the users of no-multilib profiles,

2. prebuilt software ebuilds are allowed to unconditionally use abi_*
flags in USE dependencies. Masking the native flags would break those
dependencies.

For those unaware, 2. here means that a pre-built x86 software can use
a single dependency like:

  dev-libs/libfoo[abi_x86_32]

that would satisfy both x86 (since abi_x86_32 is use.forced) and amd64
(by pulling in multilib).

Considering the two points above, I'd like to withdraw the idea of
adding global masks for abi_* and just leave it amd64-specific.


Additionally, the following important points were raised:

1. ulm believes that the use.force implicit IUSE magic is a bug in
portage and oughtn't be trusted upon,

2. keeping extra package.use.stable.masks for different arches seems
pretty much unmaintainable. It would mean a lot of duplicate package
lists in different profiles that need to be kept in sync.

Instead, I'd like to limit the scope of stable masks necessity and just
focus on stabilizing packages properly.


That is, the new position is:

1. we keep the flags for packages that are in emul-linux stable-masked
on amd64 until we decide to replace emul-linux,

2. we stabilize package dependencies properly so that there are no
dependency tree breakage when new ABIs are added.

Rationale: the multilib conversions are no longer moving in the fast
pace they used to, and the stabilization interest conflicts are less
likely than they were at the time of introducing
package.use.stable.masks.

Especially now that new multilib conversions are made package.masked,
we can give them enough time for people to be ready to stabilize
in some sane way.

We'd like to still keep the emul-linux-replacing packages masked
in order to keep emul-linux as single multilib provider for stable
packages. This is in order to avoid exposing stable users to issues
involved with partial conversions, and to confusing --autounmask
behavior that can possibly be created through mixing emul-linux
and multilib packages.


If nobody minds, I'm doing a repoman run now and in a few hours I
should have a list of packages that need to be stabilized in order to
clean up the dependency trees for multilib packages.

--
Best regards,
Michał Górny

signature.asc (985 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Patrick Lauer
In reply to this post by Ben de Groot-2
On 04/01/2014 01:13 PM, Ben de Groot wrote:

> On 1 April 2014 06:16, Michał Górny <[hidden email]> wrote:
>> Hello, all.
>>
>> The late multilib ppc issues made me re-check our stable masks on
>> abi_x86_* flags and, honestly, I'm not sure if we're doing things
>> the right way.
>>
>> That said, I have an alternate idea inspired by the ppc breakage.
>>
>> Your thoughts?
>
> In my opinion your multilib approach introduces an unnecessary degree
> of complexity, which --as has been shown here again-- is prone to
> breakage.

And it removes any chance of writing ebuilds - I seriously have no idea
how to fix those things now. They are multibuilds, not ebuilds.
>
> It would be best for our beloved distro to revert all the multilib
> changes, and try a different approach, or leave this prone-to-breakage
> implementation to an overlay for the few people who would actually
> benefit from it.
>
As a temporary stage they are kinda okish, maybe ... but ...

the whole transition strategy has been very very silly and should have
been staged in an overlay. I'd even build-test them and file bugs - just
don't do this salami tactic of one breakage a day.

Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Patrick Lauer
In reply to this post by Rich Freeman
On 04/01/2014 10:43 PM, Rich Freeman wrote:

> On Tue, Apr 1, 2014 at 9:58 AM, Alexandre Rostovtsev
> <[hidden email]> wrote:
>> On Tue, 2014-04-01 at 13:13 +0800, Ben de Groot wrote:
>>>
>>> In my opinion your multilib approach introduces an unnecessary degree
>>> of complexity, which --as has been shown here again-- is prone to
>>> breakage.
>>>
>>> It would be best for our beloved distro to revert all the multilib
>>> changes, and try a different approach, or leave this prone-to-breakage
>>> implementation to an overlay for the few people who would actually
>>> benefit from it.
>>
>> I am aware of only two solutions to the emul-linux-x86-* problems :
>> multilib-portage and multilib-build.eclass. The first requires everybody
>> to switch to a new package manager. The second allows us to keep using
>> portage, but requires library maintainers to add some simple boilerplate
>> to their ebuilds for multilib support.
>>
>> Do you have yet another alternative in mind?
>
> ++
>
> I'm all for better solutions.  I'm not in favor of abandoning
> solutions that work moderately well in favor of talking about maybe
> coming up with something better sometime down the road.
>
> The multilib eclass isn't a perfect solution.  It will have issues in
> concept, design, and implementation.  These will come up from time to
> time.  I don't think it is productive that anytime any of these pop up
> that we end up having a discussion about just reverting it entirely.

So why do we have this discussion?

> By all means work on a competing solution.  Get it working and
> supported by portage.  When the day comes that we want to endorse one
> solution or the other as the preferred solution we can have that
> discussion.

That is hilariously funny. Wow.

Now let's just continue to ignore the existing multilib-portage work so
we can claim it's irrelevant, while shifting the conditions for
accepting it whenever it is convenient, while silently adding the
competing method in-tree so it's all decided now anyway ...



Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Rich Freeman
On Tue, Apr 1, 2014 at 8:13 PM, Patrick Lauer <[hidden email]> wrote:
> Now let's just continue to ignore the existing multilib-portage work so
> we can claim it's irrelevant, while shifting the conditions for
> accepting it whenever it is convenient, while silently adding the
> competing method in-tree so it's all decided now anyway ...

Conditions for accepting it?  What are those, and how have they changed?

Rich

Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Duncan-42
In reply to this post by Michał Górny-5
Michał Górny posted on Tue, 01 Apr 2014 19:15:26 +0200 as excerpted:

> If nobody minds, I'm doing a repoman run now and in a few hours I should
> have a list of packages that need to be stabilized in order to clean up
> the dependency trees for multilib packages.

All this seems sane by me.  Thanks for all the work you and others are
putting in on this, even if it's not something I'm likely to use, because
I know other gentooers do. =:^)

--
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman


Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Ben de Groot-2
In reply to this post by Alexandre Rostovtsev-2
On 1 April 2014 21:58, Alexandre Rostovtsev <[hidden email]> wrote:

> On Tue, 2014-04-01 at 13:13 +0800, Ben de Groot wrote:
>> On 1 April 2014 06:16, Michał Górny <[hidden email]> wrote:
>> > Hello, all.
>> >
>> > The late multilib ppc issues made me re-check our stable masks on
>> > abi_x86_* flags and, honestly, I'm not sure if we're doing things
>> > the right way.
>> >
>> > That said, I have an alternate idea inspired by the ppc breakage.
>> >
>> > Your thoughts?
>>
>> In my opinion your multilib approach introduces an unnecessary degree
>> of complexity, which --as has been shown here again-- is prone to
>> breakage.
>>
>> It would be best for our beloved distro to revert all the multilib
>> changes, and try a different approach, or leave this prone-to-breakage
>> implementation to an overlay for the few people who would actually
>> benefit from it.
>
> Speaking as a wine maintainer, the emul-linux-x86-* approach has many
> times been proven to be an embarrassing failure and the main source of
> pain and frustration for wine users. The sooner emul-linux-x86-* can be
> removed from the tree, the better for Gentoo.

I would like to see an honest cost-benefit analysis of the
emul-linux-x86 approach compared to the multilib eclass approach.
Because in my experience the latter introduces more breakage and
higher maintenance costs.

> I am aware of only two solutions to the emul-linux-x86-* problems :
> multilib-portage and multilib-build.eclass. The first requires everybody
> to switch to a new package manager. The second allows us to keep using
> portage, but requires library maintainers to add some simple boilerplate
> to their ebuilds for multilib support.
>
> Do you have yet another alternative in mind?

In my mind the emul-linux-x86 approach is more acceptable. I don't
have experience with multilib-portage, as I don't have a use case for
it. Another option, which seems to me to be more reasonable and which
has greatly lower maintenance costs, is using a chroot.

--
Cheers,

Ben | yngwin
Gentoo developer

Reply | Threaded
Open this post in threaded view
|

Re: Stable masks on multilib packages

Ben de Groot-2
In reply to this post by Patrick Lauer
On 2 April 2014 07:38, Patrick Lauer <[hidden email]> wrote:

> On 04/01/2014 01:13 PM, Ben de Groot wrote:
>> On 1 April 2014 06:16, Michał Górny <[hidden email]> wrote:
>>> Hello, all.
>>>
>>> The late multilib ppc issues made me re-check our stable masks on
>>> abi_x86_* flags and, honestly, I'm not sure if we're doing things
>>> the right way.
>>>
>>> That said, I have an alternate idea inspired by the ppc breakage.
>>>
>>> Your thoughts?
>>
>> In my opinion your multilib approach introduces an unnecessary degree
>> of complexity, which --as has been shown here again-- is prone to
>> breakage.
>
> And it removes any chance of writing ebuilds - I seriously have no idea
> how to fix those things now. They are multibuilds, not ebuilds.

This is part of the complexity I mentioned. I significantly raises
maintenance costs, and I'm not convinced of the benefit.

>>
>> It would be best for our beloved distro to revert all the multilib
>> changes, and try a different approach, or leave this prone-to-breakage
>> implementation to an overlay for the few people who would actually
>> benefit from it.
>>
> As a temporary stage they are kinda okish, maybe ... but ...
>
> the whole transition strategy has been very very silly and should have
> been staged in an overlay. I'd even build-test them and file bugs - just
> don't do this salami tactic of one breakage a day.
>

I'm strongly considering reverting these changes in the packages I
maintain. I'm tired of having to deal time and again with multilib
breakage.

Either that, or someone else can take over primary maintainership.

--
Cheers,

Ben | yngwin
Gentoo developer

12