Please consider removing use.stable.mask and package.use.stable.mask

classic Classic list List threaded Threaded
111 messages Options
1234 ... 6
Reply | Threaded
Open this post in threaded view
|

Please consider removing use.stable.mask and package.use.stable.mask

Martin Vaeth
Hello.

The new "features" use.stable.mask and package.use.stable.mask
have turned maintaining systems with mixed ARCH and ~ARCH keywords
into a nightmare:

Similarly to the (fortunately dropped) concept of forcing
useflags if certain packages are installed this forces a
magic on the user which can hardly be managed since it is
not clearly presented to the user but hidden in some profiles.

As I understand, it tries to solve a "social" issue
(that an ARCH user might set a USE-flag which eventually
pulls in an ~ARCH package) on a technical level
(by forcibly disabling the USE-flag for the user).
Solving social problems by technical means is never a good
idea:

While the former gives the stable user a clear message
what is going wrong (after all, many stable users want
some package which has not a stable version yet, so this
should be something everybody should be able to handle),
the latter hides causes and makes things almost unpredictable:

Even if the user has put the dependency into
package.accept_keywords, he will not be able to use the
USE-flag on his packages unless
*he puts stable versions into package.accept_keywords*
(or - simlarly worse - overrides the profile).

The really bad thing is that this is happening by magic
"behind the user's back", i.e. contradicting the user's setting
of USE-flag and package.use: If the user does not carefully
read emerge's -v output, he even does not get informed that
the support for his unstable package is dropped from the
stable package, despite he might have specified the corresponding
USE-flag globally. Even worse, even when reading the output
carefully, the user cannot understand the reason:
Since there are many reasons why use-flags can appear in ()
he will probably not even conjecture that actually something
will not be working as he expected.

Here are some other examples of negative effects happening
just recently to me, ordered from not so severe to very bad:

1. For several reasons I always want the most current
emul-linux-x86* libraries, so they are in package.accept_keywords.
Due to global ABI_X86=32 (which I also want), this forced me
of course to put several libraries to ~amd64 since only
new version support this. Some of the libraries are actually
stable, so I have removed them from package.accept_keywords.
So far, so good.
But suddenly portage spitted unexplainable dependency errors,
and I only expert users manually reading the profiles can
understand that the reason is that use.stable.mask requires
that stable versions need to be keyworded ~amd64
(or use.stable.mask has to be overridden in my profile).

2. Just a few days ago dev-lang/python-exec:2 became stable
on amd64, but dev-python/python-exec:2 is still ~amd64.
Just to be sure to not miss anything, I have put the latter
into package.accept_keywords, and hell break loose:
Portaqe wanted me to emerge the git version of
dev-lang/python-exec:0 and reemerge dozens of packages.
I needed hours to find out what is the actual cause:
The package.accept_keywords caused the use.stable.mask of
python_targets_pypy2_0 und python_targets_python3_3 to become
ineffective for dev-python/python-exec, but of course it is still
effective for dev-lang/python-exec which is not be the case
if I unmask the git version of dev-lang/python-exec.

This is completely confusing since nowhere the git-version
is marked differently than the non-git version.

So, yes, portage's behaviour can be explained, but, no,
it is not understandable by a user who wants portage
to report what is really wrong and who does not want
to check manually by reading all profiles and hunt down
for reasons why certain flags are magically forced/disabled.

3. As a side effect of 2., I realized that libreoffice and a dozen
further packages get forced a rebuild. So, if eventually
python-3.3 becomes stable and is removed from use.stable.mask,
all stable users will have to rebuild almost their whole world,
because python-exec then has one more "dummy" USE-flag. The same
will happen again if pypy-2.0 becomes stable.
So a lot of unnecessary rebuilds happen to stable users because
of {package.,}use.stable.mask which most of the developers
(who are often ~amd64 users) do not realize.

Best Regards
Martin


Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Tom Wijsman-2
On Wed, 13 Nov 2013 10:28:02 +0000 (UTC)
Martin Vaeth <[hidden email]> wrote:

> Hello.
>
> The new "features" use.stable.mask and package.use.stable.mask
> have turned maintaining systems with mixed ARCH and ~ARCH keywords
> into a nightmare:

They are considered unsupported by many; so, going down that path you
need to be acquainted with Portage enough to keep a consistent system.

> Similarly to the (fortunately dropped) concept of forcing
> useflags if certain packages are installed this forces a
> magic on the user which can hardly be managed since it is
> not clearly presented to the user but hidden in some profiles.
>
> As I understand, it tries to solve a "social" issue
> (that an ARCH user might set a USE-flag which eventually
> pulls in an ~ARCH package) on a technical level
> (by forcibly disabling the USE-flag for the user).

That's one approach, it might also be used when a package can be
stabilized but a certain of feature of the package cannot; eg.
USE="minimal" could be broken on a certain package because it removed a
bit too much, thus it could be stabilized with USE="-minimal" forced.

Anyhow, I think we should make sure to weight "why we need to have it"
against "how it bothers which users"; and not just focus on users alone.

And other than that, are there alternatives? Something we can do better?

Sometimes problems can be resolved by better communication too; perhaps
there are things we can inform the user about in pkg_postinst, or
improve Portage to let the user know of a particular forced USE flag.

> Solving social problems by technical means is never a good
> idea:
>
> While the former gives the stable user a clear message
> what is going wrong (after all, many stable users want
> some package which has not a stable version yet, so this
> should be something everybody should be able to handle),
> the latter hides causes and makes things almost unpredictable:
>
> Even if the user has put the dependency into
> package.accept_keywords, he will not be able to use the
> USE-flag on his packages unless
> *he puts stable versions into package.accept_keywords*
> (or - simlarly worse - overrides the profile).
That are indeed the two approaches, I don't see a problem with putting
the version itself in accept_keywords or overriding the profile;
furthermore, overriding forced and/or masked USE flag is the exception,
it doesn't happen so frequently so I doubt this is a huge problem.

> The really bad thing is that this is happening by magic
> "behind the user's back", i.e. contradicting the user's setting
> of USE-flag and package.use: If the user does not carefully
> read emerge's -v output, he even does not get informed that
> the support for his unstable package is dropped from the
> stable package, despite he might have specified the corresponding
> USE-flag globally.

System upgrades have to be done carefully; so, the user skipping it is
something we cannot account for except perhaps providing support for
the user receiving some kind of verbose summary of such forces / masks
being present at the end of the emerge output. But you'll have to
contact the Portage developers to achieve such improvements.

> Even worse, even when reading the output
> carefully, the user cannot understand the reason:
> Since there are many reasons why use-flags can appear in ()
> he will probably not even conjecture that actually something
> will not be working as he expected.

Most of these reasons can be uniquely determined from the output; but
indeed, some might yield the same output, but this is again something
that the Portage developers can improve and not a reason for removal.

> Here are some other examples of negative effects happening
> just recently to me, ordered from not so severe to very bad:
>
> 1. For several reasons I always want the most current
> emul-linux-x86* libraries, so they are in package.accept_keywords.
> Due to global ABI_X86=32 (which I also want), this forced me
> of course to put several libraries to ~amd64 since only
> new version support this. Some of the libraries are actually
> stable, so I have removed them from package.accept_keywords.
> So far, so good.
> But suddenly portage spitted unexplainable dependency errors,
> and I only expert users manually reading the profiles can
> understand that the reason is that use.stable.mask requires
> that stable versions need to be keyworded ~amd64
> (or use.stable.mask has to be overridden in my profile).
This confuses me; isn't the goal to have one multilib apprach or the
other? Especially due to the blockers between them.

I agree Portage output can be better, I'm not sure if use.stable.mask
is really the problem here though; but rather the nature of mixing them.

> 2. Just a few days ago dev-lang/python-exec:2 became stable
> on amd64, but dev-python/python-exec:2 is still ~amd64.
> Just to be sure to not miss anything, I have put the latter
> into package.accept_keywords, and hell break loose:

Hell indeed breaks loose if you mix stable and unstable; but note that
this package had an accident, see the related news item for details.

> Portaqe wanted me to emerge the git version of
> dev-lang/python-exec:0 and reemerge dozens of packages.

This is a consequence of that hell; if you don't agree and this is due
to the Portage tree, please show the dependency chain that causes this.

> I needed hours to find out what is the actual cause:
> The package.accept_keywords caused the use.stable.mask of
> python_targets_pypy2_0 und python_targets_python3_3 to become
> ineffective for dev-python/python-exec,

It doesn't matter, dev-python/python-exec installs no contents.

> but of course it is still
> effective for dev-lang/python-exec which is not be the case
> if I unmask the git version of dev-lang/python-exec.

Which one is not meant to do anyway; and from what you wrote, I don't
think you intent to either.

> This is completely confusing since nowhere the git-version
> is marked differently than the non-git version.

Marked in which way? One is stable, the other unkeyworded.

> So, yes, portage's behaviour can be explained, but, no,
> it is not understandable by a user who wants portage
> to report what is really wrong and who does not want
> to check manually by reading all profiles and hunt down
> for reasons why certain flags are magically forced/disabled.

Which output are we discussing here?

If it is due to the hell above, we warn users up front for that.

> 3. As a side effect of 2., I realized that libreoffice and a dozen
> further packages get forced a rebuild. So, if eventually
> python-3.3 becomes stable and is removed from use.stable.mask,
> all stable users will have to rebuild almost their whole world,
> because python-exec then has one more "dummy" USE-flag. The same
> will happen again if pypy-2.0 becomes stable.
> So a lot of unnecessary rebuilds happen to stable users because
> of {package.,}use.stable.mask which most of the developers
> (who are often ~amd64 users) do not realize.

That is to be expected on such stabilizations, I doubt they are
unnecessary; if I misunderstood, feel free to give an example.

--
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : [hidden email]
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

signature.asc (501 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Thomas Kahle
Hi,

On 11/13/2013 12:39 PM, Tom Wijsman wrote:

> On Wed, 13 Nov 2013 10:28:02 +0000 (UTC)
> Martin Vaeth <[hidden email]> wrote:
>
>> Hello.
>>
>> The new "features" use.stable.mask and package.use.stable.mask
>> have turned maintaining systems with mixed ARCH and ~ARCH keywords
>> into a nightmare:
>
> They are considered unsupported by many; so, going down that path you
> need to be acquainted with Portage enough to keep a consistent system.
This argument has come up several times, but is it valid?  For me
and other people I know, the main reason to use Gentoo is the
rolling release model and this implies that you can mix package
versions as long as version dependencies are satisfied.  When the
dependency is "cat/package" then this should mean that it works
with every version.  If it does not, then the ebuild's
dependencies should be updated.

The handbook says nothing about "unsupported":

http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=3&chap=3

If "many" choose to change this policy, there is no reason
anymore for me to use Gentoo.

>> Similarly to the (fortunately dropped) concept of forcing
>> useflags if certain packages are installed this forces a
>> magic on the user which can hardly be managed since it is
>> not clearly presented to the user but hidden in some profiles.
>>
>> As I understand, it tries to solve a "social" issue
>> (that an ARCH user might set a USE-flag which eventually
>> pulls in an ~ARCH package) on a technical level
>> (by forcibly disabling the USE-flag for the user).
>
> That's one approach, it might also be used when a package can be
> stabilized but a certain of feature of the package cannot; eg.
> USE="minimal" could be broken on a certain package because it removed a
> bit too much, thus it could be stabilized with USE="-minimal" forced.
>
> Anyhow, I think we should make sure to weight "why we need to have it"
> against "how it bothers which users"; and not just focus on users alone.
>
> And other than that, are there alternatives? Something we can do better?
We could consider reducing the feature set of portage and live
with the "problems" that arise.  When I started using Gentoo a
package could simply not go stable until all dependencies for all
USE flags were also stable.  Masking USE flags was reserved to a
short list of very special architecture depend special cases.

[...]

>> 2. Just a few days ago dev-lang/python-exec:2 became stable
>> on amd64, but dev-python/python-exec:2 is still ~amd64.
>> Just to be sure to not miss anything, I have put the latter
>> into package.accept_keywords, and hell break loose:
>
> Hell indeed breaks loose if you mix stable and unstable; but note that
> this package had an accident, see the related news item for details.

Do you mean stable and unstable in this case, or in general?

[...]

In general I share the sentiment.  The complexity of using
portage has increased a lot lately.  Not only does it take long
to find out why things suddenly go wrong after tree sync, also
just the time until 'emerge -avUDN world' comes back with a
proposal has grown to several minutes where it was few seconds
when I started with Gentoo.

There has been a lot of effort to make revdep-rebuild unessecary,
but now that it is mostly implemented, I don't know if it was
worth the price.  I spend more time now just reconfiguring
keywords to update the system than I spent back in the old days
where revdep would just fix things.  If the answer is, that I
should not mix arch and ~arch, then I'll not use Gentoo anymore.

Cheers,
Thomas


--
Thomas Kahle


signature.asc (567 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Rich Freeman
On Wed, Nov 13, 2013 at 8:25 AM, Thomas Kahle <[hidden email]> wrote:

> On 11/13/2013 12:39 PM, Tom Wijsman wrote:
>> On Wed, 13 Nov 2013 10:28:02 +0000 (UTC)
>> Martin Vaeth <[hidden email]> wrote:
>>
>>> Hello.
>>>
>>> The new "features" use.stable.mask and package.use.stable.mask
>>> have turned maintaining systems with mixed ARCH and ~ARCH keywords
>>> into a nightmare:
>>
>> They are considered unsupported by many; so, going down that path you
>> need to be acquainted with Portage enough to keep a consistent system.
>
> This argument has come up several times, but is it valid?

Honestly, opinions vary on this one and I don't think it is a
productive path to go down.  I also feel that being able to mix
keywords is a big benefit of using Gentoo.  I'd rather focus on
practical ways to make this easier rather than whether it is
desirable.

That said, there are always going to be situations where mixing
keywords isn't practical.  You're not going to run stable chromium
against ~arch v8, or mixed keywords between kdelibs and kwin, etc.

> We could consider reducing the feature set of portage and live
> with the "problems" that arise.  When I started using Gentoo a
> package could simply not go stable until all dependencies for all
> USE flags were also stable.  Masking USE flags was reserved to a
> short list of very special architecture depend special cases.

I don't think going backwards is the solution.  Back in the old days
packages broke from time to time because we didn't have adequate ways
to express dependencies, and I don't think it is a good solution to
strip USE flags out of packages when they go stable so that users
don't even have the option to use them.

It makes more sense to identify what specifically is causing problems
and come up with better solutions to them.  That said, your original
email contained a few separate issues and they're probably best dealt
with individually.  We're not going to have a common solution for
multilib, stable use masking, python-exec, and whatever other issues
are lurking beneath the surface.

Rich

Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Tom Wijsman-2
In reply to this post by Thomas Kahle
On Wed, 13 Nov 2013 14:25:11 +0100
Thomas Kahle <[hidden email]> wrote:

> Hi,
>
> On 11/13/2013 12:39 PM, Tom Wijsman wrote:
> > On Wed, 13 Nov 2013 10:28:02 +0000 (UTC)
> > Martin Vaeth <[hidden email]> wrote:
> >
> >> Hello.
> >>
> >> The new "features" use.stable.mask and package.use.stable.mask
> >> have turned maintaining systems with mixed ARCH and ~ARCH keywords
> >> into a nightmare:
> >
> > They are considered unsupported by many; so, going down that path
> > you need to be acquainted with Portage enough to keep a consistent
> > system.
>
> This argument has come up several times, but is it valid?
For more insight see

http://comments.gmane.org/gmane.linux.gentoo.devel/87839

and I quote "quite a few bugs are closed as invalid when a mixed system
is detected" as well as "mixing stable and testing is (likely) invalid
when unmasking one package in the unstable branch causes (reverse)
dependency resolution issues with another package in the stable branch".

Not to forget that we often test with everything stable or with
everything on testing; so, if you mix packages from both you end up in
untested situations where dependencies do not match what we intend.
Some of us fix them, but not everyone might; others might even just end
up setting a dependency range because of the stable / unstable mix.

Besides it being unsupported by some (I'll drop the "many" sound to it)
there are also just users in general that suggest others to avoid it;
simply because of the blockers and conflicts it brings along, it's
something that works out well for users that deeply understand what is
going on but for new users it is just a recipe for disaster.

Regardless of that, in my opinion, we should avoid encouraging
situations that result in having to deal with more unnecessary blockers
and conflicts; but that doesn't mean we don't want to see it. The same
link above highlights how it can help rather than obstruct Gentoo too.

> For me
> and other people I know, the main reason to use Gentoo is the
> rolling release model and this implies that you can mix package
> versions as long as version dependencies are satisfied. When the
> dependency is "cat/package" then this should mean that it works
> with every version.  If it does not, then the ebuild's
> dependencies should be updated.

Given that maintainers often test against test versions and that arch
teams often test against stable versions, nobody really tests the rests
of the situations except for some people that mix and come across it;
so, effectively the version dependencies do not always match reality
and therefore cannot imply that you can mix it.

It's also very unrealistic to check every package for every bump; at
least not with the amount of work compared to the amount of manpower we
have, it does happen a lot and we are required to do so to some extent.
But you will note that this ends up in situations where the new version
just ends up being in package.mask for a long while, for example ffmpeg.

Others are either new and forgot or might show a lack of care; which
results in it being properly tested for non-mixed systems, but not
properly tested for mixed systems. Enumerating all dependency versions
is a quite busy job; we often rely on the restrictions listed
in ./configure.{in,ac} but those are not always correct either,
sometimes they are even just absent.

> The handbook says nothing about "unsupported":
>
> http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=3&chap=3
>
> If "many" choose to change this policy, there is no reason
> anymore for me to use Gentoo.

Which policy? It does not say anything about "supported" either. I
believe that its intention has always been to change it for some
packages, not for wide sets of them; the former doesn't cause much
problems, the latter is a recipe for disaster...

Note that it is merely unsupported by some developers, there are
definitely other developers that support it; and this does not imply
anything at all about our support venues.

> >> Similarly to the (fortunately dropped) concept of forcing
> >> useflags if certain packages are installed this forces a
> >> magic on the user which can hardly be managed since it is
> >> not clearly presented to the user but hidden in some profiles.
> >>
> >> As I understand, it tries to solve a "social" issue
> >> (that an ARCH user might set a USE-flag which eventually
> >> pulls in an ~ARCH package) on a technical level
> >> (by forcibly disabling the USE-flag for the user).
> >
> > That's one approach, it might also be used when a package can be
> > stabilized but a certain of feature of the package cannot; eg.
> > USE="minimal" could be broken on a certain package because it
> > removed a bit too much, thus it could be stabilized with
> > USE="-minimal" forced.
> >
> > Anyhow, I think we should make sure to weight "why we need to have
> > it" against "how it bothers which users"; and not just focus on
> > users alone.
> >
> > And other than that, are there alternatives? Something we can do
> > better?
>
> We could consider reducing the feature set of portage and live
> with the "problems" that arise.  When I started using Gentoo a
> package could simply not go stable until all dependencies for all
> USE flags were also stable.  Masking USE flags was reserved to a
> short list of very special architecture depend special cases.
>
> [...]
Doesn't that block stabilization too aggressively?
 

> >> 2. Just a few days ago dev-lang/python-exec:2 became stable
> >> on amd64, but dev-python/python-exec:2 is still ~amd64.
> >> Just to be sure to not miss anything, I have put the latter
> >> into package.accept_keywords, and hell break loose:
> >
> > Hell indeed breaks loose if you mix stable and unstable; but note
> > that this package had an accident, see the related news item for
> > details.
>
> Do you mean stable and unstable in this case, or in general?
>
> [...]
General; but well, depends on how you mix things. A few packages won't
hurt, any larger than that and blockers and conflicts are all around
your ears...

> In general I share the sentiment.  The complexity of using
> portage has increased a lot lately.  Not only does it take long
> to find out why things suddenly go wrong after tree sync, also
> just the time until 'emerge -avUDN world' comes back with a
> proposal has grown to several minutes where it was few seconds
> when I started with Gentoo.

Yes, while I'm not completely sure as it gradually changed; I have the
feeling it was much shorter back in the days. Though it could be a bit
of a bias because the amount of packages I use has grown since as well.

> There has been a lot of effort to make revdep-rebuild unessecary,
> but now that it is mostly implemented, I don't know if it was
> worth the price.

emerge-time_current = emerge-time_previous + revdep-rebuild-time

is how it feels like for me; just a move of time, but I feel like it's
not the right one. Anyhow, there are parameters to disable the
subslots; just as well you can disable preserved-rebuild.

Maybe it does make more sense to run a longer process once after all
emerges, than opposed to have it run again on every single emerge.

> I spend more time now just reconfiguring
> keywords to update the system than I spent back in the old days
> where revdep would just fix things.

Not sure what the relation between keywords and revdep-rebuild is.

> If the answer is, that I
> should not mix arch and ~arch, then I'll not use Gentoo anymore.

We suggest you not to; there are enough people around to give support
for it regardless, I'm yet to come across a problem that can't be fixed.

Please don't take my words for final and consider other people's
opinion on this thread as well; I don't know the numbers, so it is hard
to tell what the general opinion on this is. Parts of this post are
therefore personal opinion or a personal view not necessarily matching
by opinion, other parts are quoted and/or might match the truth.

--
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : [hidden email]
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

signature.asc (501 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Tom Wijsman-2
In reply to this post by Rich Freeman
On Wed, 13 Nov 2013 08:37:51 -0500
Rich Freeman <[hidden email]> wrote:

> That said, your original email contained a few separate issues and
> they're probably best dealt with individually.

Just to set things straight: Note that these were different authors.

> We're not going to have a common solution for multilib, stable use
> masking, python-exec, and whatever other issues are lurking beneath
> the surface.

Not *yet*... :D

--
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : [hidden email]
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

signature.asc (501 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Martin Vaeth
In reply to this post by Tom Wijsman-2
Tom Wijsman <[hidden email]> wrote:
>> The new "features" use.stable.mask and package.use.stable.mask
>> have turned maintaining systems with mixed ARCH and ~ARCH keywords
>> into a nightmare:
>
> They are considered unsupported by many

We can make a vote, but I would be very surprised if there are
many stable users without any unstable package.

>> As I understand, it tries to solve a "social" issue
>> (that an ARCH user might set a USE-flag which eventually
>> pulls in an ~ARCH package) on a technical level
>> (by forcibly disabling the USE-flag for the user).
>
> That's one approach, it might also be used when a package can be
> stabilized but a certain of feature of the package cannot; eg.
> USE=3D"minimal" could be broken on a certain package because it removed a
> bit too much, thus it could be stabilized with USE=3D"-minimal" forced.

As I said, it is a technical means to solve a social issue:
Instead of communicating to the users that Gentoo does not consider
USE=minimal stable, one *forces* USE=-minimal behind their back, even
if they should have decided to select it explicitly.

> Sometimes problems can be resolved by better communication too; perhaps
> there are things we can inform the user about in pkg_postinst

This would be the reasonable case for the USE=minimal example
(maybe even already in pkg_setup). There are also other ways to
communicate this to the user; I_KNOW_WHAT_I_AM_DOING names etc.
is another one.

> improve Portage to let the user know of a particular forced USE flag.

If you just remove use.stable.mask, portage will tell exactly
this to the user: That some package needs to be unmasked for
a certain dependency to be satisfied.
Then it is up to the user to decide whether he wants unmasking
or setting the use-flag. Instead making the choice behind his
back (against the use-flag) is bad IMHO.

> That are indeed the two approaches, I don't see a problem with putting
> the version itself in accept_keywords or overriding the profile;

Putting a stable version into accept_keywords will (correctly) be
considered redundant by all tools which cleanup your accept_keyword:
It makes no sense, and using it only to work around the issues
which use.stable.mask introduced appears plainly false.

Overriding in the profile of e.g. use.stable.mask cannot be done
for a single package; you can only negate a whole entry. In fact,
if you want something here, you are more or less almost forced
to negate all entries of {package.,}use.stable.mask
This is why I ask whether the latter must be there in the first place.

In fact, checking what actually is there, you find:

1. Hundreds of packages with abi_x86_32
2. python_{single,}targets_*
3. 5-10 packages with USE-flags like "unstable" or "clang" for which
   it should be clear for any stable user that he does not want to
   activate it without knowing what he is doing resp. caring about
   the dependencies.
4. Nothing else.

So practically only stuff which already caused grief plus very few
normal stuff which better should be communicated if it is not
obvious anyway.

>> The really bad thing is that this is happening by magic
>> "behind the user's back"
>
> System upgrades have to be done carefully; so, the user skipping it is
> something we cannot account for

So just to get it right:
There was something introduced to avoid communication which does
something behind the user's back which in many cases can be just
the opposite that the user wanted. You expect the user to check
the output carefully so that he can recognize that somebody is
trying to trick him, and if he analyzes the profiles he can
eventually find out what it is.
Do you really consider this better than reporting to the user
that some testing package is pulled in by a USE-flag he had set?
(Which would be the effect without use.stable.mask)

>> Since there are many reasons why use-flags can appear in ()
>> he will probably not even conjecture that actually something
>> will not be working as he expected.
>
> Most of these reasons can be uniquely determined from the output

Any mask is displayed the same. But this is not the point:
stable.mask was introduced to avoid communication with the user
by trying to guess what he means (in contrast to: doing what he
writes in the config-files).
Now you expect him to read carefully output and even improve
output only that he can decide whether portage's guesses are
right or whether he has to work against his profile.

Such tools which try to be more clever than the user always
cause trouble. Please recall this mechanism which was used
previously to "guess" useflags based on installed packages
(I even forgot the name, since this was fortunately removed).
This is nothing compared to the magic which stable.mask forces,
since the latter cannot simply be overrided by configuring
package.use appropriately.

>> But suddenly portage spitted unexplainable dependency errors,
>> and I only expert users manually reading the profiles can
>> understand that the reason is that use.stable.mask requires
>> that stable versions need to be keyworded ~amd64
>> (or use.stable.mask has to be overridden in my profile).
>
> This confuses me; isn't the goal to have one multilib apprach or the
> other? Especially due to the blockers between them.

This discussion does not belong here; I decided for ABI_X86="32"
and expect portage to respect this decision and not disabling
it randomly for some packages just because I did not mark a
stable package testing.

Yes, I understand the mechanism and how to override it, but this
is all tricky and cumbersome: Essentially, you have to spend a
lot of time just to work against the stable.mask mechanism.
The intention was probably that this mechanism should be
helpful and simplify things, but it turns out that it does
just the opposite. So one should think it over and remove it
(or replace it by something better if one finds something).

>> 2. Just a few days ago dev-lang/python-exec:2 became stable
>> on amd64, but dev-python/python-exec:2 is still ~amd64.
>> Just to be sure to not miss anything, I have put the latter
>> into package.accept_keywords, and hell break loose:
>
> Hell indeed breaks loose if you mix stable and unstable

Again, one might do a vote, but I would be surprised if not
80-90% of the stable users also have unstable packages.

> but note that this package had an accident, see the related
> news item for details.

This is why it took me so long to find out; of course, I thought
that the problem is related with the news while it was actually
related with use.stable.mask

>> Portaqe wanted me to emerge the git version of
>> dev-lang/python-exec:0 and reemerge dozens of packages.
>
> This is a consequence of that hell; if you don't agree and this is due
> to the Portage tree, please show the dependency chain that causes this.

Please read, what I had written: I have explained why this happened.
It is because use.stable.mask horribly interacts with the dependency
chain: Use flags change out of nothing if you are forced to add an
unstable keyword somewhere.

>> I needed hours to find out what is the actual cause:
>> The package.accept_keywords caused the use.stable.mask of
>> python_targets_pypy2_0 und python_targets_python3_3 to become
>> ineffective for dev-python/python-exec,
>
> It doesn't matter, dev-python/python-exec installs no contents.

This is the point: You have an actual "useless" package which
due to use.stable.mask hinder dependencies from being corrrectly
resolved. And which even forces other packages to be rebuilt,
also because of changes only in use.stable.mask.

>> but of course it is still
>> effective for dev-lang/python-exec which is not be the case
>> if I unmask the git version of dev-lang/python-exec.
>
> Which one is not meant to do anyway; and from what you wrote, I don't
> think you intent to either.

Exactly, but this is the only solution which portage can
find; again, because use.stable.mask implicitly changes the
dependency chain.

>> This is completely confusing since nowhere the git-version
>> is marked differently than the non-git version.
>
> Marked in which way? One is stable, the other unkeyworded.

They provide exactly the same USE-flags, and whenever they
occur in profiles or dependencies they occur both or none.
So one has a hard time to guess why the git version satisfies
a dependency which is not satisfied by the stable version.
Of course, if one knows the answer (use.stable.mask), it is clear.
Instead of making things simpler for the user, use.stable.mask
causes problems to the user.
It causes much more problems than it solves: Actually the only
problem which it solves (or better: tries to solve) is a
communication problem...

> If it is due to the hell above, we warn users up front for that.

Most users mixing stable and unstable have valid reasons
for each case. Telling "we have told you not to do that"
only to keep a smart-ass mechanism which produces more
problems than it solves is a bad idea.

>> 3. As a side effect of 2., I realized that libreoffice and a dozen
>> further packages get forced a rebuild. So, if eventually
>> python-3.3 becomes stable and is removed from use.stable.mask, [...]
>
> That is to be expected on such stabilizations, I doubt they are
> unnecessary

Only because some package which I have not even installed changed
its stability, it should be necessary that I reemerge everything?
And even *if* I should have installed it, the change of its
stability would make it necessary to reemerge my world?
Things appear very strange in Gentoo land meanwhile...


Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Michał Górny-5
In reply to this post by Martin Vaeth
Dnia 2013-11-13, o godz. 10:28:02
Martin Vaeth <[hidden email]> napisał(a):

> As I understand, it tries to solve a "social" issue
> (that an ARCH user might set a USE-flag which eventually
> pulls in an ~ARCH package) on a technical level
> (by forcibly disabling the USE-flag for the user).
> Solving social problems by technical means is never a good
> idea:

Then you don't understand it.

The basic issue is that we couldn't stabilize a package without
stabilizing all of its optional dependencies. For example, consider
Python 3.3. In order to get proper testing of it on testing, we had to
enable support for it in packages. But when we introduced that support,
the package immediately gained a dep on python:3.3.

If the package was supposed to go stable, we would either have to
stabilize Python 3.3 (too early for that), wait for Python 3.3
(resulting in very long to infinite) or drop Python 3.3 support. There
were even cases when I had to revbump a package in order to keep
'limited' revision to stabilize and 'full' to keep testing.

Just to be clear -- this isn't only a social issue. This is a valid QA
concern that we had no distinction between 'flags that are fine on
stable' and 'flags that are not'. If some flags work and some do not,
resulting in 'unsatisfiable dep' errors, this is technical.

> While the former gives the stable user a clear message
> what is going wrong (after all, many stable users want
> some package which has not a stable version yet, so this
> should be something everybody should be able to handle),
> the latter hides causes and makes things almost unpredictable:

Answer yourself the following questions: does portage suggest removing
the flag in the case? Does portage in any way explain that a particular
dependency was pulled in due to USE flag?

It's easy to use words like 'clear message' when you want to prove your
point. And I have no idea what you mean by 'something everybody should
be able to handle', unless you assume that the main purpose of a stable
system is to turn it into mixed-keyword system.

And just in case: the proper course of action then is to *report
a bug*. And in the end, thanks to your glorified solution, we end up
with dozens or hundreds of 'CANTFIX' bugs.

> Even if the user has put the dependency into
> package.accept_keywords, he will not be able to use the
> USE-flag on his packages unless
> *he puts stable versions into package.accept_keywords*
> (or - simlarly worse - overrides the profile).

If you have a problem with USE flag being masked, you unmask the USE
flag. It is simple like this.

I don't agree with the whole idea of unmasking flags via playing with
accept_keywords. In my opinion, it's just a terrible bug or mis-design.
It's not something that should be encouraged, or even happen.

If you unmask testing package just to get testing keywords, you quickly
lose the point of having stable keywords. Putting just the stable
versions is a pointless nightmare.

Even worse than that, people with mixed systems get extra flags even if
they don't want them. And this is an easy way to have them end up with
half of the system on testing.

> The really bad thing is that this is happening by magic
> "behind the user's back", i.e. contradicting the user's setting
> of USE-flag and package.use: If the user does not carefully
> read emerge's -v output, he even does not get informed that
> the support for his unstable package is dropped from the
> stable package, despite he might have specified the corresponding
> USE-flag globally. Even worse, even when reading the output
> carefully, the user cannot understand the reason:
> Since there are many reasons why use-flags can appear in ()
> he will probably not even conjecture that actually something
> will not be working as he expected.
It's magic only because you did it wrong.

> Here are some other examples of negative effects happening
> just recently to me, ordered from not so severe to very bad:
>
> 1. For several reasons I always want the most current
> emul-linux-x86* libraries, so they are in package.accept_keywords.
> Due to global ABI_X86=32 (which I also want), this forced me
> of course to put several libraries to ~amd64 since only
> new version support this. Some of the libraries are actually
> stable, so I have removed them from package.accept_keywords.
> So far, so good.
> But suddenly portage spitted unexplainable dependency errors,
> and I only expert users manually reading the profiles can
> understand that the reason is that use.stable.mask requires
> that stable versions need to be keyworded ~amd64
> (or use.stable.mask has to be overridden in my profile).
Which wouldn't happen if package.accept_keywords didn't implicitly
unmask flags.

> 2. Just a few days ago dev-lang/python-exec:2 became stable
> on amd64, but dev-python/python-exec:2 is still ~amd64.

And this was a bug that would have been fixed if you cared reporting it
straight to us rather than kept is as an argument.

> 3. As a side effect of 2., I realized that libreoffice and a dozen
> further packages get forced a rebuild. So, if eventually
> python-3.3 becomes stable and is removed from use.stable.mask,
> all stable users will have to rebuild almost their whole world,
> because python-exec then has one more "dummy" USE-flag. The same
> will happen again if pypy-2.0 becomes stable.
> So a lot of unnecessary rebuilds happen to stable users because
> of {package.,}use.stable.mask which most of the developers
> (who are often ~amd64 users) do not realize.

I don't get this. A masked flag is equivalent to a disabled flag. What
would cause the rebuild then?

--
Best regards,
Michał Górny

signature.asc (985 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Duncan-42
In reply to this post by Rich Freeman
Rich Freeman posted on Wed, 13 Nov 2013 08:37:51 -0500 as excerpted:

> On Wed, Nov 13, 2013 at 8:25 AM, Thomas Kahle <[hidden email]> wrote:
>> On 11/13/2013 12:39 PM, Tom Wijsman wrote:
>>> On Wed, 13 Nov 2013 10:28:02 +0000 (UTC)
>>> Martin Vaeth <[hidden email]> wrote:
>>>
>>>> Hello.
>>>>
>>>> The new "features" use.stable.mask and package.use.stable.mask have
>>>> turned maintaining systems with mixed ARCH and ~ARCH keywords into a
>>>> nightmare:
>>>
>>> They are considered unsupported by many; so, going down that path you
>>> need to be acquainted with Portage enough to keep a consistent system.
>>
>> This argument has come up several times, but is it valid?
>
> Honestly, opinions vary on this one and I don't think it is a productive
> path to go down.  I also feel that being able to mix keywords is a big
> benefit of using Gentoo.  I'd rather focus on practical ways to make
> this easier rather than whether it is desirable.
>
> That said, there are always going to be situations where mixing keywords
> isn't practical.  You're not going to run stable chromium against ~arch
> v8, or mixed keywords between kdelibs and kwin, etc.

FWIW, I believe at least part of the confusion here is based on differing
definitions of "supported".

Mixing ~arch/arch keywords is "supported" in that gentoo makes it
possible, and in the generic case, relatively easy.

It's "unsupported" in that it's a not generally tested case:

[TL;DR folks can skip this whole post from here.  It's mostly
philosophical rambling attempting to backup the above claim.]

The all-~arch tree case is basic sanity tested by devs when the introduce
the package with ~arch keywords, and further by the many users (including
me) who run all ~arch systems and wouldn't have it any other way.  So
while there's a measure of risk in it, chances are others on ~arch are
running into any problems at about the same time, and bugs will be
reported and generally either fixed or a workaround (patch in a bug if
not yet in tree, or a bug comment saying it works use flag X turned off,
etc) will be available, within a relatively short period.

The all arch-stable tree is tested as well, by the various arch devs and
arch-testers before they stabilize a particular package with tests
against an otherwise fully stable tree, and further by the various users
running that.

A mixed stable/~arch installation, however, is an undefined and mostly
untested case, and by domain definition will remain so, simply because of
the combinational explosion in the number of possible variants.

(Note that an argument of "simpler" and mostly binary distros is that
gentoo is /already/ that way, due to the combinational explosion of all
the various USE flag possibilities -- already that's beyond testing every
combination possible, but gentooers learn to live with and accept that
level of risk and the possibility that they may have to change a USE flag
to get something to work because that combination wasn't tested and
breaks, from time to time.  But the combinational possibilities are at
least within reason there, all the combinational possibilities of stable
with unstable is and will remain unsupportable in practice if not in
name, at least if "supportable" is defined with any strictness at all.)

Thus, by defintion of the mixed stable/~arch domain, it's simply
impossible to test and thus to support in any sort of "this unstable
package tested and demonstrated to work with a random mix of stable/~arch
packages" way, at all!

So in that regard, a mixed stable/~arch system is and will remain
unsupported, simply because it's impossible /to/ support.

That notwithstanding, gentoo DOES make it possible, even "easy" (note the
scare-quotes), to run a mixed stable/~arch system.  And the /ability/ to
do so remains supported.  However, users should know that in doing so
they're running a combination that isn't commonly tested, indeed, that
CANNOT be fully tested (even more so than the USE flag thing in general
means for gentoo), and should understand and be willing to accept the
added risk and manual sysadmin care necessary as a result.

And part of accepting that means that yes, they might have some bugs
closed with, effectively, "if it hurts when you bang your head against
the wall, quit banging your head against the wall", INVALID, NOTABUG,
etc, along with the associated recommendation to stick with either fully
~arch or fully arch, at least for that package and its dependencies.

And similarly, there WILL be more headaches with stuff like
package.stable.use overrides, etc, because by domain definition, that
sort of additional manual maintenance burden and responsibility comes
with the mixed stable/~arch domain.

But whether that's considered "unsupported" or not depends on how strict
and individual's definition of "supported" actually is.  THIS is where
opinions differ, at least in my /own/ opinion, as explained above.

--
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman


Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Martin Vaeth
In reply to this post by Tom Wijsman-2
Tom Wijsman <[hidden email]> wrote:
>
> and I quote "quite a few bugs are closed as invalid when a mixed system
> is detected"

I think nobody is speaking against this possibility:
If it causes too much grief to the maintainer to list all blockers
explicitly, I think everybody can agree if he says that if it breaks
you keep the pieces.

However, it is a different thing to treat mixing generally like ricing
and to introduce mechanisms which - as my example hopefully show -
can be *expected* to break usability of a mixed systems:

There are valid reasons to not have a testing system (besides the
obvious desire for stability another one is too frequent updates),
but there are also many valid reasons why you need some package(s)
which pull in a lot of unstable dependencies.

I think the "typical" use case even is a mixed system, since
most systems will require some special applications (related to
the main task this system is supposed to do; perhaps development
of something which needs a lot of unstable packages) while
on the other hand this does not mean that *everything* in the
system should be testing and frequently updated.

BTW, another reason for mixing which was not mentioned yet is
users of multiple architectures (e.g. amd64 and x86) wanting to
have the same versions installed (e.g. to use the same tarballs);
although, as a rule, amd64 and x86 do not differ *much*, they usually
always differ in a few packages since some team always stabilizes first.

> Given that maintainers often test against test versions and that arch
> teams often test against stable versions, nobody really tests the rests
> of the situations [...]
>
> It's also very unrealistic to check every package for every bump

Again, I think nobody expects that there never will be any trouble
on a mixing system.  But it is one thing whether this trouble occurs
from time to time - when things are running badly - or whether the
package manager uses a mechanism which can be expected to break on
mixing system.

> Which policy? It does not say anything about "supported" either. I
> believe that its intention has always been to change it for some
> packages, not for wide sets of them; the former doesn't cause much
> problems, the latter is a recipe for disaster...

With use.stable.mask you get problems even with one
unstable package unless you verify manually that no
useflag related to this package is in use.stable.mask
And if it is, you are lost in the sense that you have to
eliminate the provided use.stable.mask.
So what has use.stable.mask achieved? It has forced to you
to eliminate the mechanism and nothing else.

>> We could consider reducing the feature set of portage and live
>> with the "problems" that arise.  When I started using Gentoo a
>> package could simply not go stable until all dependencies for all
>> USE flags were also stable.  Masking USE flags was reserved to a
>> short list of very special architecture depend special cases.
>>
>> [...]
>
> Doesn't that block stabilization too aggressively?

Maybe, maybe not. It is exactly this social/political question which
is attempted to be solved by technical means with use.stable.mask.
This is what does not work:
You cannot solve this question (whether to relax this policy;
are there any exceptions valid and if yes which one) by a technical
tool which forces a certain other decision onto the user.
It is just shifting the problem and causes additional headache.


Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Thomas Kahle
In reply to this post by Duncan-42
On 11/13/2013 03:30 PM, Duncan wrote:

> Rich Freeman posted on Wed, 13 Nov 2013 08:37:51 -0500 as excerpted:
>
>> On Wed, Nov 13, 2013 at 8:25 AM, Thomas Kahle <[hidden email]> wrote:
>>> On 11/13/2013 12:39 PM, Tom Wijsman wrote:
>>>> On Wed, 13 Nov 2013 10:28:02 +0000 (UTC)
>>>> Martin Vaeth <[hidden email]> wrote:
>>>>
>>>>> Hello.
>>>>>
>>>>> The new "features" use.stable.mask and package.use.stable.mask have
>>>>> turned maintaining systems with mixed ARCH and ~ARCH keywords into a
>>>>> nightmare:
>>>>
>>>> They are considered unsupported by many; so, going down that path you
>>>> need to be acquainted with Portage enough to keep a consistent system.
>>>
>>> This argument has come up several times, but is it valid?
>>
>> Honestly, opinions vary on this one and I don't think it is a productive
>> path to go down.  I also feel that being able to mix keywords is a big
>> benefit of using Gentoo.  I'd rather focus on practical ways to make
>> this easier rather than whether it is desirable.
>>
>> That said, there are always going to be situations where mixing keywords
>> isn't practical.  You're not going to run stable chromium against ~arch
>> v8, or mixed keywords between kdelibs and kwin, etc.
>
> FWIW, I believe at least part of the confusion here is based on differing
> definitions of "supported".
I agree.  Generally however, we should think Gentoo (or the open
source ecosystem) more bazaar, less cathedral.  Libraries have
interfaces, and they are supposed to be mixed and matched
according to the interface definitions.  We (Gentoo) should not
think of "Gentoo stable" as a fixed product like "iOS-7".  It has
come a long way, but philosophically I still think of Gentoo as a
kind of automated Linux-from-scratch (where you also mix and
match whatever you find on the Internets).

In the end it boils down to what we mean by "supported".  For me
"supported" does not mean "tested".  As you point out, testing
every combination forbids itself.  Supported for me means that
the argument "you mixed stable and unstable" is not per se valid.
There's a huge difference between

You mixed unstable firefox with stable gcc

and

You mixed unstable X server with stable protocols.

For me mixing the trees is supported in the sense that I would
apply rational judgement to bugs.  If they are of the second
type, it can be said in a polite way that we as Gentoo can't do
anything about this combination not working.

Cheers,
Thomas

--
Thomas Kahle


signature.asc (567 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Ian Stakenvicius-2
In reply to this post by Michał Górny-5
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 13/11/13 09:10 AM, Michał Górny wrote:

>>
>> 1. For several reasons I always want the most current
>> emul-linux-x86* libraries, so they are in
>> package.accept_keywords. Due to global ABI_X86=32 (which I also
>> want), this forced me of course to put several libraries to
>> ~amd64 since only new version support this. Some of the libraries
>> are actually stable, so I have removed them from
>> package.accept_keywords. So far, so good. But suddenly portage
>> spitted unexplainable dependency errors, and I only expert users
>> manually reading the profiles can understand that the reason is
>> that use.stable.mask requires that stable versions need to be
>> keyworded ~amd64 (or use.stable.mask has to be overridden in my
>> profile).
>
> Which wouldn't happen if package.accept_keywords didn't implicitly
> unmask flags.
>

(I haven't read this whole thread yet, but in case it hasn't been
mentioned:)

It's also worth pointing out that the whole reason why abi_x86_32 is
{package.,}use.stable.masked is because trying to manage the partial
transisition between emul-* and multilib-build dependencies on stable
or mixed-keyworded systems is a horrible headache at the moment, due
to those exact same unexplainable dependency errors.  Without
{package.,}use.stable.mask, all stable users would have to deal with
this *right now* on their systems.


Note also that setting ABI_X86=32 globally isn't how it's supposed to
be used; the point of this flag is for dependency resolution when a
particular package requires it (ie, top-level package depends on
app-cat/dep[abi_x86_32], portage --autounmask-write sets the necessary
changes to /etc/portage/package.use).  But that's neither here nor there.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iF4EAREIAAYFAlKDlI4ACgkQ2ugaI38ACPBopwD8CLqHS45H50Dg4Jnz5/JgpcKP
9BkjdulcBTGSIxyJX8sA/j2d+fojR0hCAJvsPsD24h90CWBvfhxnK824//aejoQi
=2vLL
-----END PGP SIGNATURE-----

Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Ian Stakenvicius-2
In reply to this post by Thomas Kahle
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 13/11/13 09:55 AM, Thomas Kahle wrote:

> On 11/13/2013 03:30 PM, Duncan wrote:
>> Rich Freeman posted on Wed, 13 Nov 2013 08:37:51 -0500 as
>> excerpted:
>>
>>> On Wed, Nov 13, 2013 at 8:25 AM, Thomas Kahle
>>> <[hidden email]> wrote:
>>>> On 11/13/2013 12:39 PM, Tom Wijsman wrote:
>>>>> On Wed, 13 Nov 2013 10:28:02 +0000 (UTC) Martin Vaeth
>>>>> <[hidden email]> wrote:
>>>>>
>>>>>> Hello.
>>>>>>
>>>>>> The new "features" use.stable.mask and
>>>>>> package.use.stable.mask have turned maintaining systems
>>>>>> with mixed ARCH and ~ARCH keywords into a nightmare:
>>>>>
>>>>> They are considered unsupported by many; so, going down
>>>>> that path you need to be acquainted with Portage enough to
>>>>> keep a consistent system.
>>>>
>>>> This argument has come up several times, but is it valid?
>>>
>>> Honestly, opinions vary on this one and I don't think it is a
>>> productive path to go down.  I also feel that being able to mix
>>> keywords is a big benefit of using Gentoo.  I'd rather focus on
>>> practical ways to make this easier rather than whether it is
>>> desirable.
>>>
>>> That said, there are always going to be situations where mixing
>>> keywords isn't practical.  You're not going to run stable
>>> chromium against ~arch v8, or mixed keywords between kdelibs
>>> and kwin, etc.
>>
>> FWIW, I believe at least part of the confusion here is based on
>> differing definitions of "supported".
>
> I agree.  Generally however, we should think Gentoo (or the open
> source ecosystem) more bazaar, less cathedral.  Libraries have
> interfaces, and they are supposed to be mixed and matched according
> to the interface definitions.  We (Gentoo) should not think of
> "Gentoo stable" as a fixed product like "iOS-7".  It has come a
> long way, but philosophically I still think of Gentoo as a kind of
> automated Linux-from-scratch (where you also mix and match whatever
> you find on the Internets).
>
> In the end it boils down to what we mean by "supported".  For me
> "supported" does not mean "tested".  As you point out, testing
> every combination forbids itself.  Supported for me means that the
> argument "you mixed stable and unstable" is not per se valid.
> There's a huge difference between
>
> You mixed unstable firefox with stable gcc
>
> and
>
> You mixed unstable X server with stable protocols.
>
> For me mixing the trees is supported in the sense that I would
> apply rational judgement to bugs.  If they are of the second type,
> it can be said in a polite way that we as Gentoo can't do anything
> about this combination not working.


The term "supported" is a rather overloaded term which tends to mean
different things in gentoo depending on the context that it is used
(and who's using it), for sure.  It's also not analogous to "working"
or "expected to work", at all, imo.

I wonder if it might be a good idea to have a discussion and reach
consensus on what the Gentoo (Developer) definition of "Supported"
should actually be, and document this somewhere so that ambiguity can
be officially removed.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iF4EAREIAAYFAlKDl9EACgkQ2ugaI38ACPABTgEApL3YpwDO1aN6SrJ08byJi7L4
q5I1zYPImbooAUkm3OAA/2gGn2m1ulw6UzoQNLEtVkEzXr83iwMUvHX4jaGhUUEo
=b+5F
-----END PGP SIGNATURE-----

Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Martin Vaeth
In reply to this post by Michał Górny-5
Michał Górny <[hidden email]> wrote:

>
>> As I understand, it tries to solve a "social" issue
>> (that an ARCH user might set a USE-flag which eventually
>> pulls in an ~ARCH package) on a technical level
>> (by forcibly disabling the USE-flag for the user).
>> Solving social problems by technical means is never a good
>> idea:
>
> Then you don't understand it.
>
> The basic issue is that we couldn't stabilize a package without
> stabilizing all of its optional dependencies.

Exactly: The social issue is that you have this rule fixed
without any exceptions and do not want to discuss about exceptions.

> There were even cases when I had to revbump a package in order to keep
> 'limited' revision to stabilize and 'full' to keep testing.

The social issue has to be solved and it has to be clarified
when packages are allowed to have USE-flags depending on unstable
packages which the user has to resolve when he wants them.

> Just to be clear -- this isn't only a social issue. This is a valid QA
> concern that we had no distinction between 'flags that are fine on
> stable' and 'flags that are not'. If some flags work and some do not,
> resulting in 'unsatisfiable dep' errors, this is technical.

No. The user has to be told that he *should* not use such certain
flags on a stable system. This can happen by portage reporting
a message that some USE-dependency is pulling in an unstable package,
but it can also happen in a different way.
The use.stable.mask "solution" is to not inform the user but just
decide behind his back that he cannot use the flag and enforce
this decision.
Instead, e.g. one can let portage report if some useflag described
in use.stable.mask needs to be disabled, or one can use some
"I_KNOW_WHAT_I_AM_DOING" name, or whatever. There are many ways
of reporting. But forcing a decision on the user without even
communicating reasonably why this decision was forced is very bad IMHO.

> Answer yourself the following questions: does portage suggest removing
> the flag in the case? Does portage in any way explain that a particular
> dependency was pulled in due to USE flag?

The output can easily be improved: It is not rocket
science to list "required by category/package[FLAG]" instead of
"required by category/package" or to display the relevant part
of the dependency string (something similar is done already for
REQUIRED_USE, so the logic probably already exists in portage).

> And just in case: the proper course of action then is to *report
> a bug*. And in the end, thanks to your glorified solution, we end up
> with dozens or hundreds of 'CANTFIX' bugs.

Yeah, it is much better to throw the users into dependency hell
in which they have no clue how to find out. Certainly they will
never report any bugs in this case when they can so easily solve
this by just going through all dependencies and all profiles
manually and understanding all details.

> If you have a problem with USE flag being masked, you unmask the USE
> flag. It is simple like this.

Yes, the sane way to deal with use.stable.mask for the user is to
undo manually everything which is set there.
But is it really a good feature, if the user essentially only
removes it?

> I don't agree with the whole idea of unmasking flags via playing with
> accept_keywords. In my opinion, it's just a terrible bug or mis-design.
> It's not something that should be encouraged, or even happen.

I completely agree.
But the only way to unmask it otherwise is to undo use.stable.mask.
Then why have use.stable.mask in the first place?
Moreover, for use.stable.mask this cannot happen on a per-package basis.

Perhaps one can introduce a better way to unmask use.stable.mask?

> Even worse than that, people with mixed systems get extra flags even if
> they don't want them. And this is an easy way to have them end up with
> half of the system on testing.

Do you mean by "extra flags" again those of use.package.mask?
Then isn't this again an argument against use.package.mask at all?

> It's magic only because you did it wrong.

No, it is magic because it makes a decision (pulling in ~amd64
or unsetting USE) without communicating appropriately with the user
and even without giving him a clean possibility to decide differently.
(Modifying the profile is more a hack than a clean possibility).

Moreover, once the thing is removed from use.stable.mask he gets
tricked a second time by suddenly a useflag popping up.
(OK, this time, he *does* get informed and has a clean possibility
to change, so this time his surprise is limited).

> Which wouldn't happen if package.accept_keywords didn't implicitly
> unmask flags.

Exactly. This is why it becomes a dependency hell.
But wouldn't removing this go against the purpose of use.stable.mask?

> I don't get this. A masked flag is equivalent to a disabled flag. What
> would cause the rebuild then?

All python versions are use.force'd in this package.
However, those which are use.stable.mask'ed are not use.force'd,
because use.stable.mask overrides use.force (which makes sense).
So if the use.stable.mask entry changes, the active USE-flags
in the package change automatically, and the user cannot do anything
against it (except modifying the profile).


Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Mike Gilbert-2
On Wed, Nov 13, 2013 at 10:23 AM, Martin Vaeth
<[hidden email]> wrote:

> Michał Górny <[hidden email]> wrote:
>>
>>> As I understand, it tries to solve a "social" issue
>>> (that an ARCH user might set a USE-flag which eventually
>>> pulls in an ~ARCH package) on a technical level
>>> (by forcibly disabling the USE-flag for the user).
>>> Solving social problems by technical means is never a good
>>> idea:
>>
>> Then you don't understand it.
>>
>> The basic issue is that we couldn't stabilize a package without
>> stabilizing all of its optional dependencies.
>
> Exactly: The social issue is that you have this rule fixed
> without any exceptions and do not want to discuss about exceptions.
>
>> There were even cases when I had to revbump a package in order to keep
>> 'limited' revision to stabilize and 'full' to keep testing.
>
> The social issue has to be solved and it has to be clarified
> when packages are allowed to have USE-flags depending on unstable
> packages which the user has to resolve when he wants them.
>
>> Just to be clear -- this isn't only a social issue. This is a valid QA
>> concern that we had no distinction between 'flags that are fine on
>> stable' and 'flags that are not'. If some flags work and some do not,
>> resulting in 'unsatisfiable dep' errors, this is technical.
>
> No. The user has to be told that he *should* not use such certain
> flags on a stable system. This can happen by portage reporting
> a message that some USE-dependency is pulling in an unstable package,
> but it can also happen in a different way.
> The use.stable.mask "solution" is to not inform the user but just
> decide behind his back that he cannot use the flag and enforce
> this decision.
> Instead, e.g. one can let portage report if some useflag described
> in use.stable.mask needs to be disabled, or one can use some
> "I_KNOW_WHAT_I_AM_DOING" name, or whatever. There are many ways
> of reporting. But forcing a decision on the user without even
> communicating reasonably why this decision was forced is very bad IMHO.

Let's talk about the development workflow we use for a minute:

Say I want to stabilize a package on a given architecture. One of the
first tests I run is to change KEYWORDS and run repoman. repoman will
check to ensure that all dependencies are at least visible with the
given set of KEYWORDS. This is a fairly important check, and it often
prevents major fuck ups.

If we were to remove use.stable.mask, I as a developer would require
some alternate way to declare a use flag unsupported on stable so that
repoman knows not to check the optional dependencies for it with
stable keywords.

This is not a completely social issue; there is a very real
technical/QA issue that needs to be addressed on the development side.
If you can figure that out, we can talk about changing the end-user
experience.

Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Kent Fredric
In reply to this post by Martin Vaeth
On 14 November 2013 04:23, Martin Vaeth
<[hidden email]> wrote:
> The use.stable.mask "solution" is to not inform the user but just
> decide behind his back that he cannot use the flag and enforce
> this decision.
> Instead, e.g. one can let portage report if some useflag described
> in use.stable.mask needs to be disabled, or one can use some
> "I_KNOW_WHAT_I_AM_DOING" name, or whatever. There are many ways
> of reporting. But forcing a decision on the user without even
> communicating reasonably why this decision was forced is very bad IMHO.


Ah, that explains why I hadn't thought I'd seen what this subject was
about, because I expected when I saw

   (-foo)

In emerge's overview output, that the flag "foo" was simply
hard-masked profile-wide, for "some reason".

If there was better indication that these flags are simply masked due
to a stability level difference, I'd have expected a different
indicator in the output.

Seems as this is the reality we have:

ebuild:  stable, unstable, hardmasked/experimental
useflag: stable, hardmasked/experimental

Seems logical we make useflags potentially be:

useflag: stable, unstable, hardmasked/experimental

"Somehow", or at least, if that is already the case, it needs to be
more obvious what is occurring, and what steps we should be taking if
we want to work around it.

Maybe IUSE can be extended in a future EAPI to have ~ , or we could
have a seperate variable to indicate useflag stability.

ideas:
   -foo   #  I don't want foo, but defer if stability differences make
that impossible.
  -~foo   #  I don't want foo, and I don't care if removing foo
reduces stability.
  -**foo # I don't care if foo is experimental , turn that shit off

or something.

At very least, emerge --pretend  should say something like

   ((~)-foo)

To indicate foo is turned off due to a stability mask, though I don't
know how to make it clear to end users what that means. Just showing
something different however is often enough to make interested parties
curious.

--
Kent

Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Michał Górny-5
In reply to this post by Martin Vaeth
Dnia 2013-11-13, o godz. 15:23:44
Martin Vaeth <[hidden email]> napisał(a):

> Michał Górny <[hidden email]> wrote:
> >
> >> As I understand, it tries to solve a "social" issue
> >> (that an ARCH user might set a USE-flag which eventually
> >> pulls in an ~ARCH package) on a technical level
> >> (by forcibly disabling the USE-flag for the user).
> >> Solving social problems by technical means is never a good
> >> idea:
> >
> > Then you don't understand it.
> >
> > The basic issue is that we couldn't stabilize a package without
> > stabilizing all of its optional dependencies.
>
> Exactly: The social issue is that you have this rule fixed
> without any exceptions and do not want to discuss about exceptions.
Did you ever do any serious package work, or are just discussing
theory?

Repoman is not a social tool. It's a technical dep checker, and if you
start allowing exceptions to the rules, its output will *soon* become
useless since you would have to filter out all the allowed exceptions
and hope they didn't hide a valid issue.

> > Just to be clear -- this isn't only a social issue. This is a valid QA
> > concern that we had no distinction between 'flags that are fine on
> > stable' and 'flags that are not'. If some flags work and some do not,
> > resulting in 'unsatisfiable dep' errors, this is technical.
>
> No. The user has to be told that he *should* not use such certain
> flags on a stable system. This can happen by portage reporting
> a message that some USE-dependency is pulling in an unstable package,
> but it can also happen in a different way.
> The use.stable.mask "solution" is to not inform the user but just
> decide behind his back that he cannot use the flag and enforce
> this decision.
Let's move this a few blocks back. So, on a regular stable system,
should portage report you every time that this new package version is
not tested and you shouldn't be using it? But it will be just
a suggestion, you need to manually choose the older version if you want
to follow it...

Do you see now what you are proposing?

> > Answer yourself the following questions: does portage suggest removing
> > the flag in the case? Does portage in any way explain that a particular
> > dependency was pulled in due to USE flag?
>
> The output can easily be improved: It is not rocket
> science to list "required by category/package[FLAG]" instead of
> "required by category/package" or to display the relevant part
> of the dependency string (something similar is done already for
> REQUIRED_USE, so the logic probably already exists in portage).

If it can easily be improved, then why didn't *you* improve it yet? Go
for it, then tell us how easy it was. Then we can consider what to do
next.

> > And just in case: the proper course of action then is to *report
> > a bug*. And in the end, thanks to your glorified solution, we end up
> > with dozens or hundreds of 'CANTFIX' bugs.
>
> Yeah, it is much better to throw the users into dependency hell
> in which they have no clue how to find out. Certainly they will
> never report any bugs in this case when they can so easily solve
> this by just going through all dependencies and all profiles
> manually and understanding all details.

We must be having different users. Because as far as I know, we have
users throwing quite-a-long emerge outputs from time to time.

Of course, there are ricers too. The people who are going to mess up
their systems and suggest others to do the same, just to prove they
solve any problem in the worst way possible.

> > If you have a problem with USE flag being masked, you unmask the USE
> > flag. It is simple like this.
>
> Yes, the sane way to deal with use.stable.mask for the user is to
> undo manually everything which is set there.
> But is it really a good feature, if the user essentially only
> removes it?

Are stable keywords a good idea if the user essentially only overrides
them with package.accept_keywords?

The point you are missing is that we actually have stable users who
essentially want *stable* system. Those people don't want to get
testing packages unless absolutely necessary, and they'd rather wait
for Python 3 to go stable than unmask the flag for the fun of it.

> > I don't agree with the whole idea of unmasking flags via playing with
> > accept_keywords. In my opinion, it's just a terrible bug or mis-design.
> > It's not something that should be encouraged, or even happen.
>
> I completely agree.
> But the only way to unmask it otherwise is to undo use.stable.mask.
> Then why have use.stable.mask in the first place?

And by using 'package.use' you 'undo' USE defaults in packages
and global flag defaults, and... why do we have USE flags at all? We're
just all undoing stuff anyway.

> Moreover, for use.stable.mask this cannot happen on a per-package basis.

It's called package.use.stable.mask.

> Perhaps one can introduce a better way to unmask use.stable.mask?

Like what?

> > Even worse than that, people with mixed systems get extra flags even if
> > they don't want them. And this is an easy way to have them end up with
> > half of the system on testing.
>
> Do you mean by "extra flags" again those of use.package.mask?
> Then isn't this again an argument against use.package.mask at all?

I mean stable-masked flags. So you get 2-in-1 with no clear indication
that you got an extra unmask.

> > I don't get this. A masked flag is equivalent to a disabled flag. What
> > would cause the rebuild then?
>
> All python versions are use.force'd in this package.
> However, those which are use.stable.mask'ed are not use.force'd,
> because use.stable.mask overrides use.force (which makes sense).
> So if the use.stable.mask entry changes, the active USE-flags
> in the package change automatically, and the user cannot do anything
> against it (except modifying the profile).

Then the user gets to rebuild python-exec which is a tiny C executable.
Reverse dependencies are not rebuilt.

--
Best regards,
Michał Górny

signature.asc (985 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Rich Freeman
In reply to this post by Ian Stakenvicius-2
On Wed, Nov 13, 2013 at 10:02 AM, Ian Stakenvicius <[hidden email]> wrote:
> It's also worth pointing out that the whole reason why abi_x86_32 is
> {package.,}use.stable.masked is because trying to manage the partial
> transisition between emul-* and multilib-build dependencies on stable
> or mixed-keyworded systems is a horrible headache at the moment, due
> to those exact same unexplainable dependency errors.  Without
> {package.,}use.stable.mask, all stable users would have to deal with
> this *right now* on their systems.

Indeed, by mixing keywords/etc Martin is basically having to deal with
this mess *right now.*

>
>
> Note also that setting ABI_X86=32 globally isn't how it's supposed to
> be used;

I never really viewed the new multilib stuff as being "ready for
primetime" anyway.  That isn't meant to reflect on its level of
quality/etc - just that it isn't completely rolled out and the
docs/etc aren't really designed for end-user consumption yet.

Users who take advantage of new features in these kinds of states are
going to run into problems.  That's just the cost of being on the
cutting edge.

Rich

Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Martin Vaeth
In reply to this post by Kent Fredric
Kent Fredric <[hidden email]> wrote:
>
> Maybe IUSE can be extended in a future EAPI to have ~

I like this "~foo" idea very much from the user's point of view:
You see clearly why useflags are disabled, and one could have a
"simple" mechanism to override it.

However, extending IUSE is not the correct way, because
you would need a matrix in every ebuild which cannot be
reasonably maintained: Every useflag can be masked or stable on
each architecture differently. So, supporting the currently
existing {package,}.use.mask in the profiles might make sense,
just the effect needs to be modified somewhat.

1. A tiny change in the display: ~foo instead of (-foo)
2. A more severe change in the way how to unmask it:

(a) ACCEPT_KEYWORD=~ARCH can unmask it (as currenttly).

(b) /etc/portage/package.accept_keywords does *not* unmask it
    (contrary to now); this would avoid the dependency hell.
    Instead:

(c1) The entry ~foo in /etc/portage/package.use unkeywords foo.
     Alternatively (or in addition):

(c2) The entry foo in /etc/portage/package.use.accept_keywords
     unmasks it; this could be even extended with architectures
     and '**' as the current accept_keywords.

(d) Of course, changing the profile (like overriding it in
    /etc/portage/profile) can unmask it, too (as currently),
    but this should not be encouraged.

If I see it correctly, this would not even require an EAPI bump,
since /etc/portage/package.* files are not subject to pms.

I hope that (b) is not hard to implement in portage, since
it means that portage has to keep track about the cause of
an unkeywording.


Reply | Threaded
Open this post in threaded view
|

Re: Please consider removing use.stable.mask and package.use.stable.mask

Martin Vaeth
In reply to this post by Michał Górny-5
Michał Górny <[hidden email]> wrote:
>
> Repoman is not a social tool. It's a technical dep checker, and if you
> start allowing exceptions to the rules

Repoman might still access use.stable.mask without having
portage force it on users.

The social conflict I mean is: You *want* that users do not
decide for ~ARCH but instead always against enabling the
USE-flag, so you make it technically very hard for them
to even recognize that a decision is being made for them.

The solution in the other posting with displaying ~foo
(and simple unmasking) gives the user the freedom back.

> Let's move this a few blocks back. So, on a regular stable system,
> should portage report you every time that this new package version is
> not tested and you shouldn't be using it?

I did not mean that portage should report *every* time:
I meant that portage report if a stable.masked USE-Flag is disabled.
So you make the decision to either disable the USE-Flag for this
package or to disable the stable.mask by whatever mechanism
(e.g. by package.use_accept_keywords or something similar
or by overriding the profile or something else).

However, if package.accept_keywords does no magic on package.use,
perhaps a clear indication in the output like ~foo is sufficient.
(Otherwise you might not even get to such an output due to
dependency hell).

> If it can easily be improved, then why didn't *you* improve it yet?

A lot of portage output might be improved if one is willing to
spend enough time. But this is not the topic of the discussion.
Please, do not build a strawman like ...

> Then we can consider what to do next.

... and then thinking about 9 other tasks (of course, all
must be implemented *by me*) before you are willing to
discuss the issue I wanted to discuss.

> The point you are missing is that we actually have stable users who
> essentially want *stable* system. Those people don't want to get
> testing packages unless absolutely necessary, and they'd rather wait
> for Python 3 to go stable than unmask the flag for the fun of it.

These users will not have PYTHON_TARGETS="python3_3",
so not forcing use.stable.mask on them makes no difference.
If they *should* have PYTHON_TARGETS="python3_3", either by
mistake or intentional, they will see that an unstable python is
pulled in, and they have to take actions.
It is better that they see this "mistake" than if just their
choice is tacitly undone.

>> Moreover, for use.stable.mask this cannot happen on a per-package basis.
>
> It's called package.use.stable.mask.

I was expecting that you cannot undo something in
package.use.stable.mask which was set in use.stable.mask
but I just tried, and you are right.

>> Perhaps one can introduce a better way to unmask use.stable.mask?
>
> Like what?

I thought about a new config-file, but after now knowing
that package.use.stable.mask can be used to undo use.stable.mask,
I am not so convinced whether this is really necessary.

>> Do you mean by "extra flags" again those of use.package.mask?
>> Then isn't this again an argument against use.package.mask at all?
>
> I mean stable-masked flags. So you get 2-in-1 with no clear indication
> that you got an extra unmask.

With the ~foo indication this would not happen.

> Then the user gets to rebuild python-exec which is a tiny C executable.
> Reverse dependencies are not rebuilt.

I could reproduce that libreoffice and some other consuments got [r]
when I marked all */python-exec as ~amd64 in package.acccept_keywords,
but it did not get reemerged when I did not do this.
Of course, it might be that something else is going on here, or it
might even be a bug in portage.


1234 ... 6