crossdev and multilib interference

classic Classic list List threaded Threaded
85 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: crossdev and multilib interference

Joshua Kinard-2
On 06/17/2014 08:30, hasufell wrote:

> Joshua Kinard:
>> On 06/16/2014 21:47, hasufell wrote:
>>> Joshua Kinard:
>>>>
>>>> How big of a patch would this change require to the existing crossdev ebuild?
>>>>
>>>
>>> Probably quite trivial, but since vapier said "bs" to that proposal
>>> (translates to "bullshit" I guess) I'll not put any work into that.
>>>
>>> So there we go. If you are cool, you can just say "bs", vanish and leave
>>> stable arch in a broken state.
>>>
>>> Not even QA cares. Great. I'll try to get it on the next council agenda
>>> then.
>>
>> So you just take your ball and go home then?  That's not how it works.
>>
>> Create the patch, and file it as a bug.  Then, raise awareness on the ML.
>> That's how development works.  If your patch is reasonable and doesn't break
>> things, odds are likely it'll push the other members of toolchain to
>> consider incorporating it.
>>
>> Equally using the Council as a hammer all the time doesn't work in the
>> long-term, either.  If you whip a patch up, however, then not only could you
>> raise this at the next council meeting, but additionally state you've gone
>> that extra mile and created a patch that addresses the problem.
>>
>> That's taking the ball and putting it into the goal.
>>
>
> No, that's not how opensource works. You don't work on things after
> "upstream" said "not interested".
>
> https://bugs.gentoo.org/show_bug.cgi?id=504824

"upstream" didn't say anywhere in that bug that they weren't interested.
They countered your reasoning with a technical argument.  QA even states
that you need to file separate bugs for the various build failures.  You
could set up a master TRACKER bug for these crossdev-related issues, and
then link in any existing bugs or create new ones tied to it, and that way,
you have things documented.

--
Joshua Kinard
Gentoo/MIPS
[hidden email]
4096R/D25D95E3 2011-03-28

"The past tempts us, the present confuses us, the future frightens us.  And
our lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: crossdev and multilib interference

Joshua Kinard-2
In reply to this post by Rich Freeman
On 06/17/2014 08:49, Rich Freeman wrote:
>
> Is there a list/etc for crossdev?  I'd think that the users and
> maintainers of crossdev collectively have the biggest vested interest
> in addressing these issues.  They're also the ones who can vouch for
> how big of a problem this is.

toolchain is the primary maintainer for crossdev.  I actually wrote the
original implementation of crossdev years ago (~2004?).  It was a crappy,
overly-complex bash script that didn't use anything portage related (that I
can remember) to install a cross-toolchain.  It worked for simple toolchains
on common arches, i.e., i686 -> mips64 for me to build kernels for the SGI
systems.  Mike rewrote it several years later to become the crossdev we know
and love today, integrating it with emerge and taking better advantage of
existing Gentoo capabilities.


> Is this having some kind of adverse impact on anybody outside of the
> crossdev community?  If the crossdev maintainers were pushing hundreds
> of packages to change to accommodate dubious design on their part I
> could see this being a distro-wide issue.  On the other hand, if this
> is an issue that only impacts crossdev users and maintainers, then I'd
> think the simplest solution would be let them sort it out.

I certainly haven't seen crossdev related issues on my systems.  Granted, I
tend to run rather simple setups, not doing things like i686-on-x86_64 or
even running X11.  From the earlier-quoted bug, it looked like some X11
package is one of several affected by the issue being highlighted here.

What I'd like to see is a list of all affected packages so we all can get a
sense of just how big the actual problem really is.  All I am hearing so far
are unsubstantiated claims of tree-wide breakage.  Knowing which packages
are broken allows other devs to look at things and maybe come to agreement
that crossdev is the source of the problem or perhaps another solution that
applies to all of them can be worked up.


> If somebody in the crossdev community does want to sort it out and the
> problem is package squatting, then that might be a valid reason to
> escalate.  In that case the cleanest solution is to have a crossdev
> project, have the interested devs step up, hold a vote for the lead,
> and then respect the lead's role in resolving the issue.  Nobody
> "owns" a package, but in general we should be careful about stepping
> in and overriding maintainers, especially if nobody is interested in
> stepping in to take the reins long-term.

I'm a member of toolchain, but that's mostly historical because I used to
play with a lot of the cross-compile stuff for MIPS and Sparc.  Mike and
Ryan are the two primaries in toolchain right now.  If they don't see a
problem with crossdev right now, then I do have to question just how big of
a problem this really is.

--
Joshua Kinard
Gentoo/MIPS
[hidden email]
4096R/D25D95E3 2011-03-28

"The past tempts us, the present confuses us, the future frightens us.  And
our lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: crossdev and multilib interference

Michał Górny-5
In reply to this post by Ian Stakenvicius-2
Dnia 2014-06-17, o godz. 09:25:32
Ian Stakenvicius <[hidden email]> napisał(a):

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 16/06/14 07:38 PM, Joshua Kinard wrote:
> > Can $PATH be configured via our existing eselect tool to
> > enable/disable the crossdev paths when needed?
> >
>
> Technically it could but not really ; PATH is an environment thing,
> AFAIK all eselect could do is trigger the addition or removal of
> entries in /etc/env.d/ , but after that happens one would still need
> to 'env-update && . /etc/profile' to get the changes.
>
> Similarly I don't think using an eselect tool to bring the crossdev
> tools into the default path (via symlinks) is a great idea either; yes
> it'd allow users to un-eselect the crossdev tools when errors occur,
> but the errors would still occur every time a user forgets to do this
> first.
>
> It would be easier to update PATH yourself manually in the shell, I
> expect; perhaps a quick utility could do that for you (maybe opening
> up a crossdev-ready subshell) so you don't have to remember the path.
+1. That's how sane tools work.

--
Best regards,
Michał Górny

signature.asc (968 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: crossdev and multilib interference

Joshua Kinard-2
In reply to this post by hasufell
On 06/17/2014 08:48, hasufell wrote:
> Joshua Kinard:
>>
>> Equally using the Council as a hammer all the time doesn't work in the
>> long-term, either.
>
> This is exactly the case where the council has to step in to solve
> global issues and those between projects (here it is embedded gentoo
> project and multilib project).

multilib needs a lot of things fixed.  Everyone is pretty much aware that we
don't have solid support for multilib, just enough hacks in place that it
seems to JustWork(TM), for now.  But I think this is a wider issue in a lot
of GNU packages anyways, as the design mentality for many of them assumed
binary distro installs w/ a single ABI selected.

I am not convinced that a lot of the packages in the tree are being broken
by crossdev, when they are probably making assumptions about the build
environment that aren't multilib-safe.  If that's the case, it's those build
systems that need patching, not hard-masking crossdev.

That all said, you still haven't put forth a really convincing argument that
people can agree on.  And if you can't convince normal devs, do you think
you can convince council members?  What if you're rebuffed at a council
meeting on this issue?  What do you do then?

Not everything is a nail that the council needs to whack with a hammer.
Sometimes, you need an impact driver, a blow torch, or a simple Robertson
square-drive screwdriver.

--
Joshua Kinard
Gentoo/MIPS
[hidden email]
4096R/D25D95E3 2011-03-28

"The past tempts us, the present confuses us, the future frightens us.  And
our lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: crossdev and multilib interference

Joshua Kinard-2
In reply to this post by Michał Górny-5
On 06/17/2014 10:22, Michał Górny wrote:

> Dnia 2014-06-17, o godz. 09:25:32
> Ian Stakenvicius <[hidden email]> napisał(a):
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> On 16/06/14 07:38 PM, Joshua Kinard wrote:
>>> Can $PATH be configured via our existing eselect tool to
>>> enable/disable the crossdev paths when needed?
>>>
>>
>> Technically it could but not really ; PATH is an environment thing,
>> AFAIK all eselect could do is trigger the addition or removal of
>> entries in /etc/env.d/ , but after that happens one would still need
>> to 'env-update && . /etc/profile' to get the changes.
>>
>> Similarly I don't think using an eselect tool to bring the crossdev
>> tools into the default path (via symlinks) is a great idea either; yes
>> it'd allow users to un-eselect the crossdev tools when errors occur,
>> but the errors would still occur every time a user forgets to do this
>> first.
>>
>> It would be easier to update PATH yourself manually in the shell, I
>> expect; perhaps a quick utility could do that for you (maybe opening
>> up a crossdev-ready subshell) so you don't have to remember the path.
>
> +1. That's how sane tools work.

+1 for providing a technical reason why my off-the-cuff idea won't work.
This is what we need to address this perceived problem: Ideas.  Weed out the
ones that don't work and figure out what can work, then apply it.

--
Joshua Kinard
Gentoo/MIPS
[hidden email]
4096R/D25D95E3 2011-03-28

"The past tempts us, the present confuses us, the future frightens us.  And
our lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: crossdev and multilib interference

hasufell
In reply to this post by Joshua Kinard-2
Joshua Kinard:
>
> "upstream" didn't say anywhere in that bug that they weren't interested.
> They countered your reasoning with a technical argument.  QA even states
> that you need to file separate bugs for the various build failures.  You
> could set up a master TRACKER bug for these crossdev-related issues, and
> then link in any existing bugs or create new ones tied to it, and that way,
> you have things documented.
>

I appreciate that you want to help, but I'm not sure how many times I
have to explain to you that the PATH idea was neglected by the embedded
gentoo project lead. Check the history of this thread, it starts here:
https://groups.google.com/d/msg/linux.gentoo.dev/KZykx1DAJyM/YCMVUt4CzjUJ

So again, I am not doing work that goes diametral to what the project
lead wants and I am not going to fork crossdev.


I have proposed numerous ways to communicate this problem to the user
without touching any of the precious toolchain/embedded packages. If no
one responds there, I'll just pick one and apply it.

Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: crossdev and multilib interference

Alexandre Rostovtsev-2
In reply to this post by Joshua Kinard-2
On Tue, 2014-06-17 at 10:17 -0400, Joshua Kinard wrote:
> What I'd like to see is a list of all affected packages so we all can get a
> sense of just how big the actual problem really is.  All I am hearing so far
> are unsubstantiated claims of tree-wide breakage.  Knowing which packages
> are broken allows other devs to look at things and maybe come to agreement
> that crossdev is the source of the problem or perhaps another solution that
> applies to all of them can be worked up.

All multilib packages that use pkgconfig, for one thing. (Which means almost
all multilib packages.) Because current crossdev versions blindly install their
/usr/bin/i686-pc-linux-gnu-pkg-config wrapper script, overwriting the binary
belonging to pkgconfig[abi_x86_32].

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: crossdev and multilib interference

Joshua Kinard-2
In reply to this post by hasufell
On 06/17/2014 10:38, hasufell wrote:

> Joshua Kinard:
>>
>> "upstream" didn't say anywhere in that bug that they weren't interested.
>> They countered your reasoning with a technical argument.  QA even states
>> that you need to file separate bugs for the various build failures.  You
>> could set up a master TRACKER bug for these crossdev-related issues, and
>> then link in any existing bugs or create new ones tied to it, and that way,
>> you have things documented.
>>
>
> I appreciate that you want to help, but I'm not sure how many times I
> have to explain to you that the PATH idea was neglected by the embedded
> gentoo project lead. Check the history of this thread, it starts here:
> https://groups.google.com/d/msg/linux.gentoo.dev/KZykx1DAJyM/YCMVUt4CzjUJ

I already have that thread in my client, so let me quote a few choice bits:


On 03/26/2014 01:17, Mike Frysinger wrote:

>> when you run `crossdev i686-pc-linux-gnu`, it owns that tuple.  that includes
>> i686-pc-linux-gnu-pkg-config.
>>
>> if we're going to have the multilib system lie and use a tuple that doesn't
>> actually exist, we either:
>>
>> (1) override all the vars so they point back to the real toolchain.
>> this doesn't scale when you consider helper tools like the legacy sdl-config
>> or the extended set of tools that binutils/gcc/etc... install.  it's mitigated
>> by the fact the set of vars in play most of the time is low.
>>
>> (2) use tuples with loaded vendor fields to reduce the chance of collisions.
>> e.g. having an ABI=amd64 system use i686-gentoo%multilib-linux-gnu instead of
>> i686-pc-linux-gnu would defeat any automatic path searches.

On 03/26/2014 22:41, Mike Frysinger wrote:

>>
>> as i pointed out elsewhere in this thread, the problem is that multilib relies
>> on automatic detection of the toolchain *failing* so that it falls back to the
>> native value.  in other words, when you run `./configure --host=i686-pc-linux-
>> gnu`, it tries to find e.g. i686-pc-linux-gnu-ar.  it doesn't exist so the
>> fallback is used (plain `ar`).  multilib is using these tuples so that the
>> standard checks (autoconf/eclasses/etc...) trigger in the right ways for the
>> cpu/os/userland combinations.
>>
>> since crossdev installs a full proper toolchain for the target, the one
>> multilib was using to lie now exists and its toolchain is used instead.

On 03/27/2014 02:41, Mike Frysinger wrote:
>>
>> pkg-config does need fixing in some way.  we already know this.  it's why the
>> multilib eclasses currently set PKG_CONFIG_XXX vars -- preciously so the
>> correct ABI dir is utilized.  and this breaks when using some build systems
>> (like scons) where the env gets blown away (although we also know scons
>> sucks).



> So again, I am not doing work that goes diametral to what the project
> lead wants and I am not going to fork crossdev.
>
> I have proposed numerous ways to communicate this problem to the user
> without touching any of the precious toolchain/embedded packages. If no
> one responds there, I'll just pick one and apply it.

And what I am trying to tell you is that making hardmask threats don't solve
the core problem.  You're threatening to to start a mask/unmask war that
probably won't end well for you.  Mike has, in all of the messages I have in
the thread, provided clear technical explanations for why crossdev operates
the way it does, and that it isn't the source of these problems.

Provide a technical counter-argument to that or propose a solution that
people can agree on and you're going to find people are a LOT more willing
to stand with you on fixing the perceived problem.

--
Joshua Kinard
Gentoo/MIPS
[hidden email]
4096R/D25D95E3 2011-03-28

"The past tempts us, the present confuses us, the future frightens us.  And
our lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: crossdev and multilib interference

Michał Górny-5
In reply to this post by Alexandre Rostovtsev-2
Dnia 2014-06-17, o godz. 10:56:41
Alexandre Rostovtsev <[hidden email]> napisał(a):

> On Tue, 2014-06-17 at 10:17 -0400, Joshua Kinard wrote:
> > What I'd like to see is a list of all affected packages so we all can get a
> > sense of just how big the actual problem really is.  All I am hearing so far
> > are unsubstantiated claims of tree-wide breakage.  Knowing which packages
> > are broken allows other devs to look at things and maybe come to agreement
> > that crossdev is the source of the problem or perhaps another solution that
> > applies to all of them can be worked up.
>
> All multilib packages that use pkgconfig, for one thing. (Which means almost
> all multilib packages.) Because current crossdev versions blindly install their
> /usr/bin/i686-pc-linux-gnu-pkg-config wrapper script, overwriting the binary
> belonging to pkgconfig[abi_x86_32].
Just to shed some more light on this before someone goes into wrong
conclusions.

The initial multilib code didn't use prefixed pkg-config binaries but
simple PKG_CONFIG_PATH override. However, crossdev installing
i686-pc-linux-gnu-pkg-config caused configure scripts to find and use
it instead of plain 'pkg-config', and since the wrapper dumbly
overrides PKG_CONFIG_PATH it broke our original solution.

We specifically made pkg-config packages install the prefixed binaries
to trigger collisions with crossdev -- so that people who have systems
broken with crossdev's i686-pc-linux-gnu-pkg-config would be more
likely informed there's something wrong with their systems.

--
Best regards,
Michał Górny

signature.asc (968 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: crossdev and multilib interference

hasufell
In reply to this post by Joshua Kinard-2
Joshua Kinard:
>> I have proposed numerous ways to communicate this problem to the user
>> without touching any of the precious toolchain/embedded packages. If no
>> one responds there, I'll just pick one and apply it.
>
> And what I am trying to tell you is that making hardmask threats don't solve
> the core problem.

You missed my response to rich then.

Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: crossdev and multilib interference

Joshua Kinard-2
In reply to this post by Alexandre Rostovtsev-2
On 06/17/2014 10:56, Alexandre Rostovtsev wrote:

> On Tue, 2014-06-17 at 10:17 -0400, Joshua Kinard wrote:
>> What I'd like to see is a list of all affected packages so we all can get a
>> sense of just how big the actual problem really is.  All I am hearing so far
>> are unsubstantiated claims of tree-wide breakage.  Knowing which packages
>> are broken allows other devs to look at things and maybe come to agreement
>> that crossdev is the source of the problem or perhaps another solution that
>> applies to all of them can be worked up.
>
> All multilib packages that use pkgconfig, for one thing. (Which means almost
> all multilib packages.) Because current crossdev versions blindly install their
> /usr/bin/i686-pc-linux-gnu-pkg-config wrapper script, overwriting the binary
> belonging to pkgconfig[abi_x86_32].

But how many packages is that?  Is there a way to filter and count the
packages in the tree that are both multilib-capable and rely on pkgconfig?
This still doesn't convey the scale of the perceived problem, and this is
why people are not really convinced that a problem exists and that crossdev
is the source of the problem.

I am intentionally playing the role of the outsider on this because I don't
use multilib yet.  Linux/MIPS kinda started the whole multilib thing anyways
w/ the o32/n32/n64 setup that got carried over from IRIX.  Later on, you had
x86_64 join the fray, which made the problem much more noticeable.  Convince
me that there is a problem and that crossdev is the source of that problem.
 Right now, it seems to me that the problem isn't limited to just one
package created by Gentoo, but it's just that a LOT of packages in the
open-source world still haven't updated their build systems to account for
multiple-ABI installs.

Brainstorm: Would making crossdev's installation of the ${CHOST}-pkg-config
wrapper optional solve the problem somewhat?  Perhaps as a USE flag that
multilib/pkgconfig packages can check in DEPEND and throw warnings about?
Do any of the other crossdev-installed ${CHOST}- prefixed scripts or
binaries installed in /usr/bin cause similar problems, or does everything
hinge on this one script?

--
Joshua Kinard
Gentoo/MIPS
[hidden email]
4096R/D25D95E3 2011-03-28

"The past tempts us, the present confuses us, the future frightens us.  And
our lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: crossdev and multilib interference

hasufell
In reply to this post by Joshua Kinard-2
Joshua Kinard:
> Provide a technical counter-argument to that or propose a solution that
> people can agree on and you're going to find people are a LOT more willing
> to stand with you on fixing the perceived problem.
>

I start to think here is some confusion going on. We already proposed
solutions and there was no agreement. There are no more possibilities,
except rewriting crossdev, doing non-trivial hackery on toolchain
eclasses or stashing the whole multilib idea.
There were some ideas about a few make.profile hacks, but they didn't
happen either.

I reopened this thread to make clear that we need to at least
communicate this problem to the user and I proposed more ways than just
hardmasking.

Also, I am not going to work on solutions that have no agreement
whatsoever. It does not make sense.

So, I have no idea what you expect me to do that did not already happen.

Reply | Threaded
Open this post in threaded view
|

Re: crossdev and multilib interference

Ryan Hill
In reply to this post by Joshua Kinard-2
On Tue, 17 Jun 2014 10:17:26 -0400
Joshua Kinard <[hidden email]> wrote:

> I'm a member of toolchain, but that's mostly historical because I used to
> play with a lot of the cross-compile stuff for MIPS and Sparc.  Mike and
> Ryan are the two primaries in toolchain right now.  If they don't see a
> problem with crossdev right now, then I do have to question just how big of
> a problem this really is.

Thanks for being a voice of reason in this thread.

I just wanted to say that I'm not involved with crossdev development so any
opinions I might be spewing are mine alone and shouldn't be considered an
"upstream" response.


--
Ryan Hill                        psn: dirtyepic_sk
   gcc-porting/toolchain/wxwidgets @ gentoo.org

47C3 6D62 4864 0E49 8E9E  7F92 ED38 BD49 957A 8463

signature.asc (486 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: crossdev and multilib interference

Alexandre Rostovtsev-2
In reply to this post by Joshua Kinard-2
On Tue, 2014-06-17 at 11:20 -0400, Joshua Kinard wrote:

> On 06/17/2014 10:56, Alexandre Rostovtsev wrote:
> > On Tue, 2014-06-17 at 10:17 -0400, Joshua Kinard wrote:
> >> What I'd like to see is a list of all affected packages so we all can get a
> >> sense of just how big the actual problem really is.  All I am hearing so far
> >> are unsubstantiated claims of tree-wide breakage.  Knowing which packages
> >> are broken allows other devs to look at things and maybe come to agreement
> >> that crossdev is the source of the problem or perhaps another solution that
> >> applies to all of them can be worked up.
> >
> > All multilib packages that use pkgconfig, for one thing. (Which means almost
> > all multilib packages.) Because current crossdev versions blindly install their
> > /usr/bin/i686-pc-linux-gnu-pkg-config wrapper script, overwriting the binary
> > belonging to pkgconfig[abi_x86_32].
>
> But how many packages is that?  Is there a way to filter and count the
> packages in the tree that are both multilib-capable and rely on pkgconfig?
> This still doesn't convey the scale of the perceived problem, and this is
> why people are not really convinced that a problem exists and that crossdev
> is the source of the problem.
$ eix --depend -z virtual/pkgconfig -a --use -z abi_x86_32 -a --use -z abi_x86_64 --only-names | wc -l
285

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: crossdev and multilib interference

Joshua Kinard-2
On 06/18/2014 01:08, Alexandre Rostovtsev wrote:
> eix --depend -z virtual/pkgconfig -a --use -z abi_x86_32 -a --use -z abi_x86_64 --only-names | wc -l

Interesting, I got 294 (probably from my local dev tree).  Close enough, thanks!



So, taking this count-packages script here:
http://dev.gentoo.org/~dberkholz/scripts/count-packages

And running it, I get the following information about the portage tree
(synced about ~2hrs ago):

    Statistics for /usr/portage:
    Total packages = 20436
    Total categories = 174
    Average packages per category = 117
    Standard deviation of packages per category = 203
    Suggested category size within (standard deviation / 2) of average: 16
to 218
    Split categories with more than 218 packages, and do not create
categories with fewer than 16 packages.


Using the total number of packages in the tree against either 285 or 294:
>>> print "%.2f" % ((float(294)/20436) * 100)
1.44
>>> print "%.2f" % ((float(285)/20436) * 100)
1.39

IOW, it looks like less than 1.5% of the tree contains multilib packages
that rely on pkgconfig that could be affected by crossdev installing the
${CHOST}-pkg-config link into PATH.

We all have different measurements of things, but in my book, that doesn't
even begin to qualify for "non-trivial tree-wide problem".  If we were
talking close to 5%-10% of packages affected, that to me would be worth some
serious discussion.

So since there isn't a pressing need to do something about it right now,
let's try to think up a proper way to address the problem for the longterm,
because the number of multilib/pkgconfig packages will likely increase in
the future.

Sound fair?

--
Joshua Kinard
Gentoo/MIPS
[hidden email]
4096R/D25D95E3 2011-03-28

"The past tempts us, the present confuses us, the future frightens us.  And
our lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: crossdev and multilib interference

Ian Stakenvicius-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 18/06/14 02:24 AM, Joshua Kinard wrote:
> IOW, it looks like less than 1.5% of the tree contains multilib
> packages that rely on pkgconfig that could be affected by crossdev
> installing the ${CHOST}-pkg-config link into PATH.
>
> We all have different measurements of things, but in my book, that
> doesn't even begin to qualify for "non-trivial tree-wide problem".
> If we were talking close to 5%-10% of packages affected, that to me
> would be worth some serious discussion.

Only one problem with that -- if much of that 1.5% is packages in
@system , then it doesn't matter how much of the tree it is in terms
of end-user impact.  It's all about how close these packages are to
the root.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iF4EAREIAAYFAlOhn6oACgkQ2ugaI38ACPB/+gEArO/jkT/TTbIfLKiJ1IkoPSFY
/hDasDudl9jXcHLhBtQA/1qGj7fbzLfb3Sg/ptfhe2YfRiXGxWnYdh5/uWjuJeFg
=SJeW
-----END PGP SIGNATURE-----

Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: crossdev and multilib interference

Peter Stuge-4
In reply to this post by Michał Górny-5
Alexandre Rostovtsev wrote:
> current crossdev versions blindly install their
> /usr/bin/i686-pc-linux-gnu-pkg-config wrapper script, overwriting
> the binary belonging to pkgconfig[abi_x86_32].

Thanks for getting to the point.

It seems silly for two toolchains (abi_x86_32 and a crossdev i686 toolchain)
to compete for one and the same name.

Is that really desirable?

Both toolchains integrate with emerge; either one can be used to emerge
packages, right? (Where the packages are emerged to doesn't matter.)

If Gentoo would like to support different toolchains for one CHOST
then there obviously needs to be some abstraction, and creating it
should take into account that things like autoconf, cmake, waf and
friends may not support any such abstraction for a long time to come.

An obvious low-finesse solution is to only allow one toolchain per CHOST.

That would formalize the current situation where only one toolchain or
the other works correctly, and would be fine as a first step.

If long-term Gentoo does indeed want to support both an abi_x86_32
and a crossdev-built i686-pc-linux-gnu toolchain, then obviously
someone interested in that will have to come up with how it will
work.

As has been said plenty of times already, doing anything on one side
that inconveniences the other side is not really acceptable. That
makes this problem an interesting one, in that there aren't really
any useful hacks to be done, only an actual proper solution that
benefits both multilib and crossdev users.

Only someone with an interest in doing something good for both those
groups will be able to find a solution.


Michał Górny wrote:
> Just to shed some more light on this before someone goes into wrong
> conclusions.

Thanks for adding this background info! It is very helpful.


> The initial multilib code didn't use prefixed pkg-config binaries
> but simple PKG_CONFIG_PATH override.

Do you mean PKG_CONFIG_LIBDIR?


> However, crossdev installing i686-pc-linux-gnu-pkg-config caused
> configure scripts to find and use it instead of plain 'pkg-config',

That sounds to me like autoconf's pkg-config support might also be
involved in the problem?


> and since the wrapper dumbly overrides PKG_CONFIG_PATH it broke our
> original solution.
>
> We specifically made pkg-config packages install the prefixed binaries
> to trigger collisions with crossdev

That strikes me as really unhelpful. You spent time on creating something
you were sure would cause a problem, instead of on something to *avoid*
the problem. Oh well.


> -- so that people who have systems broken with crossdev's
> i686-pc-linux-gnu-pkg-config would be more likely informed there's
> something wrong with their systems.

You seem to have made an arbitrary decision that crossdev's
pkg-config is at fault. I think we need to look at this in a larger
perspective as outlined above; take a step back.


I'm glad that we're now actually trying to understand what the real
problem is, instead of only treating the symptoms.


//Peter

Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: crossdev and multilib interference

Michał Górny-5
Dnia 2014-06-18, o godz. 17:24:03
Peter Stuge <[hidden email]> napisał(a):

> Alexandre Rostovtsev wrote:
> > current crossdev versions blindly install their
> > /usr/bin/i686-pc-linux-gnu-pkg-config wrapper script, overwriting
> > the binary belonging to pkgconfig[abi_x86_32].
>
> Thanks for getting to the point.
>
> It seems silly for two toolchains (abi_x86_32 and a crossdev i686 toolchain)
> to compete for one and the same name.
>
> Is that really desirable?
No. But the user specifies prefix for crossdev. As far as I'm
concerned, crossdev could refuse prefixes that are already used
in Gentoo profiles.

> Both toolchains integrate with emerge; either one can be used to emerge
> packages, right? (Where the packages are emerged to doesn't matter.)

No, 'integrating with emerge' is a bad word. Crossdev wraps emerge
and runs on top of it. Multilib runs 'below' emerge as run by eclass.

> If Gentoo would like to support different toolchains for one CHOST
> then there obviously needs to be some abstraction, and creating it
> should take into account that things like autoconf, cmake, waf and
> friends may not support any such abstraction for a long time to come.
>
> An obvious low-finesse solution is to only allow one toolchain per CHOST.

Obviously.

> That would formalize the current situation where only one toolchain or
> the other works correctly, and would be fine as a first step.
>
> If long-term Gentoo does indeed want to support both an abi_x86_32
> and a crossdev-built i686-pc-linux-gnu toolchain, then obviously
> someone interested in that will have to come up with how it will
> work.

I'd rather see multilib gcc installing 'i686-pc-linux-gnu' wrappers
that call the native gcc with proper '-m' option. As far as I know,
there's no real difference between the code emitted by native compiler
with -m32 and cross-compiler for i686. This would remove some need for
crossdev (and therefore some breakages) and allow our users to avoid
rebuilding the same thing twice.

> > The initial multilib code didn't use prefixed pkg-config binaries
> > but simple PKG_CONFIG_PATH override.
>
> Do you mean PKG_CONFIG_LIBDIR?

Both.

> > However, crossdev installing i686-pc-linux-gnu-pkg-config caused
> > configure scripts to find and use it instead of plain 'pkg-config',
>
> That sounds to me like autoconf's pkg-config support might also be
> involved in the problem?

The autoconf's AC_PROG_TOOL macros are proper and very useful. What's
the real problem is that crossdev installs some custom wrapper that
clobbers PKG_CONFIG_* where real pkg-config is expected.

> > and since the wrapper dumbly overrides PKG_CONFIG_PATH it broke our
> > original solution.
> >
> > We specifically made pkg-config packages install the prefixed binaries
> > to trigger collisions with crossdev
>
> That strikes me as really unhelpful. You spent time on creating something
> you were sure would cause a problem, instead of on something to *avoid*
> the problem. Oh well.

This was hitting our users and crossdev team didn't care. I did what I
could from our side to fix this, though this probably wasn't good
enough.

Do you prefer if I add some logic to detect i686 crossdev and bail out
completely, telling users to remove it? That wouldn't be very friendly
but at least they wouldn't hit random build failures anymore.

> > -- so that people who have systems broken with crossdev's
> > i686-pc-linux-gnu-pkg-config would be more likely informed there's
> > something wrong with their systems.
>
> You seem to have made an arbitrary decision that crossdev's
> pkg-config is at fault. I think we need to look at this in a larger
> perspective as outlined above; take a step back.

Because it is at fault. Build systems expect *-pkg-config to be
a binary compatible with pkg-config. When crossdev installs something
that does not respect PKG_CONFIG_{LIBDIR,PATH} and instead uses some
fancy directories, it is inevitable that it will break something.

And just to be clear, we didn't invent anything new. The breakage was
there for years, we were just first ones to mix all the ingredients.

The CHOSTs used by our stuff come from profiles, we didn't invent them.
The multilib_toolchain_setup that sets the build environment comes from
multilib.eclass and was already used for multilib in the past. The code
that causes tc-getBUILD_CC to return incorrect (crossdev) compiler
comes from toolchain-funcs.eclass.

If you look at it all, you'd notice that all code that results in those
failures is maintained by toolchain team, and so is crossdev. So what
can small gx86-multilib team to do to fix it?

--
Best regards,
Michał Górny

signature.asc (968 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: Re: crossdev and multilib interference

Steven J. Long
In reply to this post by Alexandre Rostovtsev-2
On Tue, Jun 17, 2014 at 10:56 -0400, Alexandre Rostovtsev wrote:
> All multilib packages that use pkgconfig, for one thing. (Which means almost
> all multilib packages.) Because current crossdev versions blindly install their
> /usr/bin/i686-pc-linux-gnu-pkg-config wrapper script, overwriting the binary
> belonging to pkgconfig[abi_x86_32].

Well I've spent far too long at crossdev code, only to see this and realise
you can simply hard-mask:
  cross-i686-pc-linux-gnu/{binutils,gcc,glibc,pkg-config}
in the amd64 multilib profile, unless I'm missing something. You'd be
hard-pushed to install a clashing crossdev with such a mask, afaict.

If you do want to change crossdev[1], afaict you're looking at interaction
between toolchain.eclass (and toolchain-binutils, and likely -funcs),
crossdev and gcc-config. I could well be wrong, as ever. This is just my
preliminary understanding, and maybe it'll provoke a more thorough
explanation.

Crossdev set_links() links the overlay to the tree. toolchain.eclass installs
the versioned links so the internal gcc-bin path is exposed in /usr/bin, in
toolchain_src_install(). For cross-toolchains, it only installs prefixed ones:
 dosym ${BINPATH}/${CTARGET}-${x} /usr/bin/${CTARGET}-${x}-${GCC_CONFIG_VER}

For a native compiler it first, in: "${D}"${BINPATH} does:
 ln -sf ${CTARGET}-${x} ${x}     # unversioned private link
 dosym ${BINPATH}/${CTARGET}-${x} /usr/bin/${x}-${GCC_CONFIG_VER}
 
..then the above. Note that the prefixed link is effectively a race as to
which was installed last, but only in the case of a clash.

gcc-config update_wrappers() makes links in the user's path, though all of
them are run via wrapper.c[2], aka /usr/lib/misc/gcc-config. This is a
multi-switch binary based on argv[0]/$0. However it specifically notes
that it's intended to be used with a PATH-based setup [3]:

/* Find the first file with suitable name in PATH.  The idea here is
 * that we do not want to bind ourselfs to something static like the
 * default profile, or some odd environment variable, but want to be
 * able to build something with a non default gcc by just tweaking
 * the PATH ... */

crossdev dirs are added to path after system ones in env.d; that's where
gcc-config gets the paths to use from.

crossdev uninstall() removes CTARGET-based links. Note that your native
machine CBUILD == CHOST, also has CTARGET = CHOST, so this would also
be a reason to block/mask according to arch.

I haven't reviewed wrapper.c as well as I'd like: some of it seems a bit
odd but I'd need more time. I did wonder why it doesn't just use
readlink(3P) til I saw that comment.

Anyhow, that all seems a bit pointless when you can just hardmask the
specific crossdev configuration that causes the problem on multilib
profiles, and the same mechanism can be applied elsewhere, as decided
by the arch-team (eg for: o32/n32/n64)

Although canadian-cross and ROOT-based toolchains are another matter.

Regards,
steveL

[1] ie stop installing symlinks in /usr/bin for CTARGET-gcc, as well as
env.d files, and just provide a sh wrapper in each PORTAGE_CONFIGROOT
(= cross-overlay) that can be sourced from /etc/profile or anywhere
else, to add the same settings instead as and when the user chooses.

The most you'd need is the ability to choose whether it takes precedence
over current PATH or not, and that's probably easiest with a variant
'source' ('.' in shell) so cross-builds do, and the profile one doesn't.

[2] http://git.overlays.gentoo.org/gitweb/?p=proj/gcc-config.git;a=blob;f=wrapper.c;hb=HEAD
[3] ll 125-129 h=b00e8187a6063e329049ab9a57023fe9113c598d;hb=HEAD
--
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

Reply | Threaded
Open this post in threaded view
|

Re: crossdev and multilib interference

Ian Stakenvicius-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 19/06/14 05:20 PM, Steven J. Long wrote:

> On Tue, Jun 17, 2014 at 10:56 -0400, Alexandre Rostovtsev wrote:
>> All multilib packages that use pkgconfig, for one thing. (Which
>> means almost all multilib packages.) Because current crossdev
>> versions blindly install their
>> /usr/bin/i686-pc-linux-gnu-pkg-config wrapper script, overwriting
>> the binary belonging to pkgconfig[abi_x86_32].
>
> Well I've spent far too long at crossdev code, only to see this and
> realise you can simply hard-mask:
> cross-i686-pc-linux-gnu/{binutils,gcc,glibc,pkg-config} in the
> amd64 multilib profile, unless I'm missing something. You'd be
> hard-pushed to install a clashing crossdev with such a mask,
> afaict.
>
> If you do want to change crossdev[1], afaict you're looking at
> interaction between toolchain.eclass (and toolchain-binutils, and
> likely -funcs), crossdev and gcc-config. I could well be wrong, as
> ever. This is just my preliminary understanding, and maybe it'll
> provoke a more thorough explanation. [ Snip! ]

Thank you for the explanation and research!

Tangental to this, mgorny wrote a little tool yesterday that might
work well as an alternative to crossdev for multilib systems.  It
simply wraps all the native toolchain calls with proper -m and
provides the new CTARGETs.

If anybody wants to take a look, this is the link he posted on -dev :

http://git.overlays.gentoo.org/gitweb/?p=dev/mgorny.git;a=blob;f=sys-devel/multilib-gcc-wrapper/multilib-gcc-wrapper-0.ebuild;h=3e304313c0812ffc7da79603e38979fc81a83081;hb=HEAD

Whether or not this suits everyone's needs for an i686 crossdev on
amd64 systems, i don't know.  Thoughts?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iF4EAREIAAYFAlOklU4ACgkQ2ugaI38ACPBBawD/aRIYx3q5RcSom87YWKCUf6SL
jXyavRbB1g5hP8S6B1wBAMBYvZABlKiZckvZYnIQfgsaNkuW1EoPGC5nwkq1Nl24
=3JNA
-----END PGP SIGNATURE-----

12345