crossdev and multilib interference

classic Classic list List threaded Threaded
85 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Re: crossdev and multilib interference

Greg Turner

On Fri, Jun 20, 2014 at 1:10 PM, Ian Stakenvicius <[hidden email]> wrote:
> Thoughts [about wrapping gcc, so that non-native multilib-build ABI's can finally return to a world where [[ ${GCC} != *' '* ]] ]?

TLDR: good idea, I'm strongly in favor of it.

A wrapper would fix horrors like the following, which, last I checked, was sort-of required* on ABI_X86="*32*" DEFAULT_ABI="amd64" systems with crossdev i686-pc-linux-gnu targets in ${PATH}:

--- /usr/portage/eclass/cmake-utils.eclass      2014-05-06 08:31:17.000000000 -0700
+++ /usr/local/portage/gogogmt/eclass/cmake-utils.eclass        2014-06-01 12:36:40.753904765 -0700
@@ -373,14 +373,115 @@ enable_cmake-utils_src_prepare() {
 }
+
+# retrieve a variable (i.e.: FOO) using i.e.: tc_getFOO,
+# unless the variable is already defined to a nonempty value,                                                                                                                                     
+# in which case, it is returned unaltered                                                                                                                                                         
+_cmake-utils_tc_get_if_needed() {                                                                                                                                                  
+       local v="$1" s                                                                                                                                                               
+       if [[ ${!v} ]] ; then 
+               s="${!v}" 
+       else     
+               s=$(tc-get${v}) 
+       fi    
+       echo "${s}"   
+}                                                                                                                                                                                    +                           +                                                                                                                                           
+# trim the preceeding and trailing whitespace from a string,
+# leaving inner spaces intact
+_cmake-utils_trimmed() {
+       local s="$*"
+       while [[ ${s:0:1} =~ [[:space:]] ]] ; do
+               s="${s:1}"
+       done
+       while [[ ${s:$((${#s}-1))} =~ [[:space:]] ]] ; do
+               s="${s:0:$((${#s}-1))}"
+       done
+       echo "${s}"
+}
+
+# given a string, retrieve the first word (consisting of non-space characters)
+# after trimming any leading whitespace.
+_cmake-utils_first_word_of() {
+       local s=$(_cmake-utils_trimmed "$*")
+       echo "${s%% *}"
+}
+
+# given a string, retrieve the rest of the string after the first word,
+# with any whitespace trimmed, retaining any whitespace between words
+# (except whitespace that was between the first and second word)
+_cmake-utils_subsequent_words_of() {
+       local s=$(_cmake-utils_trimmed "$*")
+       if [[ ${s} ]] ; then
+               s="${s#$(_cmake-utils_first_word_of "${s}")}"
+               echo "$(_cmake-utils_trimmed "${s}")"
+       else
+               echo ""
+       fi
+}
+
+# given a variable-name, retrieve it with _cmake-utils_tc_get_if_needed
+# and then from that retreive the first word
+_cmake-utils_get_first_word() {
+       local s=$(_cmake-utils_tc_get_if_needed "$1")
+       echo "$(_cmake-utils_first_word_of "${s}")"
+}
+
+# given a command-containing variable as the first argument,
+# uses _cmake-utils_tc_get_if_needed to retrieve the var, and then
+# returns type -P of the first word of it, ignoring rest.
+_cmake-utils_get_first_word_executable() {
+       local w=$(_cmake-utils_get_first_word "$1")
+       if [[ ${w} ]] ; then
+               echo "$(type -P "${w}")"
+       else
+               echo ""
+       fi
+}
+
+# fetches any additional bits of a variable not included in
+# _cmake-utils_get_first_word.  takes the variable name as an input.
+# May return an empty string.
+_cmake-utils_get_subsequent_words() {
+       local s=$(_cmake-utils_tc_get_if_needed "$1")
+       echo "$(_cmake-utils_subsequent_words_of "${s}")"
 }
 
 # @VARIABLE: mycmakeargs
@@ -433,17 +534,46 @@ enable_cmake-utils_src_configure() {
 
        # Prepare Gentoo override rules (set valid compiler, append CPPFLAGS etc.)
        local build_rules=${BUILD_DIR}/gentoo_rules.cmake
+
+       local _CMAKE_C_COMPILER=$(_cmake-utils_get_first_word_executable CC)
+       local _CCORPHANS=$(_cmake-utils_get_subsequent_words CC)
+       local _CMAKE_C_COMPILER_ARG1_COMMAND=
+       local _CMAKE_CXX_COMPILER=$(_cmake-utils_get_first_word_executable CXX)
+       local _CXXORPHANS=$(_cmake-utils_get_subsequent_words CXX)
+       local _CMAKE_CXX_COMPILER_ARG1_COMMAND=
+
+       # This seems fairly rediculous to me, but if C{,XX}ORPHANS is nonempty, then, in order to prevent
+       # duplication of the first argument, we must handle the first word specially
+       if [[ ${_CCORPHANS} ]] ; then
+               _CMAKE_C_COMPILER_ARG1_COMMAND="SET (CMAKE_C_COMPILER_ARG1 $(_cmake-utils_first_word_of "${_CCORPHANS}") CACHE STRING \"C compiler first argument\" FORCE)"
+               _CCORPHANS=$(_cmake-utils_subsequent_words_of "${_CCORPHANS}")
+       fi
+       if [[ ${_CXXORPHANS} ]] ; then
+               _CMAKE_CXX_COMPILER_ARG1_COMMAND="SET (CMAKE_CXX_COMPILER_ARG1 $(_cmake-utils_first_word_of "${_CXXORPHANS}") CACHE STRING \"C++ compiler first argument\" FORCE)"
+               _CXXORPHANS=$(_cmake-utils_subsequent_words_of "${_CXXORPHANS}")
+       fi
+
        cat > "${build_rules}" <<- _EOF_
-               SET (CMAKE_AR $(type -P $(tc-getAR)) CACHE FILEPATH "Archive manager" FORCE)
-               SET (CMAKE_ASM_COMPILE_OBJECT "<CMAKE_C_COMPILER> <DEFINES> ${CFLAGS} <FLAGS> -o <OBJECT> -c <SOURCE>" CACHE STRING "ASM compile command" FORCE)
-               SET (CMAKE_C_COMPILER $(type -P $(tc-getCC)) CACHE FILEPATH "C compiler" FORCE)
-               SET (CMAKE_C_COMPILE_OBJECT "<CMAKE_C_COMPILER> <DEFINES> ${CPPFLAGS} <FLAGS> -o <OBJECT> -c <SOURCE>" CACHE STRING "C compile command" FORCE)
-               SET (CMAKE_CXX_COMPILER $(type -P $(tc-getCXX)) CACHE FILEPATH "C++ compiler" FORCE)
-               SET (CMAKE_CXX_COMPILE_OBJECT "<CMAKE_CXX_COMPILER> <DEFINES> ${CPPFLAGS} <FLAGS> -o <OBJECT> -c <SOURCE>" CACHE STRING "C++ compile command" FORCE)
-               SET (CMAKE_RANLIB $(type -P $(tc-getRANLIB)) CACHE FILEPATH "Archive index generator" FORCE)
-               SET (PKG_CONFIG_EXECUTABLE $(type -P $(tc-getPKG_CONFIG)) CACHE FILEPATH "pkg-config executable" FORCE)
+               SET (CMAKE_C_COMPILER "${_CMAKE_C_COMPILER}" CACHE FILEPATH "C compiler" FORCE)
+               ${_CMAKE_C_COMPILER_ARG1_COMMAND}
+               SET (CMAKE_CXX_COMPILER "${_CMAKE_CXX_COMPILER}" CACHE FILEPATH "C++ compiler" FORCE)
+               ${_CMAKE_CXX_COMPILER_ARG1_COMMAND}
+               SET (CMAKE_AR "$(_cmake-utils_get_first_word_executable AR)" CACHE FILEPATH "Archive manager" FORCE)
+               SET (CMAKE_RANLIB "$(_cmake-utils_get_first_word_executable RANLIB)" CACHE FILEPATH "Archive index generator" FORCE)
+               SET (CMAKE_ASM_COMPILE_OBJECT "<CMAKE_C_COMPILER> ${_CCORPHANS}${_CCORPHANS:+ }<DEFINES> ${CFLAGS}${CFLAGS:+ }<FLAGS> -o <OBJECT> -c <SOURCE>" CACHE STRING "ASM compile command" FORCE)
+               SET (CMAKE_C_COMPILE_OBJECT "<CMAKE_C_COMPILER> ${_CCORPHANS}${_CCORPHANS:+ }<DEFINES> ${CFLAGS}${CFLAGS:+ }<FLAGS> -o <OBJECT> -c <SOURCE>" CACHE STRING "C compile command" FORCE)
+               SET (CMAKE_CXX_COMPILE_OBJECT "<CMAKE_CXX_COMPILER> ${_CXXORPHANS}${_CXXORPHANS:+ }<DEFINES> ${CXXFLAGS}${CXXFLAGS:+ }<FLAGS> -o <OBJECT> -c <SOURCE>" CACHE STRING "C++ compile command" FORCE)
+               SET (CMAKE_C_LINK_EXECUTABLE "<CMAKE_C_COMPILER> ${_CCORPHANS}${_CCORPHANS:+ }<FLAGS> ${LDFLAGS}${LDFLAGS:+ }<CMAKE_C_LINK_FLAGS> <LINK_FLAGS> <OBJECTS> -o <TARGET> <LINK_LIBRARIES>" CACHE STRING "C link command for executables" FORCE)
+               SET (CMAKE_C_CREATE_SHARED_LIBRARY "<CMAKE_C_COMPILER> ${_CCORPHANS}${_CCORPHANS:+ }<CMAKE_SHARED_LIBRARY_C_FLAGS> <LANGUAGE_COMPILE_FLAGS> <LINK_FLAGS> ${LDFLAGS}${LDFLAGS:+ }<CMAKE_SHARED_LIBRARY_CREATE_C_FLAGS> <CMAKE_SHARED_LIBRARY_SONAME_C_FLAG><TARGET_SONAME> -o <TARGET> <OBJECTS> <LINK_LIBRARIES>" CACHE STRING "C shared library link command" FORCE)
+               SET (CMAKE_C_CREATE_SHARED_MODULE "\${CMAKE_C_CREATE_SHARED_LIBRARY}" CACHE STRING "C shared module link command" FORCE)
+               SET (CMAKE_CXX_LINK_EXECUTABLE "<CMAKE_CXX_COMPILER> ${_CXXORPHANS}${_CXXORPHANS:+ }<FLAGS> ${LDFLAGS}${LDFLAGS:+ }<CMAKE_CXX_LINK_FLAGS> <LINK_FLAGS> <OBJECTS> -o <TARGET> <LINK_LIBRARIES>" CACHE STRING "C++ link command for executables" FORCE)
+               SET (CMAKE_CXX_CREATE_SHARED_LIBRARY "<CMAKE_CXX_COMPILER> ${_CXXORPHANS}${_CXXORPHANS:+ }<CMAKE_SHARED_LIBRARY_CXX_FLAGS> <LANGUAGE_COMPILE_FLAGS> <LINK_FLAGS> ${LDFLAGS}${LDFLAGS:+ }<CMAKE_SHARED_LIBRARY_CREATE_CXX_FLAGS> <CMAKE_SHARED_LIBRARY_SONAME_CXX_FLAG><TARGET_SONAME> -o <TARGET> <OBJECTS> <LINK_LIBRARIES>" CACHE STRING "C++ shared library link command" FORCE)
+               SET (CMAKE_CXX_CREATE_SHARED_MODULE "\${CMAKE_CXX_CREATE_SHARED_LIBRARY}" CACHE STRING "C++ shared module link command" FORCE)
+               SET (PKG_CONFIG_EXECUTABLE "$(_cmake-utils_get_first_word_executable PKG_CONFIG)" CACHE FILEPATH "pkg-config executable" FORCE)
        _EOF_
 
        has "${EAPI:-0}" 0 1 2 && ! use prefix && EPREFIX=

Perhaps a straw-man argument, since, the above code could be made more beautiful, and then it would not be so ugly :P

But, it's not just that cmake can't properly handle a C{C,XX} variables with spaces in them**. There are a bunch of similar places where we had to patch Makefiles and autotools to wedge GCC="foo bar" support in.

Without hyperbole, I can honestly say, it's a lame and inelegant hack to put an /argument/ into GCC.  -m${foo} is a CFLAG, not a compiler.  "Everyone" knows GCC is a variable that names an executable -- but in multilib-build, for some god-awful reason, we've made it into some other thing, a hybrid-psuedo-thing that really only makes sense in a shell script, and even then, only in an improperly coded shell script :).

Even if that rubicon was already crossed by ccache, distcc, etc, there is a reason none of them /require/ it.  It's just asking for trouble.  We'll just keep wasting time fixing subtle bugs like the above, because of it until we break down and fix it.  I'd lay pretty long odds that five years from now we are still clinging to it, should we decide to stick with the status quo for now.  So why not bite the bullet and save ourselves the hassle?

-gmt

* perhaps today the problem is mitigated, due to the same gcc-${pseudo-CHOST} executables now existing for upstream multilib-build pseudo-targets -- but, I'll bet, not solved.  Usually trouble only crops up for larger framework builds with lots of sub-dirs, like qt or mysql-server, and manifests as incorrect attempts to link native libraries in -- which the compiler might or might not manage to save you from at the last moment...  also note that patch is mocked up for easy reading -- it won't apply.  See my overlay for the real code.  Finally, sorry for the HTML --  I'm writing from the gmail ajax which obstinately refuses to allow plain-text inline patches.

** IIRC, it would be more precise to say something like "cmake's in-built hacks for the CC='exectutable argument' circumstance are subtly incompatible with our means of injecting the portage compiler values into cmake (or so things used to be; I suspect the problem still lurks to some extent), and that this ultimately manifests as confusing, intermittent build failures and correctly issued QA warnings in larger multilib-build-based ebuilds (or will in the near future; I actually wouldn't know as I run with the above patches :P), esp. in cases where a ${CC}-${CHOST} or ${CXX}-${CHOST} executables are to be found in ${PATH}."  But if any unlucky person is still reading, they can see fully why I didn't bother to say all that :P
Reply | Threaded
Open this post in threaded view
|

Re: crossdev and multilib interference

Michał Górny-5
Dnia 2014-06-21, o godz. 03:31:30
Greg Turner <[hidden email]> napisał(a):

> On Fri, Jun 20, 2014 at 1:10 PM, Ian Stakenvicius <[hidden email]> wrote:
> > Thoughts [about wrapping gcc, so that non-native multilib-build ABI's can
> finally return to a world where [[ ${GCC} != *' '* ]] ]?
>
> TLDR: good idea, I'm strongly in favor of it.
>
> A wrapper would fix horrors like the following, which, last I checked, was
> sort-of required* on ABI_X86="*32*" DEFAULT_ABI="amd64" systems with
> crossdev i686-pc-linux-gnu targets in ${PATH}:
>
> [...]
>
> But, it's not just that cmake can't properly handle a C{C,XX} variables
> with spaces in them**. There are a bunch of similar places where we had to
> patch Makefiles and autotools to wedge GCC="foo bar" support in.
I wasn't aware of that.

I honestly would love if toolchain cooperated with us on bringing this.
However, I'm a bit pessimistic about their willing. For one, we'd first
have to fix all multilib arches to use different CHOSTs for
different ABIs -- and so far, people prefer inventing some custom hacks
in the name of keeping status quo.

As far as multilib builds are concerned, I guess I could create
${T}/bin with proper wrappers. We do similar thing in python-r1 suite
already to make sure that all 'python[23]' calls behave correctly. Of
course, this will still require fixing CHOSTs and work only for
multilib ebuilds. But on the other hand, it would allow us to avoid
masking crossdev.

--
Best regards,
Michał Górny

signature.asc (968 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: crossdev and multilib interference

Steven J. Long
In reply to this post by Ian Stakenvicius-2
On Fri, Jun 20, 2014, Ian Stakenvicius wrote:

> On 19/06/14 05:20 PM, Steven J. Long wrote:
> > Well I've spent far too long at crossdev code, only to see this and
> > realise you can simply hard-mask:
> > cross-i686-pc-linux-gnu/{binutils,gcc,glibc,pkg-config} in the
> > amd64 multilib profile, unless I'm missing something. You'd be
> > hard-pushed to install a clashing crossdev with such a mask,
> > afaict.
> >
> > If you do want to change crossdev[1], afaict you're looking at
> > interaction between toolchain.eclass (and toolchain-binutils, and
> > likely -funcs), crossdev and gcc-config. I could well be wrong, as
> > ever. This is just my preliminary understanding, and maybe it'll
> > provoke a more thorough explanation. [ Snip! ]
>
> Thank you for the explanation and research!

YW :-) shove autotools.eclass (and supporting) in there too, including
multiprocessing which had a simply painful attempt at cleverness.
I mentioned it in #gentoo-embedded when i saw it, so hopefully it'll
be corrected soon. (bashpid() function: if you can't see why it's
painfully embarrassing, /join #bash and ask.)

> Tangental to this, mgorny wrote a little tool yesterday that might
> work well as an alternative to crossdev for multilib systems.  It
> simply wraps all the native toolchain calls with proper -m and
> provides the new CTARGETs.
>
> If anybody wants to take a look, this is the link he posted on -dev :
>
> <dead url>
>
> Whether or not this suits everyone's needs for an i686 crossdev on
> amd64 systems, i don't know.  Thoughts?

It's more layer upon layer, I'm afraid. Though that file's gone from
the repo, so I imagine it's already made its way to join the rest of
the misguided hackery that is multilib. Still, it's good that his
bash has come on, though he's still too tricksy for his own good;
likely trying to emulate Frysinger, another one who needs a nice
lie-down sometimes, instead of banging out more. Idiot house-"styles"
will do that to you, as will C++.

I don't know why we can't just mask cross-*/whatever in the multilib
profile, instead of more talk of "masking crossdev" with a heavy
heart.

Nor do know if that's been done already, as I just found that the
profiles directory Changelog stopped in 2013, for some reason, and
I don't have time to chase the files right now.

Sorry for delay, been away and then busy. I was hoping to read
something more than "mask crossdev" yet again, when I got back.

Regards,
igli.
--
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

Reply | Threaded
Open this post in threaded view
|

Re: crossdev and multilib interference

Ian Stakenvicius-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 01/08/14 05:05 AM, Steven J. Long wrote:

> On Fri, Jun 20, 2014, Ian Stakenvicius wrote:
>> On 19/06/14 05:20 PM, Steven J. Long wrote:
>>> Well I've spent far too long at crossdev code, only to see this
>>> and realise you can simply hard-mask:
>>> cross-i686-pc-linux-gnu/{binutils,gcc,glibc,pkg-config} in the
>>> amd64 multilib profile, unless I'm missing something. You'd be
>>>  hard-pushed to install a clashing crossdev with such a mask,
>>> afaict.
>>>
>>> If you do want to change crossdev[1], afaict you're looking at
>>> interaction between toolchain.eclass (and toolchain-binutils,
>>> and likely -funcs), crossdev and gcc-config. I could well be
>>> wrong, as ever. This is just my preliminary understanding, and
>>> maybe it'll provoke a more thorough explanation. [ Snip! ]
>>
>> Thank you for the explanation and research!
>
> YW :-) shove autotools.eclass (and supporting) in there too,
> including multiprocessing which had a simply painful attempt at
> cleverness. I mentioned it in #gentoo-embedded when i saw it, so
> hopefully it'll be corrected soon. (bashpid() function: if you
> can't see why it's painfully embarrassing, /join #bash and ask.)
>
>> Tangental to this, mgorny wrote a little tool yesterday that
>> might work well as an alternative to crossdev for multilib
>> systems.  It simply wraps all the native toolchain calls with
>> proper -m and provides the new CTARGETs.
>>
>> If anybody wants to take a look, this is the link he posted on
>> -dev :
>>
>> <dead url>
>>
>> Whether or not this suits everyone's needs for an i686 crossdev
>> on amd64 systems, i don't know.  Thoughts?
>
> It's more layer upon layer, I'm afraid. Though that file's gone
> from the repo, so I imagine it's already made its way to join the
> rest of the misguided hackery that is multilib. Still, it's good
> that his bash has come on, though he's still too tricksy for his
> own good; likely trying to emulate Frysinger, another one who needs
> a nice lie-down sometimes, instead of banging out more. Idiot
> house-"styles" will do that to you, as will C++.
>
> I don't know why we can't just mask cross-*/whatever in the
> multilib profile, instead of more talk of "masking crossdev" with a
> heavy heart.
>
> Nor do know if that's been done already, as I just found that the
> profiles directory Changelog stopped in 2013, for some reason, and
> I don't have time to chase the files right now.
>
> Sorry for delay, been away and then busy. I was hoping to read
> something more than "mask crossdev" yet again, when I got back.
>
> Regards, igli.
>

It's a package in the tree now, actually --
sys-devel/multilib-gcc-wrapper ; it wraps the native toolchain
appropriately for the different ABI's that this toolchain supports, so
that ie an i686-*-gcc call is implemented properly via the native gcc
instead of a crossdev one.  Note, it *also* works on multilib
crossdev, too!  I have a ppc crossdev installed on my amd64 host box
and multilib-gcc-wrapper allows it to build for both ppc32 and ppc64
ABIs just fine (presumably; i don't have a ppc32 or ppc64 native
system to actually do runtime tests).

Back to the comment on masking -- would a cross-emerge (which i think
uses the target's profile, right?) end up p.masking its own toolchain?
 If it did, would that actually break anything (i'm thinking no since
it's the crossdev that manages toolchain updates for the target rather
than cross-emerge)??  I agree that masks should be minimized, at most
masking the conflicting cross-* packages in a profile.  However if
this causes issues within cross-emerge too, then perhaps adjusting the
crossdev tool to warn or error would suffice when a target that will
conflict with the native toolchain is requested.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlPbpeAACgkQ2ugaI38ACPCGRwD+JnA2ACNizXn9ZYG0kiaoitwO
wqHqahuceDxeo8z+Ps4A/158v8pElxPFZ4oWgHfVbZ43eiJm/N65Zay1x3U/vo3w
=m7AR
-----END PGP SIGNATURE-----

Reply | Threaded
Open this post in threaded view
|

Re: Re: crossdev and multilib interference

Steven J. Long
On Fri, Aug 01, 2014 at 10:36AM, Ian Stakenvicius wrote:

> On 01/08/14 05:05 AM, Steven J. Long wrote:
> > I don't know why we can't just mask cross-*/whatever in the
> > multilib profile, instead of more talk of "masking crossdev" with a
> > heavy heart.
> >
> > Nor do know if that's been done already, as I just found that the
> > profiles directory Changelog stopped in 2013, for some reason, and
> > I don't have time to chase the files right now.
> >
> > Sorry for delay, been away and then busy. I was hoping to read
> > something more than "mask crossdev" yet again, when I got back.
> >
> Back to the comment on masking -- would a cross-emerge (which i think
> uses the target's profile, right?) end up p.masking its own toolchain?

No. The cross-* part is an overlay on CBUILD (ie the machine building the
software; for a native build this is the same as CHOST in make.conf.)

>  I agree that masks should be minimized, at most
> masking the conflicting cross-* packages in a profile.  However if
> this causes issues within cross-emerge too, then perhaps adjusting the
> crossdev tool to warn or error would suffice when a target that will
> conflict with the native toolchain is requested.

Well that should happen too: it's a trivial patch, which again has
already been discussed in #-embedded. Either vapier gets on and does
it now he's back, or someone else will, for an arch they care about. I
can't see him caring if it's correct; and after all the mask on the
"overlay" itself (which is on CBUILD, remember) is only possible due
to the separation inherent in the crossdev design.

Nowadays people like to call that "belt'n'braces" or something. When I
learnt to code it was called "common-sense." Discussing it wouldn't
even arise: it goes without saying.

Regards,
igli.
--
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

12345