Re: [SoC 2009] Ping ? Pong ! News and progress..

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [SoC 2009] Ping ? Pong ! News and progress..

Patrice Clement-2
Hi folks,

Sorry for my very long silence. I had to deal with some GSoC problems,
especially with the fact that my mentor gave up. I've got a new mentor now,
which is Luca Bartodo. Thanks to him for helping me!

It's been hard to remotivate myself working on the project: I've to admit I've
lost 1 week, stuck between waiting Davide's come back, asking myself a lot
of questions I had to ask to Davide, and, tons of problems related to my
project.

I've coded a very last ebuild to emerge /usr/share/ NetBSD directory content (I
needed it due to make rules which weren't installed, mk directory). I've also
made some
polishing to my ebuilds, and removed useless directories (like
/usr/share/man/html[1-9]). Read comments to check what I've done.

I'm currently creating the stage tarball. Here are the ebuilds which are
installed inside my ROOT directory:
http://projects.boulz.org/trac/gentoo-nbsd/attachment/wiki/WikiStart/netbsd-ROOT.txt

Of course, there are missing ebuilds. I'm currently facing PAM problems. I
remember Johan Hattne sending me an email about PAM patches especially for
NetBSD. Johan, if you read me, your help would be much appreciated !

Another problem I face is Kerberos libraries compilation. It fails to compile
with this error:
http://pastebin.org/8892

I also have to change two ./configure args which are "--without-et" and
"--without-ss" to pass configure script. According to ./configure help:
--with-system-et  use system compile_et and -lcom_err [default: build and
install a local version]
--with-system-ss  use system -lss and mk_cmds [private version]

I'd like to know how did Gentoo/FreeBSD managed with Kerberos librairies ? Even
by reading ebuilds, I can't really understand. Some help would be appreciated.

And finaly: no, I haven't boot my stage yet. I'll do so after resolving every
problems in order to have something usable. Currently, when I chroot in my ROOT
directory, I can use emerge, equery, .. and every other tools to manage
packages. Let's see:

patrice@soc-netbsd ~ & uname -a
NetBSD soc-netbsd 5.0 NetBSD 5.0 (GENERIC) #0: Sun Apr 26 18:50:08 UTC 2009
[hidden email]:/home/builds/ab/netbsd-5-0-RELEASE/i386/200904260229Z-obj/home/builds/ab/netbsd-5-0-REL
EASE/src/sys/arch/i386/compile/GENERIC i386
patrice@soc-netbsd ~ & pwd
/home/patrice
patrice@soc-netbsd ~ & su -
Password:
Terminal type is vt100.
soc-netbsd# chroot /gentoo-nbsd/ /bin/bash
soc-netbsd / # cd
soc-netbsd ~ # equery b /bin/mv
[ Searching for file(s) /bin/mv in *... ]
sys-netbsd/netbsd-bin-5.0 (/bin/mv)
soc-netbsd ~ # equery b /usr/bin/crontab
[ Searching for file(s) /usr/bin/crontab in *... ]
sys-process/vixie-cron-4.1-r11 (/usr/bin/crontab)
soc-netbsd ~ # equery b /usr/lib/libpuffs.so
[ Searching for file(s) /usr/lib/libpuffs.so in *... ]
sys-netbsd/netbsd-libs-5.0 (/usr/lib/libpuffs.so -> libpuffs.so.0.0)
soc-netbsd ~ # uname -a
NetBSD soc-netbsd 5.0 NetBSD 5.0 (GENERIC) #0: Sun Apr 26 18:50:08 UTC 2009
[hidden email]:/home/builds/ab/netbsd-5-0-RELEASE/i386/200904260229Z-obj/home/builds/ab/netbsd-5-0-REL
EASE/src/sys/arch/i386/compile/GENERIC i386
soc-netbsd ~ # equery l
[I--] [M~] app-admin/eselect-1.0.12 (0)
[I--] [M~] app-admin/eselect-news-20080320 (0)
[I--] [M~] app-admin/perl-cleaner-1.05 (0)
..

And so on ..

I think we are near the end :)

Cheers!

Patrice

Reply | Threaded
Open this post in threaded view
|

Re: [SoC 2009] Ping ? Pong ! News and progress..

Alexis Ballier-2
Hi,

> Sorry for my very long silence. I had to deal with some GSoC problems,
> especially with the fact that my mentor gave up. I've got a new
> mentor now, which is Luca Bartodo. Thanks to him for helping me!
>
> It's been hard to remotivate myself working on the project: I've to
> admit I've lost 1 week, stuck between waiting Davide's come back,
> asking myself a lot of questions I had to ask to Davide, and, tons of
> problems related to my project.

I didn't know you were left alone; if you need some help you can
contact me too; unfortunately I'm way too busy these days to be able to
be a full mentor but I'm interested in seeing a g/netbsd port so I can
give a high priority in helping you towards that goal :)


I've a couple of remarks wrt
http://projects.boulz.org/trac/gentoo-nbsd/attachment/wiki/WikiStart/netbsd-gentoo-ebuilds.txt

netbsd-bin
        | sh -> app-shells/bash = OK

This doesn't sound optimal; I'd rather keep netbsd sh which is probably
a POSIX compliant sh and leave bash as /bin/bash; while in practice this
is not always the case, all gentoo packages should be fine with /bin/sh
not being bash. Esp. openrc works very well with a more
lightweight /bin/sh than bash and thus makes booting faster.

netbsd-sbin
        | rcorder -> nothing = OK

rcorder is the thing that does a topological sort on init script
dependencies, right? In that case, with openrc it's very unlikely to be
needed but IMHO it would be nice to have it available (an useflag?).

netbsd-ubin
14 | m4 -> sys-devel/m4 = OK
15 | patch -> sys-devel/patch = OK
22 | sed -> sys-apps/sed = OK

Be careful there: all of them prepend the binaries names with a g when
userland isn't gnu. On FreeBSD, with portage we use the GNU versions
(see profiles/default/bsd/fbsd which aliases them) but BSD versions have
been preferred as, respectively, m4, patch and sed. This raises the
question: what userland do you want to have? IMHO it is better to have
a full BSD userland while stuff that require GNU tools can use the g
prefixed tools; with that idea you can see portage as a pkgsrc
replacement (or ports replacement in case of fbsd) instead of a
different system with a netbsd kernel & libc but with the rest of it
being GNU.

17 | yacc -> sys-devel/bison = OK

Here too: On FreeBSD we use yacc from freebsd-ubin. bison ebuild should
be fine with that and won't install the yacc symlink.

24 | uuidgen -> sys-apps/e2fsprogs = FAIL

Keep it there and don't worry about the e2fsprogs version. The e2fsprogs
version uses libuuid which shouldn't be needed on BSD. I'm not sure
about netbsd, but freebsd has uuid functions in the libc; they come
with a different interface but do the same thing. See eg
https://bugs.gentoo.org/show_bug.cgi?id=270003
Otherwise you'll need libuuid from util-linux, and good luck getting
something working on BSD out of it.

netbsd-usbin
42 | lp,lpc,lpd,lpq,lpr,lprm,lptest,pac->net-print/cups=FAIL
43 | postfix (MKPOSTFIX) -> mail-mta/postfix = FAIL
44 | wpa_* -> net-wireless/wpa_supplicant = FAIL

We should probably address these. What's the failure there?


netbsd-libs
49 | libss -> sys-libs/ss = OK

sys-libs/ss shouldn't be used, take the one from e2fsprogs-libs.

54 | libpcap -> net-libs/libpcap = FAIL

What's the problem here?

55 | libcom_err -> sys-libs/e2fsprogs-libs with
"--disable-tls" ./configure flag = OK

What's the problem with tls?


About the ebuilds in the overlay:

netbsd-kernel -> usually we name kernel as -sources: gentoo-sources,
freebsd-sources, etc.

netbsd-binutils, netbsd-gcc: what's the point of these wrt
sys-devel/{binutils,gcc} ?

wrt the way you disable bundled libs (eg in netbsd-libs): can't this be
done on a more generic way than patching, like the way freebsd.eclass
does with REMOVE_SUBDIRS ? Doesn't dummy-mk from bsdmk.eclass work on
netbsd makefiles?

> Another problem I face is Kerberos libraries compilation. It fails to
> compile with this error:
> http://pastebin.org/8892

Sounds like openssl related; what provides /usr/include/openssl/evp.h ?

> I also have to change two ./configure args which are "--without-et"
> and "--without-ss" to pass configure script. According to ./configure
> help: --with-system-et  use system compile_et and -lcom_err [default:
> build and install a local version]
> --with-system-ss  use system -lss and mk_cmds [private version]

Why do you need this? e2fsprogs-libs should provide libcomm_err and
libss and we should seriously avoid using bundled libs in packages.

> I'd like to know how did Gentoo/FreeBSD managed with Kerberos
> librairies ? Even by reading ebuilds, I can't really understand. Some
> help would be appreciated.

That I'm afraid I don't know :/

> I think we are near the end :)

Great!


Alexis.

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [SoC 2009] Ping ? Pong ! News and progress..

Alexis Ballier-2
In reply to this post by Patrice Clement-2
> I'm currently facing PAM
> problems. I remember Johan Hattne sending me an email about PAM
> patches especially for NetBSD. Johan, if you read me, your help would
> be much appreciated !

Just in case, you're using sys-auth/openpam, right?

Alexis.

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [SoC 2009] Ping ? Pong ! News and progress..

Patrice Clement-2
In reply to this post by Alexis Ballier-2
Friday 14 Aug 2009 13:15:07 (+0200), Alexis Ballier wrote :
> I didn't know you were left alone; if you need some help you can
> contact me too; unfortunately I'm way too busy these days to be able to
> be a full mentor but I'm interested in seeing a g/netbsd port so I can
> give a high priority in helping you towards that goal :)

Shit happens. But thanks for your help. :)
I'm gonna ping you on IRC.

>
> I've a couple of remarks wrt
> http://projects.boulz.org/trac/gentoo-nbsd/attachment/wiki/WikiStart/netbsd-gentoo-ebuilds.txt
>
> netbsd-bin
> | sh -> app-shells/bash = OK
>
> This doesn't sound optimal; I'd rather keep netbsd sh which is probably
> a POSIX compliant sh and leave bash as /bin/bash; while in practice this
> is not always the case, all gentoo packages should be fine with /bin/sh
> not being bash. Esp. openrc works very well with a more
> lightweight /bin/sh than bash and thus makes booting faster.

Ok, I'll correct that.

>
> netbsd-sbin
> | rcorder -> nothing = OK
>
> rcorder is the thing that does a topological sort on init script
> dependencies, right? In that case, with openrc it's very unlikely to be
> needed but IMHO it would be nice to have it available (an useflag?).

Using OpenRC makes rcorder useless: it's used in /etc/rc script.

>
> netbsd-ubin
> 14 | m4 -> sys-devel/m4 = OK
> 15 | patch -> sys-devel/patch = OK
> 22 | sed -> sys-apps/sed = OK
>
> Be careful there: all of them prepend the binaries names with a g when
> userland isn't gnu. On FreeBSD, with portage we use the GNU versions
> (see profiles/default/bsd/fbsd which aliases them) but BSD versions have
> been preferred as, respectively, m4, patch and sed. This raises the
> question: what userland do you want to have? IMHO it is better to have
> a full BSD userland while stuff that require GNU tools can use the g
> prefixed tools; with that idea you can see portage as a pkgsrc
> replacement (or ports replacement in case of fbsd) instead of a
> different system with a netbsd kernel & libc but with the rest of it
> being GNU.

I didn't know that, and it's great if "g" suffix is added.
But Portage needs some GNU tools like GNU sed: some scripts/patches
don't work if you don't use it. Same thing about awk:
I read ebuild.sh source code and "gawk" command is used. I'll correct this but
I'll still install them in my stage.

>
> 17 | yacc -> sys-devel/bison = OK
>
> Here too: On FreeBSD we use yacc from freebsd-ubin. bison ebuild should
> be fine with that and won't install the yacc symlink.

On my "correct" list.

>
> 24 | uuidgen -> sys-apps/e2fsprogs = FAIL
>
> Keep it there and don't worry about the e2fsprogs version. The e2fsprogs
> version uses libuuid which shouldn't be needed on BSD. I'm not sure
> about netbsd, but freebsd has uuid functions in the libc; they come
> with a different interface but do the same thing. See eg
> https://bugs.gentoo.org/show_bug.cgi?id=270003
> Otherwise you'll need libuuid from util-linux, and good luck getting
> something working on BSD out of it.

Yes, NetBSD has also his own version of uuidgen and libuuid. On my list.

>
> netbsd-usbin
> 42 | lp,lpc,lpd,lpq,lpr,lprm,lptest,pac->net-print/cups=FAIL
> 43 | postfix (MKPOSTFIX) -> mail-mta/postfix = FAIL
> 44 | wpa_* -> net-wireless/wpa_supplicant = FAIL
>
> We should probably address these. What's the failure there?
>

*

> netbsd-libs
> 49 | libss -> sys-libs/ss = OK
>
> sys-libs/ss shouldn't be used, take the one from e2fsprogs-libs.
>
> 54 | libpcap -> net-libs/libpcap = FAIL
>
> What's the problem here?
>

*

> 55 | libcom_err -> sys-libs/e2fsprogs-libs with
> "--disable-tls" ./configure flag = OK
>
> What's the problem with tls?
>

*,
which means: we should talk about this on IRC. I'll give you errors logs. It's
mainly compilation errors.

>
> About the ebuilds in the overlay:
>
> netbsd-kernel -> usually we name kernel as -sources: gentoo-sources,
> freebsd-sources, etc.

I'm gonna rename it.

>
> netbsd-binutils, netbsd-gcc: what's the point of these wrt
> sys-devel/{binutils,gcc} ?
>

NetBSD devs seem to have patched GCC sources to suit to NetBSD. I maybe writing
a big mistake. But I have tried every GCC's ebuilds and none of them have
worked. Even by downloading GCC sources on gcc.gnu.org website, ./configure and
make, it simply doesn't work. That's why I've created an ebuild..

>
> wrt the way you disable bundled libs (eg in netbsd-libs): can't this be
> done on a more generic way than patching, like the way freebsd.eclass
> does with REMOVE_SUBDIRS ? Doesn't dummy-mk from bsdmk.eclass work on
> netbsd makefiles?
>

Yes, it should work. In fact, it's because I haven't yes packaged my ebuilds
that I've used patches. To compile and install one of my ebuilds, you'll need
to download sources via CVS. My ebuilds are just a frontend to /usr/src. I know
it's not the right way to do: Davide seemed to be fine with that, but Luca
explained me how to proceed. For the moment, I'll stay with this configuration
because it "works". But, of course, I'll correct this after GSoC.

>
> > Another problem I face is Kerberos libraries compilation. It fails to
> > compile with this error:
> > http://pastebin.org/8892
>
> Sounds like openssl related; what provides /usr/include/openssl/evp.h ?
>

~ grep "EVP_PKEY_decrypt" /usr/include/openssl/evp.h
int EVP_PKEY_decrypt(EVP_PKEY_CTX *ctx, unsigned char *out, size_t *outlen, const unsigned char *in, size_t inlen);

> > I also have to change two ./configure args which are "--without-et"
> > and "--without-ss" to pass configure script. According to ./configure
> > help: --with-system-et  use system compile_et and -lcom_err [default:
> > build and install a local version]
> > --with-system-ss  use system -lss and mk_cmds [private version]
>
> Why do you need this? e2fsprogs-libs should provide libcomm_err and
> libss and we should seriously avoid using bundled libs in packages.
>

I've tried to compile Kerberos 5 1.7 this morning and it works without changing
any ./configure flags. The only error comes from PKINIT: it is an extension
to Kerberos that adds public key cryptography. Do we really need it ?
Thus, if I remove it from Makefile, compilation works fine.

>
>> I'm currently facing PAM
>> problems. I remember Johan Hattne sending me an email about PAM
>> patches especially for NetBSD. Johan, if you read me, your help would
>> be much appreciated !
>
> Just in case, you're using sys-auth/openpam, right?
>
> Alexis.
>

Yes. The problem comes from "passwd" program which segfaults. I need to dig on
this more. I can provide a .core if you want.

Anyway, thanks again for your help. :)

Cheers!

Patrice

Reply | Threaded
Open this post in threaded view
|

Re: [SoC 2009] Ping ? Pong ! News and progress..

Johan Hattne-2
In reply to this post by Patrice Clement-2
Dear all;

I did attempt to get NetBSD running with Gentoo on a spare Alpha I had
lying around a couple of years ago.  I have since then moved to where
there's 120V in the wall sockets, and the cheap PSU I used to house the
board cannot deal with it.  I still haven't found the time to buy a new
power supply, and hence the alpha is currently serving as a door stop.


On 08/14/09 06:58, Alexis Ballier wrote:

> On 08/13/09 19:56, Patrice Clement wrote:
>
>> I'm currently facing PAM problems. I remember Johan Hattne sending
>> me an email about PAM patches especially for NetBSD. Johan, if you
>> read me, your help would be much appreciated !
>
> Just in case, you're using sys-auth/openpam, right?

I seem to recall the NetBSD people introduced new functionality into
OpenPAM without which NetBSD won't work.  I've haven't been following
NetBSD or OpenPAM very closely since I gave up on my alpha, so I'm not
sure how much help my old patch would be.  But you can make your own!
Just make a diff between the sources of sys-auth/openpam and
corresponding sources in the NetBSD tree.


On 08/14/09 06:58, Alexis Ballier wrote:

> On 08/13/09 19:56, Patrice Clement wrote:
>
>> Another problem I face is Kerberos libraries compilation. It fails to
>> compile with this error:
>> http://pastebin.org/8892
>
> Sounds like openssl related; what provides /usr/include/openssl/evp.h

I vaguely recall having problems here as well.  As Alexis points out, I
think it turned out to be related to OpenSSL.  It may be worth checking
_very_ carefully that your openssl build went fine.  As I remember it,
it all seemed to compile OK, but due to something being misdetected or
something in one of those early perl scripts a lot of it was actually
garbage.

If you really want, I'll try to get the alpha running again.  Maybe you
could help me get it to boot to multiuser?  I never did figure out how
to get the rc system and baselayout and whatnot to agree with NetBSD (I
particularly remember that there was no fuser on NetBSD and porting it
from FreeBSD was not trivial).

// Cheers; Johan

Reply | Threaded
Open this post in threaded view
|

Re: [SoC 2009] Ping ? Pong ! News and progress..

Javier Villavicencio-2
In reply to this post by Patrice Clement-2
Patrice Clement wrote:

>> netbsd-ubin
>> 14 | m4 -> sys-devel/m4 = OK
>> 15 | patch -> sys-devel/patch = OK
>> 22 | sed -> sys-apps/sed = OK
>>
>> Be careful there: all of them prepend the binaries names with a g when
>> userland isn't gnu. On FreeBSD, with portage we use the GNU versions
>> (see profiles/default/bsd/fbsd which aliases them) but BSD versions have
>> been preferred as, respectively, m4, patch and sed. This raises the
>> question: what userland do you want to have? IMHO it is better to have
>> a full BSD userland while stuff that require GNU tools can use the g
>> prefixed tools; with that idea you can see portage as a pkgsrc
>> replacement (or ports replacement in case of fbsd) instead of a
>> different system with a netbsd kernel & libc but with the rest of it
>> being GNU.
>
> I didn't know that, and it's great if "g" suffix is added.
> But Portage needs some GNU tools like GNU sed: some scripts/patches
> don't work if you don't use it. Same thing about awk:
> I read ebuild.sh source code and "gawk" command is used. I'll correct this but
> I'll still install them in my stage.
>

Basically, keep NetBSD versions of those utilities to get NetBSD itself
built successfully, then alias the GNU versions for portage to compile
ebuild successfully.

>> netbsd-binutils, netbsd-gcc: what's the point of these wrt
>> sys-devel/{binutils,gcc} ?
>>
>
> NetBSD devs seem to have patched GCC sources to suit to NetBSD. I maybe writing
> a big mistake. But I have tried every GCC's ebuilds and none of them have
> worked. Even by downloading GCC sources on gcc.gnu.org website, ./configure and
> make, it simply doesn't work. That's why I've created an ebuild..
>

Ahh good old times. FreeBSD devs did also patch GCC sources to work on
FreeBSD. See http://bugs.gentoo.org/192403 for what it takes to get "our
toolchain" to behave properly on FreeBSD. And that's why I asked some
time ago if you managed to compile the kernel with a gentoo toolchain,
it's a big pain in the back :D.

There might be many things patched, however those in the gcc spec are
key to get a working gcc, and lucky us, the specs don't vary too much,
so you can try a diff between NetBSD gcc specs versus GNU gcc specs to
spot what's missing/required.

From there you can start a "copy" of your sys-devel/gcc ebuild of
choice, and add patches as required, once you get a somewhat working
version the patches can be tested/reviewed by the GNU devs to iron out
what's not required, what's correct, and what's wrong.

And here is where the legal BSD vs GPL battle starts, cause you can't
add BSD copyrighted code into GPL code :D, fun fun.

As a last note, beware that some of the ebuilds in portage may not
compile correctly or at all if they don't use a GNU (gentooized, but gnu
at it's heart) toolchain, which includes the versions of the utils
stated above (m4, sed, awk, etc), binutils, and gcc.

Salu2!,
Javier.

Reply | Threaded
Open this post in threaded view
|

Re: [SoC 2009] Ping ? Pong ! News and progress..

Patrice Clement-2
Hi Javier, and thank you for your reply.

Saturday 15 Aug 2009 10:44:27 (-0300), Javier Villavicencio wrote :

>
> Ahh good old times. FreeBSD devs did also patch GCC sources to work on
> FreeBSD. See http://bugs.gentoo.org/192403 for what it takes to get "our
> toolchain" to behave properly on FreeBSD. And that's why I asked some
> time ago if you managed to compile the kernel with a gentoo toolchain,
> it's a big pain in the back :D.
>
> There might be many things patched, however those in the gcc spec are
> key to get a working gcc, and lucky us, the specs don't vary too much,
> so you can try a diff between NetBSD gcc specs versus GNU gcc specs to
> spot what's missing/required.
>
> From there you can start a "copy" of your sys-devel/gcc ebuild of
> choice, and add patches as required, once you get a somewhat working
> version the patches can be tested/reviewed by the GNU devs to iron out
> what's not required, what's correct, and what's wrong.
>
> And here is where the legal BSD vs GPL battle starts, cause you can't
> add BSD copyrighted code into GPL code :D, fun fun.
>
> As a last note, beware that some of the ebuilds in portage may not
> compile correctly or at all if they don't use a GNU (gentooized, but gnu
> at it's heart) toolchain, which includes the versions of the utils
> stated above (m4, sed, awk, etc), binutils, and gcc.
>
> Salu2!,
> Javier.
>

That seems to be a huge amount of work. As said before, I'll keep things
"as-is" for now because it just works. This week, I'll focus on problems
solving with PAM and stage building. I'm sorry but I'm a bit in a hurry due to
lack of time. I want to bring something functionnal for everyone. I really want
to continue my work after GSoC, don't doubt on my motivation. You, guys and
devs of this mailing list, were always here to bring me some piece of advices
when I was in trouble.. I want to give a back to you. :)

Cheers!

Patrice