'Continuous Integration' for Gentoo Prefix?

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

'Continuous Integration' for Gentoo Prefix?

Sam Pfeiffer
Hello,

First, let me apologise if I have any wrong preexisting ideas/guesses about Gentoo Prefix and how it is developed. Secondly, sorry if this email is a bit too long.

I'll present myself, explain my use-case and my troubles and tricks, and then ask for feedback.

Hello, I'm Sammy Pfeiffer. I am a PhD student at University of Technology Sydney. I'm a (software) robotics engineer pursuing my PhD in a lab with a bunch of different robots and machine learning setups. I'm using Gentoo Prefix for deploying a big bunch of software into robots where the OS is old and frozen (and also to deploy in computation farms experiments... once again with no option to touch the OS). (The bunch of software is ROS, the 'Robotics Operating System' which has a ton of dependencies, and there is an existing overlay which I'm helping on maintaining and improving (https://github.com/ros/ros-overlay)).

I'd like to share the trick I found to overcome a few annoying bits of my platforms:
* I have no root access in the machines I need to deploy.
* The machines don't have Docker installed (too old kernel anyways).
* I have 32 bit and 64 bit machines (the OS running in them is).
* I have a different username (and home folder) on each machine.
* Each machine has a different disk(s)/partition(s) structure (main issue here is that I may need to use a different disk to store my data, cause of size constrains).

The trick to be able to use Gentoo Prefix with all these constraints bootstrapping it only once is to set the EPREFIX to /tmp/gentoo.
Then you can deploy the full bootstrapped system in any folder/disk and just do a softlink to /tmp/gentoo. This works nicely (I was scared of the softlink breaking stuff somewhere, but it was alright).
The other trick is to bootstrap a 32bit Gentoo Prefix, which can be run in any 32b or 64b box (it's not ideal but simplifies my deployment currently).

Also, having all these machines use the same Gentoo Prefix in the same place, with the same 32bit compilation, I can setup a binary package server, which all the deployments can point to and just get all the packages skipping the long compilation times (specially in very old and low powered machines).

The final trick I'm experimenting with is to use a set of Docker images (and soon in a continuous integration environment) to bootstrap all the system. With that I can save snapshot of successfully built systems & packages to serve as an easy deployment (and easy installation of extra packages thru the binary package server).

Given my particular annoyance of the 32bit system (and noting that my hosts are 64bits, as is the standard nowadays) I found that I can use either:
https://github.com/docker-32bit/ubuntu a 32bit Ubuntu Docker image
https://github.com/gentoo/gentoo-docker-images stage3-x86 32bit Gentoo Docker image

And then execute the build step of Docker with:

setarch i686 docker build -t my_bootstrapping_gentoo_prefix_32b_image .

Which will trick any program trying to do uname -m to assume 32bit machine. (Previously I used the variable CHOST=i686-pc-linux-gnu for bootstrapping, but I found some problems, which didn't appear with this method).

Once I have all this setup working nicely, I'd like to trigger rebuilds every X time, and on changes on main players (or all dependencies actually) of my setup, like the bootstrapping of Gentoo Prefix (Ideally, on every change of a part of the system just trigger a rebuild from that point on, Docker layers make this possible).

With Azure announcing unlimited minutes on CI/CD for open source projects: 

Even bootstrapping Gentoo prefix, with pieces of software like gcc taking very long to compile, is possible.

The point is: I have been trying to build Gentoo Prefix during the last days after a few months of break since the last time I touched the system. And it's failing. I haven't managed yet to bootstrap it completely. I feel there is no CI/CD setup to catch these issues and be able to offer a working version of Gentoo Prefix at any time.

I was going to build it for myself (cause I need it), but instead, I'd like to offer my help to build it for the community. At least offer as an option ready-to-use /tmp/gentoo EPREFIX'ed built Gentoo Prefix from a Docker image, just copy the full folder structure, do the softlink, and you are ready to play with Gentoo. If you mess up anything, just re-deploy.

To do this I'd need a bit of help as I'm quite new to Gentoo and I tend to get blocked on little issues that take a while to google or debug (specially with the long compilation times of the bootstrap and some big packages).

I've posted a short issue in the Docker repo of Gentoo images about this (https://github.com/gentoo/gentoo-docker-images/issues/62) but I think the maintainers probably don't usually work with Gentoo Prefix.

Thank you very much for your time, and for your open source efforts.

--

Sammy Pfeiffer
PhD Candidate at The Magic Lab within UTS.
Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

Fabian Groffen-2
Hi Sam,

On 26-11-2018 15:02:26 +1100, Sam Pfeiffer wrote:

> Hello,
>
> First, let me apologise if I have any wrong preexisting ideas/guesses about
> Gentoo Prefix and how it is developed. Secondly, sorry if this email is a bit
> too long.
>
> I'll present myself, explain my use-case and my troubles and tricks, and then
> ask for feedback.
>
> Hello, I'm Sammy Pfeiffer. I am a PhD student at University of Technology
> Sydney. I'm a (software) robotics engineer pursuing my PhD in a lab with a bunch
> of different robots and machine learning setups. I'm using Gentoo Prefix for
> deploying a big bunch of software into robots where the OS is old and frozen
> (and also to deploy in computation farms experiments... once again with no
> option to touch the OS). (The bunch of software is ROS, the 'Robotics Operating
> System' which has a ton of dependencies, and there is an existing overlay which
> I'm helping on maintaining and improving
> ([1]https://github.com/ros/ros-overlay)).
>
> I'd like to share the trick I found to overcome a few annoying bits of my
> platforms:
>
> * I have no root access in the machines I need to deploy.
>
> * The machines don't have Docker installed (too old kernel anyways).
>
> * I have 32 bit and 64 bit machines (the OS running in them is).
>
> * I have a different username (and home folder) on each machine.
>
> * Each machine has a different disk(s)/partition(s) structure (main issue here
> is that I may need to use a different disk to store my data, cause of size
> constrains).
>
> The trick to be able to use Gentoo Prefix with all these constraints
> bootstrapping it only once is to set the EPREFIX to /tmp/gentoo.
That will work best, I presume.

> Then you can deploy the full bootstrapped system in any folder/disk and just do
> a softlink to /tmp/gentoo. This works nicely (I was scared of the softlink
> breaking stuff somewhere, but it was alright).
>
> The other trick is to bootstrap a 32bit Gentoo Prefix, which can be run in any
> 32b or 64b box (it's not ideal but simplifies my deployment currently).
>
> Also, having all these machines use the same Gentoo Prefix in the same place,
> with the same 32bit compilation, I can setup a binary package server, which all
> the deployments can point to and just get all the packages skipping the long
> compilation times (specially in very old and low powered machines).
Prefix branch knows a trick which can "relocate" packages.  The
constrains is that it can only "shorten" a prefix.  Thus, if you'd build
for /a/very/very/very/long/prefix you could install those binary
packages to /a/shorter/prefix.  I think this will also addresses your
userid problems (if any).

> The final trick I'm experimenting with is to use a set of Docker images (and
> soon in a continuous integration environment) to bootstrap all the system. With
> that I can save snapshot of successfully built systems & packages to serve as an
> easy deployment (and easy installation of extra packages thru the binary package
> server).

Neat!

> Given my particular annoyance of the 32bit system (and noting that my hosts are
> 64bits, as is the standard nowadays) I found that I can use either:
>
> * [2]https://github.com/docker-32bit/ubuntu a 32bit Ubuntu Docker image
>
> * [3]https://github.com/gentoo/gentoo-docker-images stage3-x86 32bit Gentoo
> Docker image
>
> And then execute the build step of Docker with:
>
> setarch i686 docker build -t my_bootstrapping_gentoo_prefix_32b_image .
>
> Which will trick any program trying to do uname -m to assume 32bit machine.
> (Previously I used the variable CHOST=i686-pc-linux-gnu for bootstrapping, but I
> found some problems, which didn't appear with this method).
Hmmm.

> Once I have all this setup working nicely, I'd like to trigger rebuilds every X
> time, and on changes on main players (or all dependencies actually) of my setup,
> like the bootstrapping of Gentoo Prefix (Ideally, on every change of a part of
> the system just trigger a rebuild from that point on, Docker layers make this
> possible).

This may be a bit too much, but you could start from every day
(LATEST_TREE_YES), or we can see to building a tar more frequently from
rsync0.

> With Azure announcing unlimited minutes on CI/CD for open source projects: 
>
> [4]https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
>
> Even bootstrapping Gentoo prefix, with pieces of software like gcc taking very
> long to compile, is possible.

I'm still missing the infra to administer this, but I have a wrapper
script to bootstrap-prefix.sh (I'm sure others (haubi?) have too) to do
an unattended bootstrap.  I want this to run automatically on some boxes
I have, but some of them will take several days to complete.
Another thing is for a bootstrapped prefix to periodically emerge --sync
and emerge -Dua world.  Add on top of that detecting which packages have
keywords for said prefix and installing them, and we should be able to
be notice breakage rather sooner than later.

> The point is: I have been trying to build Gentoo Prefix during the last days
> after a few months of break since the last time I touched the system. And it's
> failing. I haven't managed yet to bootstrap it completely. I feel there is no
> CI/CD setup to catch these issues and be able to offer a working version of
> Gentoo Prefix at any time.

Sorry about that.  I need to make another pass over it.

> I was going to build it for myself (cause I need it), but instead, I'd like to
> offer my help to build it for the community. At least offer as an option
> ready-to-use /tmp/gentoo EPREFIX'ed built Gentoo Prefix from a Docker image,
> just copy the full folder structure, do the softlink, and you are ready to play
> with Gentoo. If you mess up anything, just re-deploy.
>
> To do this I'd need a bit of help as I'm quite new to Gentoo and I tend to get
> blocked on little issues that take a while to google or debug (specially with
> the long compilation times of the bootstrap and some big packages).
>
> I've posted a short issue in the Docker repo of Gentoo images about this
> ([5]https://github.com/gentoo/gentoo-docker-images/issues/62) but I think the
> maintainers probably don't usually work with Gentoo Prefix.
I'm not very familiar myself with Docker (mostly hiding under a rock
here), but I suspect the gentoo-docker-images project responds better to
email to their list (as in Contributing from their README).  I may be
wrong though.

> Thank you very much for your time, and for your open source efforts.

Thanks for your efforts sofar.  I hope we can pull something together.
At least in the coming period I hope to be able to fix the bootstrapping
issues for Solaris/OpenIndiana/Darwin.

Thanks,
Fabian

--
Fabian Groffen
Gentoo on a different level

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

M. J. Everitt
On 26/11/18 08:11, Fabian Groffen wrote [as excerpted]:

> Hi Sam,
>
> On 26-11-2018 15:02:26 +1100, Sam Pfeiffer wrote:
>> With Azure announcing unlimited minutes on CI/CD for open source projects: 
>>
>> [4]https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
>>
>> Even bootstrapping Gentoo prefix, with pieces of software like gcc taking very
>> long to compile, is possible.
> I'm still missing the infra to administer this, but I have a wrapper
> script to bootstrap-prefix.sh (I'm sure others (haubi?) have too) to do
> an unattended bootstrap.  I want this to run automatically on some boxes
> I have, but some of them will take several days to complete.
> Another thing is for a bootstrapped prefix to periodically emerge --sync
> and emerge -Dua world.  Add on top of that detecting which packages have
> keywords for said prefix and installing them, and we should be able to
> be notice breakage rather sooner than later.
>
From what I have observed, breakage seems to be undesirably frequent, and
there are regularly users in #gentoo-prefix on IRC complaining that
bootstrap-prefix.sh fails.

I'm sure that if native Gentoo hardware isn't available, it should be
possible to approach OSL or other providers (I, myself, am using
IntegriCloud for powerpc development) to sponsor a host, and/or put
together a funding bid to the Trustees (there is capital available).

TL;DR I don't think "ENOINFRA" is a particularly good excuse in this
day-and-age of cloud-based solutions.

Let us help you, and perhaps we can get the very popular Gentoo Prefix onto
a slightly firmer footing! :)

Best Regards,

Michael "veremitz" Everitt.


signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

Fabian Groffen-2
On 26-11-2018 08:27:41 +0000, M. J. Everitt wrote:

> On 26/11/18 08:11, Fabian Groffen wrote [as excerpted]:
> > Hi Sam,
> >
> > On 26-11-2018 15:02:26 +1100, Sam Pfeiffer wrote:
> >> With Azure announcing unlimited minutes on CI/CD for open source projects: 
> >>
> >> [4]https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
> >>
> >> Even bootstrapping Gentoo prefix, with pieces of software like gcc taking very
> >> long to compile, is possible.
> > I'm still missing the infra to administer this, but I have a wrapper
> > script to bootstrap-prefix.sh (I'm sure others (haubi?) have too) to do
> > an unattended bootstrap.  I want this to run automatically on some boxes
> > I have, but some of them will take several days to complete.
> > Another thing is for a bootstrapped prefix to periodically emerge --sync
> > and emerge -Dua world.  Add on top of that detecting which packages have
> > keywords for said prefix and installing them, and we should be able to
> > be notice breakage rather sooner than later.
> >
> From what I have observed, breakage seems to be undesirably frequent, and
> there are regularly users in #gentoo-prefix on IRC complaining that
> bootstrap-prefix.sh fails.
>
> I'm sure that if native Gentoo hardware isn't available, it should be
> possible to approach OSL or other providers (I, myself, am using
> IntegriCloud for powerpc development) to sponsor a host, and/or put
> together a funding bid to the Trustees (there is capital available).
>
> TL;DR I don't think "ENOINFRA" is a particularly good excuse in this
> day-and-age of cloud-based solutions.
Sorry.  with Infra, I didn't mean the team or the hardware, I meant the
scripting/software to keep track of the builds and visualise their
results.  I can fire off a bootstrap right now, but there is nothing
picking up on their results.

> Let us help you, and perhaps we can get the very popular Gentoo Prefix onto
> a slightly firmer footing! :)

At this point I think he project at the very least needs a lot of
helping hands.  I totally agree, automation is the way forward, but it
always will need caretakers.

Thanks,
Fabian


--
Fabian Groffen
Gentoo on a different level

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

M. J. Everitt
On 26/11/18 08:32, Fabian Groffen wrote:

> On 26-11-2018 08:27:41 +0000, M. J. Everitt wrote:
>> On 26/11/18 08:11, Fabian Groffen wrote [as excerpted]:
>>> Hi Sam,
>>>
>>> On 26-11-2018 15:02:26 +1100, Sam Pfeiffer wrote:
>>>> With Azure announcing unlimited minutes on CI/CD for open source projects: 
>>>>
>>>> [4]https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
>>>>
>>>> Even bootstrapping Gentoo prefix, with pieces of software like gcc taking very
>>>> long to compile, is possible.
>>> I'm still missing the infra to administer this, but I have a wrapper
>>> script to bootstrap-prefix.sh (I'm sure others (haubi?) have too) to do
>>> an unattended bootstrap.  I want this to run automatically on some boxes
>>> I have, but some of them will take several days to complete.
>>> Another thing is for a bootstrapped prefix to periodically emerge --sync
>>> and emerge -Dua world.  Add on top of that detecting which packages have
>>> keywords for said prefix and installing them, and we should be able to
>>> be notice breakage rather sooner than later.
>>>
>> From what I have observed, breakage seems to be undesirably frequent, and
>> there are regularly users in #gentoo-prefix on IRC complaining that
>> bootstrap-prefix.sh fails.
>>
>> I'm sure that if native Gentoo hardware isn't available, it should be
>> possible to approach OSL or other providers (I, myself, am using
>> IntegriCloud for powerpc development) to sponsor a host, and/or put
>> together a funding bid to the Trustees (there is capital available).
>>
>> TL;DR I don't think "ENOINFRA" is a particularly good excuse in this
>> day-and-age of cloud-based solutions.
> Sorry.  with Infra, I didn't mean the team or the hardware, I meant the
> scripting/software to keep track of the builds and visualise their
> results.  I can fire off a bootstrap right now, but there is nothing
> picking up on their results.
>
>> Let us help you, and perhaps we can get the very popular Gentoo Prefix onto
>> a slightly firmer footing! :)
> At this point I think he project at the very least needs a lot of
> helping hands.  I totally agree, automation is the way forward, but it
> always will need caretakers.
>
> Thanks,
> Fabian
>
>
I'll throw my 'oar' in here too - I've been helping out extensively with
the kernelCI project @alicef spawned. We're presently using buildbot to do
automated emerges and a 'make defconfig' on the kernel sources, just to
make sure patching works, and compiles. We've actually reached a point too,
where we QEMU-boot said kernel image into a pre-built stage4.

I'm happy to share some of the know-how to get some of this spun up for
prefix if it's desirable. I've had to replicate the 'production' setup
locally in order to develop and test changes, so its a process I'm becoming
pretty familiar with.

Buildbot is python, so it integrates pretty well with portage, and is
accessible for devs. Let me know if this is a direction that would be
favourable.

https://github.com/gentoo/Gentoo_kernelCI/

Best regards,

veremitz/Michael.


signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

Michael Haubenwallner-3
In reply to this post by Fabian Groffen-2
Hi Sam,

On 11/26/2018 09:11 AM, Fabian Groffen wrote:
<snip intro>

>> Given my particular annoyance of the 32bit system (and noting that my hosts are
>> 64bits, as is the standard nowadays) I found that I can use either:
>>
>> * [2]https://github.com/docker-32bit/ubuntu a 32bit Ubuntu Docker image
>>
>> * [3]https://github.com/gentoo/gentoo-docker-images stage3-x86 32bit Gentoo
>> Docker image
>>
>> And then execute the build step of Docker with:
>>
>> setarch i686 docker build -t my_bootstrapping_gentoo_prefix_32b_image .
>>
>> Which will trick any program trying to do uname -m to assume 32bit machine.
>> (Previously I used the variable CHOST=i686-pc-linux-gnu for bootstrapping, but I
>> found some problems, which didn't appear with this method).
>
> Hmmm.

On 64bit machines, one can use 'linux32' to trick the autodetection into 32bit:
 $ uname -m
 x86_64
 $ linux32 uname -m
 i686

>> Once I have all this setup working nicely, I'd like to trigger rebuilds every X
>> time, and on changes on main players (or all dependencies actually) of my setup,
>> like the bootstrapping of Gentoo Prefix (Ideally, on every change of a part of
>> the system just trigger a rebuild from that point on, Docker layers make this
>> possible).
>
> This may be a bit too much, but you could start from every day
> (LATEST_TREE_YES), or we can see to building a tar more frequently from
> rsync0.
>
>> With Azure announcing unlimited minutes on CI/CD for open source projects: 
>>
>> [4]https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/

Nice, wasn't aware of that one!

>>
>> Even bootstrapping Gentoo prefix, with pieces of software like gcc taking very
>> long to compile, is possible.
>
> I'm still missing the infra to administer this, but I have a wrapper
> script to bootstrap-prefix.sh (I'm sure others (haubi?) have too) to do
> an unattended bootstrap.  I want this to run automatically on some boxes
> I have, but some of them will take several days to complete.
> Another thing is for a bootstrapped prefix to periodically emerge --sync
> and emerge -Dua world.  Add on top of that detecting which packages have
> keywords for said prefix and installing them, and we should be able to
> be notice breakage rather sooner than later.

Yes, I do have a buildbot setup within the company I'm working at,
but unfortunately I don't have the time to inspect the results and
fix upcoming problems on a regular base.
Beyond that, I cannot open the buildbot-master to the public, but
I believe I could share build results if we had some public place
to store them.

Anyway - the Prefix buildbot slaves here are doing a clean bootstrap
(with LATEST_TREE_YES=1, noninteractive) once per week, and an emerge
update/deep/world once per day. The list of "KEYWORD (hosts)" here is:
ppc-aix (AIX 6.1/64bit, AIX 7.1/64bit)
x86_64-cygwin (Windows Server 2012)
sparc-solaris (Solaris 10/64bit)
x86-linux (SLES11.1/64bit, SLES12.0/64bit, RHEL7.1/64bit, RHEL6.2/64bit)
amd64-linux (Gentoo/64bit)

Also, I do have need for Prefix Guest only, no RAP here - but there should be
hardware resources left to perform both builds.

>> The point is: I have been trying to build Gentoo Prefix during the last days
>> after a few months of break since the last time I touched the system. And it's
>> failing. I haven't managed yet to bootstrap it completely. I feel there is no
>> CI/CD setup to catch these issues and be able to offer a working version of
>> Gentoo Prefix at any time.
>
> Sorry about that.  I need to make another pass over it.

Indeed - only the 64bit (Cygwin & Linux) ones were succeeding recently,
all the 32bit ones failed.

>> I was going to build it for myself (cause I need it), but instead, I'd like to
>> offer my help to build it for the community. At least offer as an option
>> ready-to-use /tmp/gentoo EPREFIX'ed built Gentoo Prefix from a Docker image,
>> just copy the full folder structure, do the softlink, and you are ready to play
>> with Gentoo. If you mess up anything, just re-deploy.
>>
>> To do this I'd need a bit of help as I'm quite new to Gentoo and I tend to get
>> blocked on little issues that take a while to google or debug (specially with
>> the long compilation times of the bootstrap and some big packages).
>>
>> I've posted a short issue in the Docker repo of Gentoo images about this
>> ([5]https://github.com/gentoo/gentoo-docker-images/issues/62) but I think the
>> maintainers probably don't usually work with Gentoo Prefix.
>
> I'm not very familiar myself with Docker (mostly hiding under a rock
> here), but I suspect the gentoo-docker-images project responds better to
> email to their list (as in Contributing from their README).  I may be
> wrong though.
>
>> Thank you very much for your time, and for your open source efforts.
>
> Thanks for your efforts sofar.  I hope we can pull something together.
> At least in the coming period I hope to be able to fix the bootstrapping
> issues for Solaris/OpenIndiana/Darwin.

I would love to see my buildbot slave's results somewhere in the public,
thank you for triggering this!

/haubi/

Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

Benda Xu
In reply to this post by Sam Pfeiffer
Hi Sam,

Sam Pfeiffer <[hidden email]> writes:

> With Azure announcing unlimited minutes on CI/CD for open source
> projects:
> https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
>
> Even bootstrapping Gentoo prefix, with pieces of software like gcc
> taking very long to compile, is possible.
>
> The point is: I have been trying to build Gentoo Prefix during the
> last days after a few months of break since the last time I touched
> the system. And it's failing. I haven't managed yet to bootstrap it
> completely. I feel there is no CI/CD setup to catch these issues and
> be able to offer a working version of Gentoo Prefix at any time.

I completely agree with you.  I hope you can carry on this project to
setup proper CI for Gentoo Prefix.  I am all in for help, portage/ebuild
mentoring and coorperation.

A CI for Gentoo Prefix has been on my list for ages.  Thank you for
triggering this.

Yours,
Benda

Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

Sam Pfeiffer
Hello everyone,

I'm very excited to see you are interested in adding continuous integration!

I don't know that much about continuous integration, I've only used it (with systems already setup for me) with in-house Jenkins servers and with the ROS buildfarm, based on Travis CI on Github. Also a little bit of Gitlab CI in my lab.

I did a bit of research/testing.

Given it's quite a hassle to maintain custom machines, I'd try to use some of the free for opensource CI services. I've checked the conditions of a few to see which could fit better:

* Gitlab CI: 2000 minutes / month
* Travis CI: Unlimited minutes / month. But only 50 minutes long per step (like per script executed).
* Azure pipelines: Unlimited minutes / month. 360 minutes long per step (like per script executed).

There are probably many more, but those are the ones I knew about.

Given that I wanted to give a try to Azure pipelines. And I did!


Where I activated Azure pipelines on it. After around 15min of reading the docs and playing around with the web-gui I got my first pipeline running.

As an initial setup I thought I would create a Docker image where I bootstrap Gentoo Prefix from a Ubuntu 16.04 (as I'm familiar with it with my projects).

The repo contains two important things:
1) The Dockerfile where I mainly trigger the bootstrap: https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/Dockerfile
2) The configuration file for Azure pipelines on what to do: https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/azure-pipelines.yml

I've implemented here that it tries to build Gentoo Prefix, and whatever the result, it uploads a Docker image to my DockerHub account with the results. This implies that:
If the bootstrap is successful, one can just [docker pull] and [docker run] the image to play with Gentoo Prefix.
If the bootstrap is unsuccessful, one can just [docker pull] and [docker run] to find oneself in the exact state of the system after the bootstrap command. And one can recover the full console log from the Azure pipelines web interface (even tho it would be nice to find out how to post it publicly straight away).

If all goes well in a few hours anyone will be able to find in my DockerHub account said image (most probably the failed one), just doing:
docker pull awesomebytes/gentoo_prefix_latest_image:latest
docker run -it gentoo_prefix_latest_image /bin/bash
You'll be inside of a Ubuntu 16.04 box with a user called 'user' and with all the bootstrapped stuff in /tmp/gentoo.

As curiosity, I checked the machines I got served as 'agents' to run my jobs, and they were of the kind:

CPUs: 2x Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
RAM: 7GB
Disk: 94GB free disk space

More than enough to bootstrap Gentoo Prefix!

I don't know if this is the way to go. But at least is interesting to have it in mind.


On Tue, Nov 27, 2018 at 12:01 AM Benda Xu <[hidden email]> wrote:
Hi Sam,

Sam Pfeiffer <[hidden email]> writes:

> With Azure announcing unlimited minutes on CI/CD for open source
> projects:
> https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
>
> Even bootstrapping Gentoo prefix, with pieces of software like gcc
> taking very long to compile, is possible.
>
> The point is: I have been trying to build Gentoo Prefix during the
> last days after a few months of break since the last time I touched
> the system. And it's failing. I haven't managed yet to bootstrap it
> completely. I feel there is no CI/CD setup to catch these issues and
> be able to offer a working version of Gentoo Prefix at any time.

I completely agree with you.  I hope you can carry on this project to
setup proper CI for Gentoo Prefix.  I am all in for help, portage/ebuild
mentoring and coorperation.

A CI for Gentoo Prefix has been on my list for ages.  Thank you for
triggering this.

Yours,
Benda



--

Sammy Pfeiffer
PhD Candidate at The Magic Lab within UTS.
Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

Sam Pfeiffer
Little update: The full build log is viewable to anyone with the link, so here you can see the progress of the current job: 


(Or I should say, the log of it, for whenever you open the link).


On Tue, Nov 27, 2018 at 2:02 AM Sam Pfeiffer <[hidden email]> wrote:
Hello everyone,

I'm very excited to see you are interested in adding continuous integration!

I don't know that much about continuous integration, I've only used it (with systems already setup for me) with in-house Jenkins servers and with the ROS buildfarm, based on Travis CI on Github. Also a little bit of Gitlab CI in my lab.

I did a bit of research/testing.

Given it's quite a hassle to maintain custom machines, I'd try to use some of the free for opensource CI services. I've checked the conditions of a few to see which could fit better:

* Gitlab CI: 2000 minutes / month
* Travis CI: Unlimited minutes / month. But only 50 minutes long per step (like per script executed).
* Azure pipelines: Unlimited minutes / month. 360 minutes long per step (like per script executed).

There are probably many more, but those are the ones I knew about.

Given that I wanted to give a try to Azure pipelines. And I did!


Where I activated Azure pipelines on it. After around 15min of reading the docs and playing around with the web-gui I got my first pipeline running.

As an initial setup I thought I would create a Docker image where I bootstrap Gentoo Prefix from a Ubuntu 16.04 (as I'm familiar with it with my projects).

The repo contains two important things:
1) The Dockerfile where I mainly trigger the bootstrap: https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/Dockerfile
2) The configuration file for Azure pipelines on what to do: https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/azure-pipelines.yml

I've implemented here that it tries to build Gentoo Prefix, and whatever the result, it uploads a Docker image to my DockerHub account with the results. This implies that:
If the bootstrap is successful, one can just [docker pull] and [docker run] the image to play with Gentoo Prefix.
If the bootstrap is unsuccessful, one can just [docker pull] and [docker run] to find oneself in the exact state of the system after the bootstrap command. And one can recover the full console log from the Azure pipelines web interface (even tho it would be nice to find out how to post it publicly straight away).

If all goes well in a few hours anyone will be able to find in my DockerHub account said image (most probably the failed one), just doing:
docker pull awesomebytes/gentoo_prefix_latest_image:latest
docker run -it gentoo_prefix_latest_image /bin/bash
You'll be inside of a Ubuntu 16.04 box with a user called 'user' and with all the bootstrapped stuff in /tmp/gentoo.

As curiosity, I checked the machines I got served as 'agents' to run my jobs, and they were of the kind:

CPUs: 2x Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
RAM: 7GB
Disk: 94GB free disk space

More than enough to bootstrap Gentoo Prefix!

I don't know if this is the way to go. But at least is interesting to have it in mind.


On Tue, Nov 27, 2018 at 12:01 AM Benda Xu <[hidden email]> wrote:
Hi Sam,

Sam Pfeiffer <[hidden email]> writes:

> With Azure announcing unlimited minutes on CI/CD for open source
> projects:
> https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
>
> Even bootstrapping Gentoo prefix, with pieces of software like gcc
> taking very long to compile, is possible.
>
> The point is: I have been trying to build Gentoo Prefix during the
> last days after a few months of break since the last time I touched
> the system. And it's failing. I haven't managed yet to bootstrap it
> completely. I feel there is no CI/CD setup to catch these issues and
> be able to offer a working version of Gentoo Prefix at any time.

I completely agree with you.  I hope you can carry on this project to
setup proper CI for Gentoo Prefix.  I am all in for help, portage/ebuild
mentoring and coorperation.

A CI for Gentoo Prefix has been on my list for ages.  Thank you for
triggering this.

Yours,
Benda



--

Sammy Pfeiffer
PhD Candidate at The Magic Lab within UTS.


--

Sammy Pfeiffer
PhD Candidate at The Magic Lab within UTS.
Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

Sam Pfeiffer
Hello all,

Another little update on my test, it took me a while to find the actual way to increase the timeout to the maximum (or in other words, get out of the default of 60minutes), but I finally found how.

Now I have a job that just tries to bootstrap Gentoo Prefix, the last run I made can be found here: https://dev.azure.com/12719821/12719821/_build/results?buildId=21&view=logs
You can see all the log. In this case it failed after 1h35min. It failed compiling GCC 8.2.0-r4.

The error was:

2018-11-27T03:20:31.7540250Z /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/xg++ -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/ -B/tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ -nostdinc++ -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs  -isystem /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu  -isystem /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include  -isystem /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-8.2.0/libstdc++-v3/libsupc++ -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs -no-pie -fno-PIE    -m64 -O2 -pipe -O2 -pipe -gtoggle -DIN_GCC     -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings   -DHAVE_CONFIG_H -static-libstdc++ -static-libgcc  gcov.o \
2018-11-27T03:20:31.7545947Z hash-table.o ggc-none.o libcommon.a ../libcpp/libcpp.a   ../libbacktrace/.libs/libbacktrace.a ../libiberty/libiberty.a ../libdecnumber/libdecnumber.a  -o gcov
2018-11-27T03:20:31.7670930Z /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/collect-ld: 106: exec: /tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ld: Invalid argument
2018-11-27T03:20:31.8025313Z collect2: error: ld returned 2 exit status
2018-11-27T03:20:31.8026192Z Makefile:2935: recipe for target 'gcov' failed

In the end of the log we get the usual:

2018-11-27T03:23:00.8458538Z  * ERROR: sys-devel/gcc-8.2.0-r4::gentoo failed (compile phase):
2018-11-27T03:23:00.8480853Z  *   emake failed
2018-11-27T03:23:00.8501574Z  * 
2018-11-27T03:23:00.8524914Z  * If you need support, post the output of `emerge --info '=sys-devel/gcc-8.2.0-r4::gentoo'`,
2018-11-27T03:23:00.8550010Z  * the complete build log and the output of `emerge -pqv '=sys-devel/gcc-8.2.0-r4::gentoo'`.
2018-11-27T03:23:00.8572142Z  * The complete build log is located at '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/temp/build.log'.
2018-11-27T03:23:00.8593188Z  * The ebuild environment file is located at '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/temp/environment'.
2018-11-27T03:23:00.8622509Z  * Working directory: '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build'
2018-11-27T03:23:00.8642956Z  * S: '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-8.2.0'
2018-11-27T03:23:02.9054285Z  * 
2018-11-27T03:23:02.9078678Z  * Please include /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-build-logs.tar.bz2 in your bug report.

The cool thing comes now, how do you actually execute those commands to fill up a bug report?

Easy, in the job I upload a Docker image with the system exactly as it was after the boostrap-prefix.sh command.

So, if anyone wants to debug what is going on, they just need to do (this works right now):

docker pull awesomebytes/gentoo_prefix_latest_image
docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash

And you are inside of the box:

user@b70d59ff9703:~$ ls
bootstrap-prefix.sh  start_bootstrap_date.txt
user@b70d59ff9703:~$ cat start_bootstrap_date.txt 
Tue Nov 27 01:52:20 UTC 2018
user@b70d59ff9703:~$ ls /tmp/gentoo
bin  etc  lib  lib64  run  sbin  stage1.log  stage2.log  stage3.log  tmp  usr  var

So you can do those commands suggested by emerge:
/tmp/gentoo/tmp/usr/bin/emerge --info '=sys-devel/gcc-8.2.0-r4::gentoo' --> https://pastebin.com/LWS3Bb7S
/tmp/gentoo/tmp/usr/bin/emerge -pqv '=sys-devel/gcc-8.2.0-r4::gentoo'  --> https://pastebin.com/gipspmnD
etc.

Or... we could even generate a new bug report automatically from this will all the necessary information, including the instructions on how to get into this box with exactly that state for anyone to help on fixing it.

I'll try to dig further in with things that I find useful (and I hope you also find useful).

Have a nice day!

P.S.: With this I created a bug report easily: https://bugs.gentoo.org/672042


On Tue, Nov 27, 2018 at 2:07 AM Sam Pfeiffer <[hidden email]> wrote:
Little update: The full build log is viewable to anyone with the link, so here you can see the progress of the current job: 


(Or I should say, the log of it, for whenever you open the link).


On Tue, Nov 27, 2018 at 2:02 AM Sam Pfeiffer <[hidden email]> wrote:
Hello everyone,

I'm very excited to see you are interested in adding continuous integration!

I don't know that much about continuous integration, I've only used it (with systems already setup for me) with in-house Jenkins servers and with the ROS buildfarm, based on Travis CI on Github. Also a little bit of Gitlab CI in my lab.

I did a bit of research/testing.

Given it's quite a hassle to maintain custom machines, I'd try to use some of the free for opensource CI services. I've checked the conditions of a few to see which could fit better:

* Gitlab CI: 2000 minutes / month
* Travis CI: Unlimited minutes / month. But only 50 minutes long per step (like per script executed).
* Azure pipelines: Unlimited minutes / month. 360 minutes long per step (like per script executed).

There are probably many more, but those are the ones I knew about.

Given that I wanted to give a try to Azure pipelines. And I did!


Where I activated Azure pipelines on it. After around 15min of reading the docs and playing around with the web-gui I got my first pipeline running.

As an initial setup I thought I would create a Docker image where I bootstrap Gentoo Prefix from a Ubuntu 16.04 (as I'm familiar with it with my projects).

The repo contains two important things:
1) The Dockerfile where I mainly trigger the bootstrap: https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/Dockerfile
2) The configuration file for Azure pipelines on what to do: https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/azure-pipelines.yml

I've implemented here that it tries to build Gentoo Prefix, and whatever the result, it uploads a Docker image to my DockerHub account with the results. This implies that:
If the bootstrap is successful, one can just [docker pull] and [docker run] the image to play with Gentoo Prefix.
If the bootstrap is unsuccessful, one can just [docker pull] and [docker run] to find oneself in the exact state of the system after the bootstrap command. And one can recover the full console log from the Azure pipelines web interface (even tho it would be nice to find out how to post it publicly straight away).

If all goes well in a few hours anyone will be able to find in my DockerHub account said image (most probably the failed one), just doing:
docker pull awesomebytes/gentoo_prefix_latest_image:latest
docker run -it gentoo_prefix_latest_image /bin/bash
You'll be inside of a Ubuntu 16.04 box with a user called 'user' and with all the bootstrapped stuff in /tmp/gentoo.

As curiosity, I checked the machines I got served as 'agents' to run my jobs, and they were of the kind:

CPUs: 2x Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
RAM: 7GB
Disk: 94GB free disk space

More than enough to bootstrap Gentoo Prefix!

I don't know if this is the way to go. But at least is interesting to have it in mind.


On Tue, Nov 27, 2018 at 12:01 AM Benda Xu <[hidden email]> wrote:
Hi Sam,

Sam Pfeiffer <[hidden email]> writes:

> With Azure announcing unlimited minutes on CI/CD for open source
> projects:
> https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
>
> Even bootstrapping Gentoo prefix, with pieces of software like gcc
> taking very long to compile, is possible.
>
> The point is: I have been trying to build Gentoo Prefix during the
> last days after a few months of break since the last time I touched
> the system. And it's failing. I haven't managed yet to bootstrap it
> completely. I feel there is no CI/CD setup to catch these issues and
> be able to offer a working version of Gentoo Prefix at any time.

I completely agree with you.  I hope you can carry on this project to
setup proper CI for Gentoo Prefix.  I am all in for help, portage/ebuild
mentoring and coorperation.

A CI for Gentoo Prefix has been on my list for ages.  Thank you for
triggering this.

Yours,
Benda



--

Sammy Pfeiffer
PhD Candidate at The Magic Lab within UTS.


--

Sammy Pfeiffer
PhD Candidate at The Magic Lab within UTS.


--

Sammy Pfeiffer
PhD Candidate at The Magic Lab within UTS.
Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

Fabian Groffen-2
I don't want to depress this entire discussion, but it would be really
nice if we could somehow interact with special machines people have at
their company or at home.  Prefix needs testing on many different
machines (non-Linux) which usually don't exist in docker images.

That said, focussing on the (usually fast) boxes like this to catch
dependency problems and more is useful.  In the case below it looks like
the ld-wrapper is having issues.  Would it be possible to enter the
environment for that failed run?

Thanks,
Fabian

On 27-11-2018 17:14:19 +1100, Sam Pfeiffer wrote:

> Hello all,
>
> Another little update on my test, it took me a while to find the actual way to
> increase the timeout to the maximum (or in other words, get out of the default
> of 60minutes), but I finally found how.
>
> Now I have a job that just tries to bootstrap Gentoo Prefix, the last run I made
> can be found
> here: [1]https://dev.azure.com/12719821/12719821/_build/results?buildId=21&view=logs
>
> You can see all the log. In this case it failed after 1h35min. It failed
> compiling GCC 8.2.0-r4.
>
> The error was:
>
> 2018-11-27T03:20:31.7540250Z
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/xg++
> -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/
> -B/tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ -nostdinc++
> -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs
> -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs 
> -isystem
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu 
> -isystem
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include 
> -isystem
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-8.2.0/libstdc++-v3/libsupc++
> -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs
> -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs
> -no-pie -fno-PIE    -m64 -O2 -pipe -O2 -pipe -gtoggle -DIN_GCC   
>  -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing
> -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -Woverloaded-virtual
> -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings 
>  -DHAVE_CONFIG_H -static-libstdc++ -static-libgcc  gcov.o \
>
> 2018-11-27T03:20:31.7545947Z hash-table.o ggc-none.o libcommon.a
> ../libcpp/libcpp.a   ../libbacktrace/.libs/libbacktrace.a
> ../libiberty/libiberty.a ../libdecnumber/libdecnumber.a  -o gcov
>
> 2018-11-27T03:20:31.7670930Z
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/collect-ld:
> 106: exec: /tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ld: Invalid argument
>
> 2018-11-27T03:20:31.8025313Z collect2: error: ld returned 2 exit status
>
> 2018-11-27T03:20:31.8026192Z Makefile:2935: recipe for target 'gcov' failed
>
> In the end of the log we get the usual:
>
> 2018-11-27T03:23:00.8458538Z  * ERROR: sys-devel/gcc-8.2.0-r4::gentoo failed
> (compile phase):
>
> 2018-11-27T03:23:00.8480853Z  *   emake failed
>
> 2018-11-27T03:23:00.8501574Z  * 
>
> 2018-11-27T03:23:00.8524914Z  * If you need support, post the output of `emerge
> --info '=sys-devel/gcc-8.2.0-r4::gentoo'`,
>
> 2018-11-27T03:23:00.8550010Z  * the complete build log and the output of `emerge
> -pqv '=sys-devel/gcc-8.2.0-r4::gentoo'`.
>
> 2018-11-27T03:23:00.8572142Z  * The complete build log is located at
> '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/temp/build.log'.
>
> 2018-11-27T03:23:00.8593188Z  * The ebuild environment file is located at
> '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/temp/environment'.
>
> 2018-11-27T03:23:00.8622509Z  * Working directory:
> '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build'
>
> 2018-11-27T03:23:00.8642956Z  * S:
> '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-8.2.0'
>
> 2018-11-27T03:23:02.9054285Z  * 
>
> 2018-11-27T03:23:02.9078678Z  * Please include
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-build-logs.tar.bz2
> in your bug report.
>
> The cool thing comes now, how do you actually execute those commands to fill up
> a bug report?
>
> Easy, in the job I upload a Docker image with the system exactly as it was after
> the boostrap-prefix.sh command.
>
> So, if anyone wants to debug what is going on, they just need to do (this works
> right now):
>
> docker pull awesomebytes/gentoo_prefix_latest_image
>
> docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash
>
> And you are inside of the box:
>
> > user@b70d59ff9703:~$ ls
>
> > bootstrap-prefix.sh  start_bootstrap_date.txt
>
> > user@b70d59ff9703:~$ cat start_bootstrap_date.txt 
>
> > Tue Nov 27 01:52:20 UTC 2018
>
> > user@b70d59ff9703:~$ ls /tmp/gentoo
>
> > bin  etc  lib  lib64  run  sbin  stage1.log  stage2.log  stage3.log  tmp  usr 
> > var
>
> So you can do those commands suggested by emerge:
>
> /tmp/gentoo/tmp/usr/bin/emerge --info '=sys-devel/gcc-8.2.0-r4::gentoo' -->
> [2]https://pastebin.com/LWS3Bb7S
>
> /tmp/gentoo/tmp/usr/bin/emerge -pqv '=sys-devel/gcc-8.2.0-r4::gentoo' 
> --> [3]https://pastebin.com/gipspmnD
>
> etc.
>
> Or... we could even generate a new bug report automatically from this will all
> the necessary information, including the instructions on how to get into this
> box with exactly that state for anyone to help on fixing it.
>
> I'll try to dig further in with things that I find useful (and I hope you also
> find useful).
>
> Have a nice day!
>
> P.S.: With this I created a bug report easily: [4]https://bugs.gentoo.org/672042
>
> On Tue, Nov 27, 2018 at 2:07 AM Sam Pfeiffer <[5][hidden email]> wrote:
>
> > Little update: The full build log is viewable to anyone with the link, so here
> > you can see the progress of the current job: 
>
> > [6]https://dev.azure.com/12719821/12719821/_build/results?buildId=17&view=logs
>
> > (Or I should say, the log of it, for whenever you open the link).
>
> > On Tue, Nov 27, 2018 at 2:02 AM Sam Pfeiffer <[7][hidden email]>
> > wrote:
>
> >> Hello everyone,
>
> >> I'm very excited to see you are interested in adding continuous integration!
>
> >> I don't know that much about continuous integration, I've only used it (with
> >> systems already setup for me) with in-house Jenkins servers and with the ROS
> >> buildfarm, based on Travis CI on Github. Also a little bit of Gitlab CI in my
> >> lab.
>
> >> I did a bit of research/testing.
>
> >> Given it's quite a hassle to maintain custom machines, I'd try to use some of
> >> the free for opensource CI services. I've checked the conditions of a few to
> >> see which could fit better:
>
> >> * Gitlab CI: 2000 minutes / month
>
> >> * Travis CI: Unlimited minutes / month. But only 50 minutes long per step
> >> (like per script executed).
>
> >> * Azure pipelines: Unlimited minutes / month. 360 minutes long per step (like
> >> per script executed).
>
> >> There are probably many more, but those are the ones I knew about.
>
> >> Given that I wanted to give a try to Azure pipelines. And I did!
>
> >> I created this repo: [8]https://github.com/awesomebytes/gentoo_prefix_ci_test
>
> >> Where I activated Azure pipelines on it. After around 15min of reading the
> >> docs and playing around with the web-gui I got my first pipeline running.
>
> >> As an initial setup I thought I would create a Docker image where I bootstrap
> >> Gentoo Prefix from a Ubuntu 16.04 (as I'm familiar with it with my projects).
>
> >> The repo contains two important things:
>
> >> 1) The Dockerfile where I mainly trigger the
> >> bootstrap: [9]https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/Dockerfile
>
> >> 2) The configuration file for Azure pipelines on what to
> >> do: [10]https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/azure-pipelines.yml
>
> >> I've implemented here that it tries to build Gentoo Prefix, and whatever the
> >> result, it uploads a Docker image to my DockerHub account with the results.
> >> This implies that:
>
> >> If the bootstrap is successful, one can just [docker pull] and [docker run]
> >> the image to play with Gentoo Prefix.
>
> >> If the bootstrap is unsuccessful, one can just [docker pull] and [docker run]
> >> to find oneself in the exact state of the system after the bootstrap command.
> >> And one can recover the full console log from the Azure pipelines web
> >> interface (even tho it would be nice to find out how to post it publicly
> >> straight away).
>
> >> If all goes well in a few hours anyone will be able to find in my DockerHub
> >> account said image (most probably the failed one), just doing:
>
> >> docker pull awesomebytes/gentoo_prefix_latest_image:latest
>
> >> docker run -it gentoo_prefix_latest_image /bin/bash
>
> >> You'll be inside of a Ubuntu 16.04 box with a user called 'user' and with all
> >> the bootstrapped stuff in /tmp/gentoo.
>
> >> As curiosity, I checked the machines I got served as 'agents' to run my jobs,
> >> and they were of the kind:
>
> >> CPUs: 2x Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
>
> >> RAM: 7GB
>
> >> Disk: 94GB free disk space
>
> >> More than enough to bootstrap Gentoo Prefix!
>
> >> I don't know if this is the way to go. But at least is interesting to have it
> >> in mind.
>
> >> On Tue, Nov 27, 2018 at 12:01 AM Benda Xu <[11][hidden email]> wrote:
>
> >>> Hi Sam,
>
> >>> Sam Pfeiffer <[12][hidden email]> writes:
>
> >>> > With Azure announcing unlimited minutes on CI/CD for open source
> >>> > projects:
> >>> >
> >>> [13]https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
> >>> >
> >>> > Even bootstrapping Gentoo prefix, with pieces of software like gcc
> >>> > taking very long to compile, is possible.
> >>> >
> >>> > The point is: I have been trying to build Gentoo Prefix during the
> >>> > last days after a few months of break since the last time I touched
> >>> > the system. And it's failing. I haven't managed yet to bootstrap it
> >>> > completely. I feel there is no CI/CD setup to catch these issues and
> >>> > be able to offer a working version of Gentoo Prefix at any time.
>
> >>> I completely agree with you.  I hope you can carry on this project to
> >>> setup proper CI for Gentoo Prefix.  I am all in for help, portage/ebuild
> >>> mentoring and coorperation.
>
> >>> A CI for Gentoo Prefix has been on my list for ages.  Thank you for
> >>> triggering this.
>
> >>> Yours,
> >>> Benda
>
> >> --
>
> >> Sammy Pfeiffer
> >> PhD Candidate at The Magic Lab within UTS.
>
> > --
>
> > Sammy Pfeiffer
> > PhD Candidate at The Magic Lab within UTS.
>
> --
>
> Sammy Pfeiffer
> PhD Candidate at The Magic Lab within UTS.
>
>
>
>  References:
>    1. https://dev.azure.com/12719821/12719821/_build/results?buildId=21&amp;view=logs
>    2. https://pastebin.com/LWS3Bb7S
>    3. https://pastebin.com/gipspmnD
>    4. https://bugs.gentoo.org/672042
>    5. mailto:[hidden email]
>    6. https://dev.azure.com/12719821/12719821/_build/results?buildId=17&amp;view=logs
>    7. mailto:[hidden email]
>    8. https://github.com/awesomebytes/gentoo_prefix_ci_test
>    9. https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/Dockerfile
>   10. https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/azure-pipelines.yml
>   11. mailto:[hidden email]
>   12. mailto:[hidden email]
>   13. https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
>
> read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
--
Fabian Groffen
Gentoo on a different level

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

Sam Pfeiffer


On Tue, Nov 27, 2018 at 7:20 PM Fabian Groffen <[hidden email]> wrote:
I don't want to depress this entire discussion, but it would be really
nice if we could somehow interact with special machines people have at
their company or at home.  Prefix needs testing on many different
machines (non-Linux) which usually don't exist in docker images.

That's alright, we can use QEMU for some more esoteric hardware platforms, if it's an OS that runs on a normal amd64/x86 computer a Docker image can be built (I'm not an expert but there are images to learn how to do it). Or in the worst case we can create an old-school VM for those weird OS and automate the interaction with it (I did it for a robot by dumping all disk once and creating a VM from it, it worked ok).
 

That said, focussing on the (usually fast) boxes like this to catch
dependency problems and more is useful.  In the case below it looks like
the ld-wrapper is having issues.  Would it be possible to enter the
environment for that failed run?

Glad you see the use of it :) Yeah as I mentioned in the previous mail, having docker installed in your machine, to access that exact environment after the failed bootstrap just do:

# This will download the image to your machine (it may take a bit of time if its the first time you use docker its around 1GB of data I think)
docker pull awesomebytes/gentoo_prefix_latest_image
# This will drop you into a shell in that environment, ready to play!
docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash

When you are done you can just type exit.
 

Thanks,
Fabian

On 27-11-2018 17:14:19 +1100, Sam Pfeiffer wrote:
> Hello all,
>
> Another little update on my test, it took me a while to find the actual way to
> increase the timeout to the maximum (or in other words, get out of the default
> of 60minutes), but I finally found how.
>
> Now I have a job that just tries to bootstrap Gentoo Prefix, the last run I made
> can be found
> here: [1]https://dev.azure.com/12719821/12719821/_build/results?buildId=21&view=logs
>
> You can see all the log. In this case it failed after 1h35min. It failed
> compiling GCC 8.2.0-r4.
>
> The error was:
>
> 2018-11-27T03:20:31.7540250Z
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/xg++
> -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/
> -B/tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ -nostdinc++
> -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs
> -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs 
> -isystem
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu 
> -isystem
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include 
> -isystem
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-8.2.0/libstdc++-v3/libsupc++
> -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs
> -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs
> -no-pie -fno-PIE    -m64 -O2 -pipe -O2 -pipe -gtoggle -DIN_GCC   
>  -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing
> -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -Woverloaded-virtual
> -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings 
>  -DHAVE_CONFIG_H -static-libstdc++ -static-libgcc  gcov.o \
>
> 2018-11-27T03:20:31.7545947Z hash-table.o ggc-none.o libcommon.a
> ../libcpp/libcpp.a   ../libbacktrace/.libs/libbacktrace.a
> ../libiberty/libiberty.a ../libdecnumber/libdecnumber.a  -o gcov
>
> 2018-11-27T03:20:31.7670930Z
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/collect-ld:
> 106: exec: /tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ld: Invalid argument
>
> 2018-11-27T03:20:31.8025313Z collect2: error: ld returned 2 exit status
>
> 2018-11-27T03:20:31.8026192Z Makefile:2935: recipe for target 'gcov' failed
>
> In the end of the log we get the usual:
>
> 2018-11-27T03:23:00.8458538Z  * ERROR: sys-devel/gcc-8.2.0-r4::gentoo failed
> (compile phase):
>
> 2018-11-27T03:23:00.8480853Z  *   emake failed
>
> 2018-11-27T03:23:00.8501574Z  * 
>
> 2018-11-27T03:23:00.8524914Z  * If you need support, post the output of `emerge
> --info '=sys-devel/gcc-8.2.0-r4::gentoo'`,
>
> 2018-11-27T03:23:00.8550010Z  * the complete build log and the output of `emerge
> -pqv '=sys-devel/gcc-8.2.0-r4::gentoo'`.
>
> 2018-11-27T03:23:00.8572142Z  * The complete build log is located at
> '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/temp/build.log'.
>
> 2018-11-27T03:23:00.8593188Z  * The ebuild environment file is located at
> '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/temp/environment'.
>
> 2018-11-27T03:23:00.8622509Z  * Working directory:
> '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build'
>
> 2018-11-27T03:23:00.8642956Z  * S:
> '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-8.2.0'
>
> 2018-11-27T03:23:02.9054285Z  * 
>
> 2018-11-27T03:23:02.9078678Z  * Please include
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-build-logs.tar.bz2
> in your bug report.
>
> The cool thing comes now, how do you actually execute those commands to fill up
> a bug report?
>
> Easy, in the job I upload a Docker image with the system exactly as it was after
> the boostrap-prefix.sh command.
>
> So, if anyone wants to debug what is going on, they just need to do (this works
> right now):
>
> docker pull awesomebytes/gentoo_prefix_latest_image
>
> docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash
>
> And you are inside of the box:
>
> > user@b70d59ff9703:~$ ls
>
> > bootstrap-prefix.sh  start_bootstrap_date.txt
>
> > user@b70d59ff9703:~$ cat start_bootstrap_date.txt 
>
> > Tue Nov 27 01:52:20 UTC 2018
>
> > user@b70d59ff9703:~$ ls /tmp/gentoo
>
> > bin  etc  lib  lib64  run  sbin  stage1.log  stage2.log  stage3.log  tmp  usr 
> > var
>
> So you can do those commands suggested by emerge:
>
> /tmp/gentoo/tmp/usr/bin/emerge --info '=sys-devel/gcc-8.2.0-r4::gentoo' -->
> [2]https://pastebin.com/LWS3Bb7S
>
> /tmp/gentoo/tmp/usr/bin/emerge -pqv '=sys-devel/gcc-8.2.0-r4::gentoo' 
> --> [3]https://pastebin.com/gipspmnD
>
> etc.
>
> Or... we could even generate a new bug report automatically from this will all
> the necessary information, including the instructions on how to get into this
> box with exactly that state for anyone to help on fixing it.
>
> I'll try to dig further in with things that I find useful (and I hope you also
> find useful).
>
> Have a nice day!
>
> P.S.: With this I created a bug report easily: [4]https://bugs.gentoo.org/672042
>
> On Tue, Nov 27, 2018 at 2:07 AM Sam Pfeiffer <[5][hidden email]> wrote:
>
> > Little update: The full build log is viewable to anyone with the link, so here
> > you can see the progress of the current job: 
>
> > [6]https://dev.azure.com/12719821/12719821/_build/results?buildId=17&view=logs
>
> > (Or I should say, the log of it, for whenever you open the link).
>
> > On Tue, Nov 27, 2018 at 2:02 AM Sam Pfeiffer <[7][hidden email]>
> > wrote:
>
> >> Hello everyone,
>
> >> I'm very excited to see you are interested in adding continuous integration!
>
> >> I don't know that much about continuous integration, I've only used it (with
> >> systems already setup for me) with in-house Jenkins servers and with the ROS
> >> buildfarm, based on Travis CI on Github. Also a little bit of Gitlab CI in my
> >> lab.
>
> >> I did a bit of research/testing.
>
> >> Given it's quite a hassle to maintain custom machines, I'd try to use some of
> >> the free for opensource CI services. I've checked the conditions of a few to
> >> see which could fit better:
>
> >> * Gitlab CI: 2000 minutes / month
>
> >> * Travis CI: Unlimited minutes / month. But only 50 minutes long per step
> >> (like per script executed).
>
> >> * Azure pipelines: Unlimited minutes / month. 360 minutes long per step (like
> >> per script executed).
>
> >> There are probably many more, but those are the ones I knew about.
>
> >> Given that I wanted to give a try to Azure pipelines. And I did!
>
> >> I created this repo: [8]https://github.com/awesomebytes/gentoo_prefix_ci_test
>
> >> Where I activated Azure pipelines on it. After around 15min of reading the
> >> docs and playing around with the web-gui I got my first pipeline running.
>
> >> As an initial setup I thought I would create a Docker image where I bootstrap
> >> Gentoo Prefix from a Ubuntu 16.04 (as I'm familiar with it with my projects).
>
> >> The repo contains two important things:
>
> >> 1) The Dockerfile where I mainly trigger the
> >> bootstrap: [9]https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/Dockerfile
>
> >> 2) The configuration file for Azure pipelines on what to
> >> do: [10]https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/azure-pipelines.yml
>
> >> I've implemented here that it tries to build Gentoo Prefix, and whatever the
> >> result, it uploads a Docker image to my DockerHub account with the results.
> >> This implies that:
>
> >> If the bootstrap is successful, one can just [docker pull] and [docker run]
> >> the image to play with Gentoo Prefix.
>
> >> If the bootstrap is unsuccessful, one can just [docker pull] and [docker run]
> >> to find oneself in the exact state of the system after the bootstrap command.
> >> And one can recover the full console log from the Azure pipelines web
> >> interface (even tho it would be nice to find out how to post it publicly
> >> straight away).
>
> >> If all goes well in a few hours anyone will be able to find in my DockerHub
> >> account said image (most probably the failed one), just doing:
>
> >> docker pull awesomebytes/gentoo_prefix_latest_image:latest
>
> >> docker run -it gentoo_prefix_latest_image /bin/bash
>
> >> You'll be inside of a Ubuntu 16.04 box with a user called 'user' and with all
> >> the bootstrapped stuff in /tmp/gentoo.
>
> >> As curiosity, I checked the machines I got served as 'agents' to run my jobs,
> >> and they were of the kind:
>
> >> CPUs: 2x Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
>
> >> RAM: 7GB
>
> >> Disk: 94GB free disk space
>
> >> More than enough to bootstrap Gentoo Prefix!
>
> >> I don't know if this is the way to go. But at least is interesting to have it
> >> in mind.
>
> >> On Tue, Nov 27, 2018 at 12:01 AM Benda Xu <[11][hidden email]> wrote:
>
> >>> Hi Sam,
>
> >>> Sam Pfeiffer <[12][hidden email]> writes:
>
> >>> > With Azure announcing unlimited minutes on CI/CD for open source
> >>> > projects:
> >>> >
> >>> [13]https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
> >>> >
> >>> > Even bootstrapping Gentoo prefix, with pieces of software like gcc
> >>> > taking very long to compile, is possible.
> >>> >
> >>> > The point is: I have been trying to build Gentoo Prefix during the
> >>> > last days after a few months of break since the last time I touched
> >>> > the system. And it's failing. I haven't managed yet to bootstrap it
> >>> > completely. I feel there is no CI/CD setup to catch these issues and
> >>> > be able to offer a working version of Gentoo Prefix at any time.
>
> >>> I completely agree with you.  I hope you can carry on this project to
> >>> setup proper CI for Gentoo Prefix.  I am all in for help, portage/ebuild
> >>> mentoring and coorperation.
>
> >>> A CI for Gentoo Prefix has been on my list for ages.  Thank you for
> >>> triggering this.
>
> >>> Yours,
> >>> Benda
>
> >> --
>
> >> Sammy Pfeiffer
> >> PhD Candidate at The Magic Lab within UTS.
>
> > --
>
> > Sammy Pfeiffer
> > PhD Candidate at The Magic Lab within UTS.
>
> --
>
> Sammy Pfeiffer
> PhD Candidate at The Magic Lab within UTS.
>
>
>
>  References:
>    1. https://dev.azure.com/12719821/12719821/_build/results?buildId=21&amp;view=logs
>    2. https://pastebin.com/LWS3Bb7S
>    3. https://pastebin.com/gipspmnD
>    4. https://bugs.gentoo.org/672042
>    5. mailto:[hidden email]
>    6. https://dev.azure.com/12719821/12719821/_build/results?buildId=17&amp;view=logs
>    7. mailto:[hidden email]
>    8. https://github.com/awesomebytes/gentoo_prefix_ci_test
>    9. https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/Dockerfile
>   10. https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/azure-pipelines.yml
>   11. mailto:[hidden email]
>   12. mailto:[hidden email]
>   13. https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
>
> read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
--
Fabian Groffen
Gentoo on a different level


--

Sammy Pfeiffer
PhD Candidate at The Magic Lab within UTS.
Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

Michael Haubenwallner-3
On 11/27/2018 09:37 AM, Sam Pfeiffer wrote:
> On Tue, Nov 27, 2018 at 7:20 PM Fabian Groffen <[hidden email] <mailto:[hidden email]>> wrote:
>
> > I don't want to depress this entire discussion, but it would be really
> > nice if we could somehow interact with special machines people have at
> > their company or at home.  Prefix needs testing on many different
> > machines (non-Linux) which usually don't exist in docker images.

I second this - and let me add a further aspect here:
What I know from buildbot setup is that the master does provide (mostly shell)
commands to be executed on the slave. This is fine as long as there is limited
visibility for the master. But when a public buildbot master is being hijacked,
it feels too easy to execute malicious commands even on the slave machines.

So over a buildbot like setup, I would prefer a Jenkins like setup, where the
master does provide only trigger information to slaves. And even more appealing
would be a standalone slave setup, where the master does just receive the build
logs for the public, without access to slave machines at all.

> That's alright, we can use QEMU for some more esoteric hardware platforms,> if it's an OS that runs on a normal amd64/x86 computer a Docker image can be
> built (I'm not an expert but there are images to learn how to do it).> Or in the worst case we can create an old-school VM for those weird OS
> and automate the interaction with it (I did it for a robot by dumping all
> disk once and creating a VM from it, it worked ok).

Well... there's a bunch of OSs I fail to imagine the use of cloud driven
hardware for them, like hppa-hpux or ia64-hpux for past ones, and ppc-aix,
ppc-macos, sparc-solaris, arm-linux or m68k-mint for current ones.

> > That said, focussing on the (usually fast) boxes like this to catch
> > dependency problems and more is useful.  In the case below it looks like
> > the ld-wrapper is having issues.  Would it be possible to enter the
> > environment for that failed run?
>
> Glad you see the use of it :) Yeah as I mentioned in the previous mail,
> having docker installed in your machine, to access that exact environment
> after the failed bootstrap just do:
>
> # This will download the image to your machine (it may take a bit of time if its the first time you use docker its around 1GB of data I think)
> docker pull awesomebytes/gentoo_prefix_latest_image
> # This will drop you into a shell in that environment, ready to play!
> docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash
>
> When you are done you can just type exit.

Nevertheless, having the breaking environment as a docker image where
possible (true for the major OSs we support) really is awesome!

/haubi/

Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

Sam Pfeiffer
Hello,

Just wanted to update you up to where I got.

Now I have two working repositories:

They have continuous integration setup with Azure pipelines where every night they will try to bootstrap Gentoo Prefix on amd64 and x86 using Docker images.

As the READMEs state, this is currently done in 3 steps. This is done for two reasons. First, the 6h limit of one job running. Secondly, to be able to have intermediate state Docker images to maybe try to fix the current issues bootstrapping Gentoo Prefix in a more elegant way.

Using as an example the amd64 build:

On the releases section: https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases one can find .tar.gz files with the latest (currently done by hand, I'll automate that soon on successful builds) Gentoo Prefix that bootstrapped. It's bootstrapped in /tmp/gentoo and explained how to use it as I explained in this email thread before.

On the builds page: https://dev.azure.com/12719821/12719821/_build?definitionId=2 one can find the full logs of every step.

This fulfils my immediate needs, now I'll need to spend some time doing something similar to emerge all the stuff I need for ros_overlay and offer a binary repo. But I'm open to talk about what I did, improve it, maybe move it somewhere else... You let me know!

P.S.: Most of the work, if not all, is documented in bug #668940 and more detailed and in order in this notepad originally from Olivier Huber.
P.S.2: The help I got from the people in the IRC at #gentoo-prefix was great.


On Tue, Nov 27, 2018 at 8:21 PM Michael Haubenwallner <[hidden email]> wrote:
On 11/27/2018 09:37 AM, Sam Pfeiffer wrote:
> On Tue, Nov 27, 2018 at 7:20 PM Fabian Groffen <[hidden email] <mailto:[hidden email]>> wrote:
>
> > I don't want to depress this entire discussion, but it would be really
> > nice if we could somehow interact with special machines people have at
> > their company or at home.  Prefix needs testing on many different
> > machines (non-Linux) which usually don't exist in docker images.

I second this - and let me add a further aspect here:
What I know from buildbot setup is that the master does provide (mostly shell)
commands to be executed on the slave. This is fine as long as there is limited
visibility for the master. But when a public buildbot master is being hijacked,
it feels too easy to execute malicious commands even on the slave machines.

So over a buildbot like setup, I would prefer a Jenkins like setup, where the
master does provide only trigger information to slaves. And even more appealing
would be a standalone slave setup, where the master does just receive the build
logs for the public, without access to slave machines at all.

> That's alright, we can use QEMU for some more esoteric hardware platforms,> if it's an OS that runs on a normal amd64/x86 computer a Docker image can be
> built (I'm not an expert but there are images to learn how to do it).> Or in the worst case we can create an old-school VM for those weird OS
> and automate the interaction with it (I did it for a robot by dumping all
> disk once and creating a VM from it, it worked ok).

Well... there's a bunch of OSs I fail to imagine the use of cloud driven
hardware for them, like hppa-hpux or ia64-hpux for past ones, and ppc-aix,
ppc-macos, sparc-solaris, arm-linux or m68k-mint for current ones.

> > That said, focussing on the (usually fast) boxes like this to catch
> > dependency problems and more is useful.  In the case below it looks like
> > the ld-wrapper is having issues.  Would it be possible to enter the
> > environment for that failed run?
>
> Glad you see the use of it :) Yeah as I mentioned in the previous mail,
> having docker installed in your machine, to access that exact environment
> after the failed bootstrap just do:
>
> # This will download the image to your machine (it may take a bit of time if its the first time you use docker its around 1GB of data I think)
> docker pull awesomebytes/gentoo_prefix_latest_image
> # This will drop you into a shell in that environment, ready to play!
> docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash
>
> When you are done you can just type exit.

Nevertheless, having the breaking environment as a docker image where
possible (true for the major OSs we support) really is awesome!

/haubi/



--

Sammy Pfeiffer
PhD Candidate at The Magic Lab within UTS.
Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

Fabian Groffen-2
Cool!  Thanks a lot!

Fabian

On 03-12-2018 21:48:15 +1100, Sam Pfeiffer wrote:

> Hello,
>
> Just wanted to update you up to where I got.
>
> Now I have two working repositories:
>
> * [1]https://github.com/awesomebytes/gentoo_prefix_ci
>
> * [2]https://github.com/awesomebytes/gentoo_prefix_ci_32b
>
> They have continuous integration setup with Azure pipelines where every night
> they will try to bootstrap Gentoo Prefix on amd64 and x86 using Docker images.
>
> As the READMEs state, this is currently done in 3 steps. This is done for two
> reasons. First, the 6h limit of one job running. Secondly, to be able to have
> intermediate state Docker images to maybe try to fix the current issues
> bootstrapping Gentoo Prefix in a more elegant way.
>
> Using as an example the amd64 build:
>
> On the releases
> section: [3]https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases one
> can find .tar.gz files with the latest (currently done by hand, I'll automate
> that soon on successful builds) Gentoo Prefix that bootstrapped. It's
> bootstrapped in /tmp/gentoo and explained how to use it as I explained in this
> email thread before.
>
> On the builds
> page: [4]https://dev.azure.com/12719821/12719821/_build?definitionId=2 one can
> find the full logs of every step.
>
> This fulfils my immediate needs, now I'll need to spend some time doing
> something similar to emerge all the stuff I need for [5]ros_overlay and offer a
> binary repo. But I'm open to talk about what I did, improve it, maybe move it
> somewhere else... You let me know!
>
> P.S.: Most of the work, if not all, is documented in bug [6]#668940 and more
> detailed and in order in [7]this notepad originally from Olivier Huber.
>
> P.S.2: The help I got from the people in the IRC at #gentoo-prefix was great.
>
> On Tue, Nov 27, 2018 at 8:21 PM Michael Haubenwallner <[8][hidden email]>
> wrote:
>
> > On 11/27/2018 09:37 AM, Sam Pfeiffer wrote:
> > > On Tue, Nov 27, 2018 at 7:20 PM Fabian Groffen <[9][hidden email]
> > <mailto:[10][hidden email]>> wrote:
> > >
> > > > I don't want to depress this entire discussion, but it would be really
> > > > nice if we could somehow interact with special machines people have at
> > > > their company or at home.  Prefix needs testing on many different
> > > > machines (non-Linux) which usually don't exist in docker images.
>
> > I second this - and let me add a further aspect here:
> > What I know from buildbot setup is that the master does provide (mostly shell)
> > commands to be executed on the slave. This is fine as long as there is limited
> > visibility for the master. But when a public buildbot master is being
> > hijacked,
> > it feels too easy to execute malicious commands even on the slave machines.
>
> > So over a buildbot like setup, I would prefer a Jenkins like setup, where the
> > master does provide only trigger information to slaves. And even more
> > appealing
> > would be a standalone slave setup, where the master does just receive the
> > build
> > logs for the public, without access to slave machines at all.
>
> > > That's alright, we can use QEMU for some more esoteric hardware platforms,>
> > if it's an OS that runs on a normal amd64/x86 computer a Docker image can be
> > > built (I'm not an expert but there are images to learn how to do it).> Or in
> > the worst case we can create an old-school VM for those weird OS
> > > and automate the interaction with it (I did it for a robot by dumping all
> > > disk once and creating a VM from it, it worked ok).
>
> > Well... there's a bunch of OSs I fail to imagine the use of cloud driven
> > hardware for them, like hppa-hpux or ia64-hpux for past ones, and ppc-aix,
> > ppc-macos, sparc-solaris, arm-linux or m68k-mint for current ones.
>
> > > > That said, focussing on the (usually fast) boxes like this to catch
> > > > dependency problems and more is useful.  In the case below it looks like
> > > > the ld-wrapper is having issues.  Would it be possible to enter the
> > > > environment for that failed run?
> > >
> > > Glad you see the use of it :) Yeah as I mentioned in the previous mail,
> > > having docker installed in your machine, to access that exact environment
> > > after the failed bootstrap just do:
> > >
> > > # This will download the image to your machine (it may take a bit of time if
> > its the first time you use docker its around 1GB of data I think)
> > > docker pull awesomebytes/gentoo_prefix_latest_image
> > > # This will drop you into a shell in that environment, ready to play!
> > > docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash
> > >
> > > When you are done you can just type exit.
>
> > Nevertheless, having the breaking environment as a docker image where
> > possible (true for the major OSs we support) really is awesome!
>
> > /haubi/
>
> --
>
> Sammy Pfeiffer
> PhD Candidate at The Magic Lab within UTS.
>
>
>
>  References:
>    1. https://github.com/awesomebytes/gentoo_prefix_ci
>    2. https://github.com/awesomebytes/gentoo_prefix_ci_32b
>    3. https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
>    4. https://dev.azure.com/12719821/12719821/_build?definitionId=2
>    5. https://github.com/ros/ros-overlay
>    6. https://bugs.gentoo.org/668940
>    7. https://pad.crans.org/p/gentoo-prefix
>    8. mailto:[hidden email]
>    9. mailto:[hidden email]
>   10. mailto:[hidden email]
>
> read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
--
Fabian Groffen
Gentoo on a different level

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

Fabian Groffen-2
Hey Sam,

Examining some of your build logs, I wonder if you are aware you can do
this:

  ./bootstrap-prefix.sh ${EPREFIX} noninteractive

instead of forcing input to the script.

Perl currently fails it seems in the logs, though the bootstrap is
reported to be successful.  Is this by design?

Thanks,
Fabian

On 03-12-2018 11:53:02 +0100, Fabian Groffen wrote:

> Cool!  Thanks a lot!
>
> Fabian
>
> On 03-12-2018 21:48:15 +1100, Sam Pfeiffer wrote:
> > Hello,
> >
> > Just wanted to update you up to where I got.
> >
> > Now I have two working repositories:
> >
> > * [1]https://github.com/awesomebytes/gentoo_prefix_ci
> >
> > * [2]https://github.com/awesomebytes/gentoo_prefix_ci_32b
> >
> > They have continuous integration setup with Azure pipelines where every night
> > they will try to bootstrap Gentoo Prefix on amd64 and x86 using Docker images.
> >
> > As the READMEs state, this is currently done in 3 steps. This is done for two
> > reasons. First, the 6h limit of one job running. Secondly, to be able to have
> > intermediate state Docker images to maybe try to fix the current issues
> > bootstrapping Gentoo Prefix in a more elegant way.
> >
> > Using as an example the amd64 build:
> >
> > On the releases
> > section: [3]https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases one
> > can find .tar.gz files with the latest (currently done by hand, I'll automate
> > that soon on successful builds) Gentoo Prefix that bootstrapped. It's
> > bootstrapped in /tmp/gentoo and explained how to use it as I explained in this
> > email thread before.
> >
> > On the builds
> > page: [4]https://dev.azure.com/12719821/12719821/_build?definitionId=2 one can
> > find the full logs of every step.
> >
> > This fulfils my immediate needs, now I'll need to spend some time doing
> > something similar to emerge all the stuff I need for [5]ros_overlay and offer a
> > binary repo. But I'm open to talk about what I did, improve it, maybe move it
> > somewhere else... You let me know!
> >
> > P.S.: Most of the work, if not all, is documented in bug [6]#668940 and more
> > detailed and in order in [7]this notepad originally from Olivier Huber.
> >
> > P.S.2: The help I got from the people in the IRC at #gentoo-prefix was great.
> >
> > On Tue, Nov 27, 2018 at 8:21 PM Michael Haubenwallner <[8][hidden email]>
> > wrote:
> >
> > > On 11/27/2018 09:37 AM, Sam Pfeiffer wrote:
> > > > On Tue, Nov 27, 2018 at 7:20 PM Fabian Groffen <[9][hidden email]
> > > <mailto:[10][hidden email]>> wrote:
> > > >
> > > > > I don't want to depress this entire discussion, but it would be really
> > > > > nice if we could somehow interact with special machines people have at
> > > > > their company or at home.  Prefix needs testing on many different
> > > > > machines (non-Linux) which usually don't exist in docker images.
> >
> > > I second this - and let me add a further aspect here:
> > > What I know from buildbot setup is that the master does provide (mostly shell)
> > > commands to be executed on the slave. This is fine as long as there is limited
> > > visibility for the master. But when a public buildbot master is being
> > > hijacked,
> > > it feels too easy to execute malicious commands even on the slave machines.
> >
> > > So over a buildbot like setup, I would prefer a Jenkins like setup, where the
> > > master does provide only trigger information to slaves. And even more
> > > appealing
> > > would be a standalone slave setup, where the master does just receive the
> > > build
> > > logs for the public, without access to slave machines at all.
> >
> > > > That's alright, we can use QEMU for some more esoteric hardware platforms,>
> > > if it's an OS that runs on a normal amd64/x86 computer a Docker image can be
> > > > built (I'm not an expert but there are images to learn how to do it).> Or in
> > > the worst case we can create an old-school VM for those weird OS
> > > > and automate the interaction with it (I did it for a robot by dumping all
> > > > disk once and creating a VM from it, it worked ok).
> >
> > > Well... there's a bunch of OSs I fail to imagine the use of cloud driven
> > > hardware for them, like hppa-hpux or ia64-hpux for past ones, and ppc-aix,
> > > ppc-macos, sparc-solaris, arm-linux or m68k-mint for current ones.
> >
> > > > > That said, focussing on the (usually fast) boxes like this to catch
> > > > > dependency problems and more is useful.  In the case below it looks like
> > > > > the ld-wrapper is having issues.  Would it be possible to enter the
> > > > > environment for that failed run?
> > > >
> > > > Glad you see the use of it :) Yeah as I mentioned in the previous mail,
> > > > having docker installed in your machine, to access that exact environment
> > > > after the failed bootstrap just do:
> > > >
> > > > # This will download the image to your machine (it may take a bit of time if
> > > its the first time you use docker its around 1GB of data I think)
> > > > docker pull awesomebytes/gentoo_prefix_latest_image
> > > > # This will drop you into a shell in that environment, ready to play!
> > > > docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash
> > > >
> > > > When you are done you can just type exit.
> >
> > > Nevertheless, having the breaking environment as a docker image where
> > > possible (true for the major OSs we support) really is awesome!
> >
> > > /haubi/
> >
> > --
> >
> > Sammy Pfeiffer
> > PhD Candidate at The Magic Lab within UTS.
> >
> >
> >
> >  References:
> >    1. https://github.com/awesomebytes/gentoo_prefix_ci
> >    2. https://github.com/awesomebytes/gentoo_prefix_ci_32b
> >    3. https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
> >    4. https://dev.azure.com/12719821/12719821/_build?definitionId=2
> >    5. https://github.com/ros/ros-overlay
> >    6. https://bugs.gentoo.org/668940
> >    7. https://pad.crans.org/p/gentoo-prefix
> >    8. mailto:[hidden email]
> >    9. mailto:[hidden email]
> >   10. mailto:[hidden email]
> >
> > read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
> --
> Fabian Groffen
> Gentoo on a different level


--
Fabian Groffen
Gentoo on a different level

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

Sam Pfeiffer
Hello Fabian,

I was unaware I could do that. Looks cleaner and more convenient! Thanks!

You could say it is 'by design'. I have no idea how to fix the perl stuff, neither the other problems (9 workarounds needed in total to be able to bootstrap Gentoo Prefix, documented here https://pad.crans.org/p/gentoo-prefix and in my Dockerfiles). Given I need Gentoo Prefix bootstrappable to keep building stuff upon it... That's the best I could do.

I'd love to remove the hacks and be able to just run the bootstrap-prefix.sh script and call it a day though.

On Wed, Dec 12, 2018 at 12:40 AM Fabian Groffen <[hidden email]> wrote:
Hey Sam,

Examining some of your build logs, I wonder if you are aware you can do
this:

  ./bootstrap-prefix.sh ${EPREFIX} noninteractive

instead of forcing input to the script.

Perl currently fails it seems in the logs, though the bootstrap is
reported to be successful.  Is this by design?

Thanks,
Fabian

On 03-12-2018 11:53:02 +0100, Fabian Groffen wrote:
> Cool!  Thanks a lot!
>
> Fabian
>
> On 03-12-2018 21:48:15 +1100, Sam Pfeiffer wrote:
> > Hello,
> >
> > Just wanted to update you up to where I got.
> >
> > Now I have two working repositories:
> >
> > * [1]https://github.com/awesomebytes/gentoo_prefix_ci
> >
> > * [2]https://github.com/awesomebytes/gentoo_prefix_ci_32b
> >
> > They have continuous integration setup with Azure pipelines where every night
> > they will try to bootstrap Gentoo Prefix on amd64 and x86 using Docker images.
> >
> > As the READMEs state, this is currently done in 3 steps. This is done for two
> > reasons. First, the 6h limit of one job running. Secondly, to be able to have
> > intermediate state Docker images to maybe try to fix the current issues
> > bootstrapping Gentoo Prefix in a more elegant way.
> >
> > Using as an example the amd64 build:
> >
> > On the releases
> > section: [3]https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases one
> > can find .tar.gz files with the latest (currently done by hand, I'll automate
> > that soon on successful builds) Gentoo Prefix that bootstrapped. It's
> > bootstrapped in /tmp/gentoo and explained how to use it as I explained in this
> > email thread before.
> >
> > On the builds
> > page: [4]https://dev.azure.com/12719821/12719821/_build?definitionId=2 one can
> > find the full logs of every step.
> >
> > This fulfils my immediate needs, now I'll need to spend some time doing
> > something similar to emerge all the stuff I need for [5]ros_overlay and offer a
> > binary repo. But I'm open to talk about what I did, improve it, maybe move it
> > somewhere else... You let me know!
> >
> > P.S.: Most of the work, if not all, is documented in bug [6]#668940 and more
> > detailed and in order in [7]this notepad originally from Olivier Huber.
> >
> > P.S.2: The help I got from the people in the IRC at #gentoo-prefix was great.
> >
> > On Tue, Nov 27, 2018 at 8:21 PM Michael Haubenwallner <[8][hidden email]>
> > wrote:
> >
> > > On 11/27/2018 09:37 AM, Sam Pfeiffer wrote:
> > > > On Tue, Nov 27, 2018 at 7:20 PM Fabian Groffen <[9][hidden email]
> > > <mailto:[10][hidden email]>> wrote:
> > > >
> > > > > I don't want to depress this entire discussion, but it would be really
> > > > > nice if we could somehow interact with special machines people have at
> > > > > their company or at home.  Prefix needs testing on many different
> > > > > machines (non-Linux) which usually don't exist in docker images.
> >
> > > I second this - and let me add a further aspect here:
> > > What I know from buildbot setup is that the master does provide (mostly shell)
> > > commands to be executed on the slave. This is fine as long as there is limited
> > > visibility for the master. But when a public buildbot master is being
> > > hijacked,
> > > it feels too easy to execute malicious commands even on the slave machines.
> >
> > > So over a buildbot like setup, I would prefer a Jenkins like setup, where the
> > > master does provide only trigger information to slaves. And even more
> > > appealing
> > > would be a standalone slave setup, where the master does just receive the
> > > build
> > > logs for the public, without access to slave machines at all.
> >
> > > > That's alright, we can use QEMU for some more esoteric hardware platforms,>
> > > if it's an OS that runs on a normal amd64/x86 computer a Docker image can be
> > > > built (I'm not an expert but there are images to learn how to do it).> Or in
> > > the worst case we can create an old-school VM for those weird OS
> > > > and automate the interaction with it (I did it for a robot by dumping all
> > > > disk once and creating a VM from it, it worked ok).
> >
> > > Well... there's a bunch of OSs I fail to imagine the use of cloud driven
> > > hardware for them, like hppa-hpux or ia64-hpux for past ones, and ppc-aix,
> > > ppc-macos, sparc-solaris, arm-linux or m68k-mint for current ones.
> >
> > > > > That said, focussing on the (usually fast) boxes like this to catch
> > > > > dependency problems and more is useful.  In the case below it looks like
> > > > > the ld-wrapper is having issues.  Would it be possible to enter the
> > > > > environment for that failed run?
> > > >
> > > > Glad you see the use of it :) Yeah as I mentioned in the previous mail,
> > > > having docker installed in your machine, to access that exact environment
> > > > after the failed bootstrap just do:
> > > >
> > > > # This will download the image to your machine (it may take a bit of time if
> > > its the first time you use docker its around 1GB of data I think)
> > > > docker pull awesomebytes/gentoo_prefix_latest_image
> > > > # This will drop you into a shell in that environment, ready to play!
> > > > docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash
> > > >
> > > > When you are done you can just type exit.
> >
> > > Nevertheless, having the breaking environment as a docker image where
> > > possible (true for the major OSs we support) really is awesome!
> >
> > > /haubi/
> >
> > --
> >
> > Sammy Pfeiffer
> > PhD Candidate at The Magic Lab within UTS.
> >
> >
> >
> >  References:
> >    1. https://github.com/awesomebytes/gentoo_prefix_ci
> >    2. https://github.com/awesomebytes/gentoo_prefix_ci_32b
> >    3. https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
> >    4. https://dev.azure.com/12719821/12719821/_build?definitionId=2
> >    5. https://github.com/ros/ros-overlay
> >    6. https://bugs.gentoo.org/668940
> >    7. https://pad.crans.org/p/gentoo-prefix
> >    8. mailto:[hidden email]
> >    9. mailto:[hidden email]
> >   10. mailto:[hidden email]
> >
> > read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
> --
> Fabian Groffen
> Gentoo on a different level



--
Fabian Groffen
Gentoo on a different level


--

Sammy Pfeiffer
PhD Candidate at The Magic Lab within UTS.
Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

Fabian Groffen-2
Hi Sam,

At this point I managed to get bootstrap to succeed on ppc-macos
(minority arch), I will continue with x64-macos and x64-solaris shortly.

I thought I fixed the Perl problem, aparently not.  I'll release a new
snapshot today/tomorrow, perhaps this brings some of the fixes forward.

Thanks,
Fabian

On 12-12-2018 00:57:35 +1100, Sam Pfeiffer wrote:

> Hello Fabian,
>
> I was unaware I could do that. Looks cleaner and more convenient! Thanks!
>
> You could say it is 'by design'. I have no idea how to fix the perl stuff,
> neither the other problems (9 workarounds needed in total to be able to
> bootstrap Gentoo Prefix, documented here
> [1]https://pad.crans.org/p/gentoo-prefix and in my Dockerfiles). Given I need
> Gentoo Prefix bootstrappable to keep building stuff upon it... That's the best I
> could do.
>
> I'd love to remove the hacks and be able to just run the bootstrap-prefix.sh
> script and call it a day though.
>
> On Wed, Dec 12, 2018 at 12:40 AM Fabian Groffen <[2][hidden email]> wrote:
>
> > Hey Sam,
>
> > Examining some of your build logs, I wonder if you are aware you can do
> > this:
>
> >   ./bootstrap-prefix.sh ${EPREFIX} noninteractive
>
> > instead of forcing input to the script.
>
> > Perl currently fails it seems in the logs, though the bootstrap is
> > reported to be successful.  Is this by design?
>
> > Thanks,
> > Fabian
>
> > On 03-12-2018 11:53:02 +0100, Fabian Groffen wrote:
> > > Cool!  Thanks a lot!
> > >
> > > Fabian
> > >
> > > On 03-12-2018 21:48:15 +1100, Sam Pfeiffer wrote:
> > > > Hello,
> > > >
> > > > Just wanted to update you up to where I got.
> > > >
> > > > Now I have two working repositories:
> > > >
> > > > * [1][3]https://github.com/awesomebytes/gentoo_prefix_ci
> > > >
> > > > * [2][4]https://github.com/awesomebytes/gentoo_prefix_ci_32b
> > > >
> > > > They have continuous integration setup with Azure pipelines where every
> > night
> > > > they will try to bootstrap Gentoo Prefix on amd64 and x86 using Docker
> > images.
> > > >
> > > > As the READMEs state, this is currently done in 3 steps. This is done for
> > two
> > > > reasons. First, the 6h limit of one job running. Secondly, to be able to
> > have
> > > > intermediate state Docker images to maybe try to fix the current issues
> > > > bootstrapping Gentoo Prefix in a more elegant way.
> > > >
> > > > Using as an example the amd64 build:
> > > >
> > > > On the releases
> > > >
> > section: [3][5]https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
> > one
> > > > can find .tar.gz files with the latest (currently done by hand, I'll
> > automate
> > > > that soon on successful builds) Gentoo Prefix that bootstrapped. It's
> > > > bootstrapped in /tmp/gentoo and explained how to use it as I explained in
> > this
> > > > email thread before.
> > > >
> > > > On the builds
> > > > page: [4][6]https://dev.azure.com/12719821/12719821/_build?definitionId=2
> > one can
> > > > find the full logs of every step.
> > > >
> > > > This fulfils my immediate needs, now I'll need to spend some time doing
> > > > something similar to emerge all the stuff I need for [5]ros_overlay and
> > offer a
> > > > binary repo. But I'm open to talk about what I did, improve it, maybe move
> > it
> > > > somewhere else... You let me know!
> > > >
> > > > P.S.: Most of the work, if not all, is documented in bug [6]#668940 and
> > more
> > > > detailed and in order in [7]this notepad originally from Olivier Huber.
> > > >
> > > > P.S.2: The help I got from the people in the IRC at #gentoo-prefix was
> > great.
> > > >
> > > > On Tue, Nov 27, 2018 at 8:21 PM Michael Haubenwallner
> > <[8][7][hidden email]>
> > > > wrote:
> > > >
> > > > > On 11/27/2018 09:37 AM, Sam Pfeiffer wrote:
> > > > > > On Tue, Nov 27, 2018 at 7:20 PM Fabian Groffen
> > <[9][8][hidden email]
> > > > > <mailto:[10][9][hidden email]>> wrote:
> > > > > >
> > > > > > > I don't want to depress this entire discussion, but it would be
> > really
> > > > > > > nice if we could somehow interact with special machines people have
> > at
> > > > > > > their company or at home.  Prefix needs testing on many different
> > > > > > > machines (non-Linux) which usually don't exist in docker images.
> > > >
> > > > > I second this - and let me add a further aspect here:
> > > > > What I know from buildbot setup is that the master does provide (mostly
> > shell)
> > > > > commands to be executed on the slave. This is fine as long as there is
> > limited
> > > > > visibility for the master. But when a public buildbot master is being
> > > > > hijacked,
> > > > > it feels too easy to execute malicious commands even on the slave
> > machines.
> > > >
> > > > > So over a buildbot like setup, I would prefer a Jenkins like setup,
> > where the
> > > > > master does provide only trigger information to slaves. And even more
> > > > > appealing
> > > > > would be a standalone slave setup, where the master does just receive
> > the
> > > > > build
> > > > > logs for the public, without access to slave machines at all.
> > > >
> > > > > > That's alright, we can use QEMU for some more esoteric hardware
> > platforms,>
> > > > > if it's an OS that runs on a normal amd64/x86 computer a Docker image
> > can be
> > > > > > built (I'm not an expert but there are images to learn how to do it).>
> > Or in
> > > > > the worst case we can create an old-school VM for those weird OS
> > > > > > and automate the interaction with it (I did it for a robot by dumping
> > all
> > > > > > disk once and creating a VM from it, it worked ok).
> > > >
> > > > > Well... there's a bunch of OSs I fail to imagine the use of cloud driven
> > > > > hardware for them, like hppa-hpux or ia64-hpux for past ones, and
> > ppc-aix,
> > > > > ppc-macos, sparc-solaris, arm-linux or m68k-mint for current ones.
> > > >
> > > > > > > That said, focussing on the (usually fast) boxes like this to catch
> > > > > > > dependency problems and more is useful.  In the case below it looks
> > like
> > > > > > > the ld-wrapper is having issues.  Would it be possible to enter the
> > > > > > > environment for that failed run?
> > > > > >
> > > > > > Glad you see the use of it :) Yeah as I mentioned in the previous
> > mail,
> > > > > > having docker installed in your machine, to access that exact
> > environment
> > > > > > after the failed bootstrap just do:
> > > > > >
> > > > > > # This will download the image to your machine (it may take a bit of
> > time if
> > > > > its the first time you use docker its around 1GB of data I think)
> > > > > > docker pull awesomebytes/gentoo_prefix_latest_image
> > > > > > # This will drop you into a shell in that environment, ready to play!
> > > > > > docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash
> > > > > >
> > > > > > When you are done you can just type exit.
> > > >
> > > > > Nevertheless, having the breaking environment as a docker image where
> > > > > possible (true for the major OSs we support) really is awesome!
> > > >
> > > > > /haubi/
> > > >
> > > > --
> > > >
> > > > Sammy Pfeiffer
> > > > PhD Candidate at The Magic Lab within UTS.
> > > >
> > > >
> > > >
> > > >  References:
> > > >    1. [10]https://github.com/awesomebytes/gentoo_prefix_ci
> > > >    2. [11]https://github.com/awesomebytes/gentoo_prefix_ci_32b
> > > >    3. [12]https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
> > > >    4. [13]https://dev.azure.com/12719821/12719821/_build?definitionId=2
> > > >    5. [14]https://github.com/ros/ros-overlay
> > > >    6. [15]https://bugs.gentoo.org/668940
> > > >    7. [16]https://pad.crans.org/p/gentoo-prefix
> > > >    8. mailto:[17][hidden email]
> > > >    9. mailto:[18][hidden email]
> > > >   10. mailto:[19][hidden email]
> > > >
> > > > read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
> > > --
> > > Fabian Groffen
> > > Gentoo on a different level
>
> > --
> > Fabian Groffen
> > Gentoo on a different level
>
> --
>
> Sammy Pfeiffer
> PhD Candidate at The Magic Lab within UTS.
>
>
>
>  References:
>    1. https://pad.crans.org/p/gentoo-prefix
>    2. mailto:[hidden email]
>    3. https://github.com/awesomebytes/gentoo_prefix_ci
>    4. https://github.com/awesomebytes/gentoo_prefix_ci_32b
>    5. https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
>    6. https://dev.azure.com/12719821/12719821/_build?definitionId=2
>    7. mailto:[hidden email]
>    8. mailto:[hidden email]
>    9. mailto:[hidden email]
>   10. https://github.com/awesomebytes/gentoo_prefix_ci
>   11. https://github.com/awesomebytes/gentoo_prefix_ci_32b
>   12. https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
>   13. https://dev.azure.com/12719821/12719821/_build?definitionId=2
>   14. https://github.com/ros/ros-overlay
>   15. https://bugs.gentoo.org/668940
>   16. https://pad.crans.org/p/gentoo-prefix
>   17. mailto:[hidden email]
>   18. mailto:[hidden email]
>   19. mailto:[hidden email]
>
> read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
--
Fabian Groffen
Gentoo on a different level

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

Fabian Groffen-2
x86-solaris and x64-solaris bootstraps work, I expected the perl problem
to show up for those.  The pipeline still seems working on this.

On 11-12-2018 15:09:46 +0100, Fabian Groffen wrote:

> Hi Sam,
>
> At this point I managed to get bootstrap to succeed on ppc-macos
> (minority arch), I will continue with x64-macos and x64-solaris shortly.
>
> I thought I fixed the Perl problem, aparently not.  I'll release a new
> snapshot today/tomorrow, perhaps this brings some of the fixes forward.
>
> Thanks,
> Fabian
>
> On 12-12-2018 00:57:35 +1100, Sam Pfeiffer wrote:
> > Hello Fabian,
> >
> > I was unaware I could do that. Looks cleaner and more convenient! Thanks!
> >
> > You could say it is 'by design'. I have no idea how to fix the perl stuff,
> > neither the other problems (9 workarounds needed in total to be able to
> > bootstrap Gentoo Prefix, documented here
> > [1]https://pad.crans.org/p/gentoo-prefix and in my Dockerfiles). Given I need
> > Gentoo Prefix bootstrappable to keep building stuff upon it... That's the best I
> > could do.
> >
> > I'd love to remove the hacks and be able to just run the bootstrap-prefix.sh
> > script and call it a day though.
> >
> > On Wed, Dec 12, 2018 at 12:40 AM Fabian Groffen <[2][hidden email]> wrote:
> >
> > > Hey Sam,
> >
> > > Examining some of your build logs, I wonder if you are aware you can do
> > > this:
> >
> > >   ./bootstrap-prefix.sh ${EPREFIX} noninteractive
> >
> > > instead of forcing input to the script.
> >
> > > Perl currently fails it seems in the logs, though the bootstrap is
> > > reported to be successful.  Is this by design?
> >
> > > Thanks,
> > > Fabian
> >
> > > On 03-12-2018 11:53:02 +0100, Fabian Groffen wrote:
> > > > Cool!  Thanks a lot!
> > > >
> > > > Fabian
> > > >
> > > > On 03-12-2018 21:48:15 +1100, Sam Pfeiffer wrote:
> > > > > Hello,
> > > > >
> > > > > Just wanted to update you up to where I got.
> > > > >
> > > > > Now I have two working repositories:
> > > > >
> > > > > * [1][3]https://github.com/awesomebytes/gentoo_prefix_ci
> > > > >
> > > > > * [2][4]https://github.com/awesomebytes/gentoo_prefix_ci_32b
> > > > >
> > > > > They have continuous integration setup with Azure pipelines where every
> > > night
> > > > > they will try to bootstrap Gentoo Prefix on amd64 and x86 using Docker
> > > images.
> > > > >
> > > > > As the READMEs state, this is currently done in 3 steps. This is done for
> > > two
> > > > > reasons. First, the 6h limit of one job running. Secondly, to be able to
> > > have
> > > > > intermediate state Docker images to maybe try to fix the current issues
> > > > > bootstrapping Gentoo Prefix in a more elegant way.
> > > > >
> > > > > Using as an example the amd64 build:
> > > > >
> > > > > On the releases
> > > > >
> > > section: [3][5]https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
> > > one
> > > > > can find .tar.gz files with the latest (currently done by hand, I'll
> > > automate
> > > > > that soon on successful builds) Gentoo Prefix that bootstrapped. It's
> > > > > bootstrapped in /tmp/gentoo and explained how to use it as I explained in
> > > this
> > > > > email thread before.
> > > > >
> > > > > On the builds
> > > > > page: [4][6]https://dev.azure.com/12719821/12719821/_build?definitionId=2
> > > one can
> > > > > find the full logs of every step.
> > > > >
> > > > > This fulfils my immediate needs, now I'll need to spend some time doing
> > > > > something similar to emerge all the stuff I need for [5]ros_overlay and
> > > offer a
> > > > > binary repo. But I'm open to talk about what I did, improve it, maybe move
> > > it
> > > > > somewhere else... You let me know!
> > > > >
> > > > > P.S.: Most of the work, if not all, is documented in bug [6]#668940 and
> > > more
> > > > > detailed and in order in [7]this notepad originally from Olivier Huber.
> > > > >
> > > > > P.S.2: The help I got from the people in the IRC at #gentoo-prefix was
> > > great.
> > > > >
> > > > > On Tue, Nov 27, 2018 at 8:21 PM Michael Haubenwallner
> > > <[8][7][hidden email]>
> > > > > wrote:
> > > > >
> > > > > > On 11/27/2018 09:37 AM, Sam Pfeiffer wrote:
> > > > > > > On Tue, Nov 27, 2018 at 7:20 PM Fabian Groffen
> > > <[9][8][hidden email]
> > > > > > <mailto:[10][9][hidden email]>> wrote:
> > > > > > >
> > > > > > > > I don't want to depress this entire discussion, but it would be
> > > really
> > > > > > > > nice if we could somehow interact with special machines people have
> > > at
> > > > > > > > their company or at home.  Prefix needs testing on many different
> > > > > > > > machines (non-Linux) which usually don't exist in docker images.
> > > > >
> > > > > > I second this - and let me add a further aspect here:
> > > > > > What I know from buildbot setup is that the master does provide (mostly
> > > shell)
> > > > > > commands to be executed on the slave. This is fine as long as there is
> > > limited
> > > > > > visibility for the master. But when a public buildbot master is being
> > > > > > hijacked,
> > > > > > it feels too easy to execute malicious commands even on the slave
> > > machines.
> > > > >
> > > > > > So over a buildbot like setup, I would prefer a Jenkins like setup,
> > > where the
> > > > > > master does provide only trigger information to slaves. And even more
> > > > > > appealing
> > > > > > would be a standalone slave setup, where the master does just receive
> > > the
> > > > > > build
> > > > > > logs for the public, without access to slave machines at all.
> > > > >
> > > > > > > That's alright, we can use QEMU for some more esoteric hardware
> > > platforms,>
> > > > > > if it's an OS that runs on a normal amd64/x86 computer a Docker image
> > > can be
> > > > > > > built (I'm not an expert but there are images to learn how to do it).>
> > > Or in
> > > > > > the worst case we can create an old-school VM for those weird OS
> > > > > > > and automate the interaction with it (I did it for a robot by dumping
> > > all
> > > > > > > disk once and creating a VM from it, it worked ok).
> > > > >
> > > > > > Well... there's a bunch of OSs I fail to imagine the use of cloud driven
> > > > > > hardware for them, like hppa-hpux or ia64-hpux for past ones, and
> > > ppc-aix,
> > > > > > ppc-macos, sparc-solaris, arm-linux or m68k-mint for current ones.
> > > > >
> > > > > > > > That said, focussing on the (usually fast) boxes like this to catch
> > > > > > > > dependency problems and more is useful.  In the case below it looks
> > > like
> > > > > > > > the ld-wrapper is having issues.  Would it be possible to enter the
> > > > > > > > environment for that failed run?
> > > > > > >
> > > > > > > Glad you see the use of it :) Yeah as I mentioned in the previous
> > > mail,
> > > > > > > having docker installed in your machine, to access that exact
> > > environment
> > > > > > > after the failed bootstrap just do:
> > > > > > >
> > > > > > > # This will download the image to your machine (it may take a bit of
> > > time if
> > > > > > its the first time you use docker its around 1GB of data I think)
> > > > > > > docker pull awesomebytes/gentoo_prefix_latest_image
> > > > > > > # This will drop you into a shell in that environment, ready to play!
> > > > > > > docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash
> > > > > > >
> > > > > > > When you are done you can just type exit.
> > > > >
> > > > > > Nevertheless, having the breaking environment as a docker image where
> > > > > > possible (true for the major OSs we support) really is awesome!
> > > > >
> > > > > > /haubi/
> > > > >
> > > > > --
> > > > >
> > > > > Sammy Pfeiffer
> > > > > PhD Candidate at The Magic Lab within UTS.
> > > > >
> > > > >
> > > > >
> > > > >  References:
> > > > >    1. [10]https://github.com/awesomebytes/gentoo_prefix_ci
> > > > >    2. [11]https://github.com/awesomebytes/gentoo_prefix_ci_32b
> > > > >    3. [12]https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
> > > > >    4. [13]https://dev.azure.com/12719821/12719821/_build?definitionId=2
> > > > >    5. [14]https://github.com/ros/ros-overlay
> > > > >    6. [15]https://bugs.gentoo.org/668940
> > > > >    7. [16]https://pad.crans.org/p/gentoo-prefix
> > > > >    8. mailto:[17][hidden email]
> > > > >    9. mailto:[18][hidden email]
> > > > >   10. mailto:[19][hidden email]
> > > > >
> > > > > read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
> > > > --
> > > > Fabian Groffen
> > > > Gentoo on a different level
> >
> > > --
> > > Fabian Groffen
> > > Gentoo on a different level
> >
> > --
> >
> > Sammy Pfeiffer
> > PhD Candidate at The Magic Lab within UTS.
> >
> >
> >
> >  References:
> >    1. https://pad.crans.org/p/gentoo-prefix
> >    2. mailto:[hidden email]
> >    3. https://github.com/awesomebytes/gentoo_prefix_ci
> >    4. https://github.com/awesomebytes/gentoo_prefix_ci_32b
> >    5. https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
> >    6. https://dev.azure.com/12719821/12719821/_build?definitionId=2
> >    7. mailto:[hidden email]
> >    8. mailto:[hidden email]
> >    9. mailto:[hidden email]
> >   10. https://github.com/awesomebytes/gentoo_prefix_ci
> >   11. https://github.com/awesomebytes/gentoo_prefix_ci_32b
> >   12. https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
> >   13. https://dev.azure.com/12719821/12719821/_build?definitionId=2
> >   14. https://github.com/ros/ros-overlay
> >   15. https://bugs.gentoo.org/668940
> >   16. https://pad.crans.org/p/gentoo-prefix
> >   17. mailto:[hidden email]
> >   18. mailto:[hidden email]
> >   19. mailto:[hidden email]
> >
> > read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
> --
> Fabian Groffen
> Gentoo on a different level


--
Fabian Groffen
Gentoo on a different level

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 'Continuous Integration' for Gentoo Prefix?

Sam Pfeiffer
That's great news tho!

I've been building upon my amd64 and x86 bootstrapped images (which keep building nightly) and I've got a system that emerges 400 packages correctly over this (well, I needed to add a few hacks on the way, which I'm reporting them all and proposing patches when I know how).
Given there are 19000~ packages available, it does maybe not mean that much, but I emerge everything up to showing 3D visualization qt windows which has quite a few dependencies.



On Sat, Dec 15, 2018 at 5:37 AM Fabian Groffen <[hidden email]> wrote:
x86-solaris and x64-solaris bootstraps work, I expected the perl problem
to show up for those.  The pipeline still seems working on this.

On 11-12-2018 15:09:46 +0100, Fabian Groffen wrote:
> Hi Sam,
>
> At this point I managed to get bootstrap to succeed on ppc-macos
> (minority arch), I will continue with x64-macos and x64-solaris shortly.
>
> I thought I fixed the Perl problem, aparently not.  I'll release a new
> snapshot today/tomorrow, perhaps this brings some of the fixes forward.
>
> Thanks,
> Fabian
>
> On 12-12-2018 00:57:35 +1100, Sam Pfeiffer wrote:
> > Hello Fabian,
> >
> > I was unaware I could do that. Looks cleaner and more convenient! Thanks!
> >
> > You could say it is 'by design'. I have no idea how to fix the perl stuff,
> > neither the other problems (9 workarounds needed in total to be able to
> > bootstrap Gentoo Prefix, documented here
> > [1]https://pad.crans.org/p/gentoo-prefix and in my Dockerfiles). Given I need
> > Gentoo Prefix bootstrappable to keep building stuff upon it... That's the best I
> > could do.
> >
> > I'd love to remove the hacks and be able to just run the bootstrap-prefix.sh
> > script and call it a day though.
> >
> > On Wed, Dec 12, 2018 at 12:40 AM Fabian Groffen <[2][hidden email]> wrote:
> >
> > > Hey Sam,
> >
> > > Examining some of your build logs, I wonder if you are aware you can do
> > > this:
> >
> > >   ./bootstrap-prefix.sh ${EPREFIX} noninteractive
> >
> > > instead of forcing input to the script.
> >
> > > Perl currently fails it seems in the logs, though the bootstrap is
> > > reported to be successful.  Is this by design?
> >
> > > Thanks,
> > > Fabian
> >
> > > On 03-12-2018 11:53:02 +0100, Fabian Groffen wrote:
> > > > Cool!  Thanks a lot!
> > > >
> > > > Fabian
> > > >
> > > > On 03-12-2018 21:48:15 +1100, Sam Pfeiffer wrote:
> > > > > Hello,
> > > > >
> > > > > Just wanted to update you up to where I got.
> > > > >
> > > > > Now I have two working repositories:
> > > > >
> > > > > * [1][3]https://github.com/awesomebytes/gentoo_prefix_ci
> > > > >
> > > > > * [2][4]https://github.com/awesomebytes/gentoo_prefix_ci_32b
> > > > >
> > > > > They have continuous integration setup with Azure pipelines where every
> > > night
> > > > > they will try to bootstrap Gentoo Prefix on amd64 and x86 using Docker
> > > images.
> > > > >
> > > > > As the READMEs state, this is currently done in 3 steps. This is done for
> > > two
> > > > > reasons. First, the 6h limit of one job running. Secondly, to be able to
> > > have
> > > > > intermediate state Docker images to maybe try to fix the current issues
> > > > > bootstrapping Gentoo Prefix in a more elegant way.
> > > > >
> > > > > Using as an example the amd64 build:
> > > > >
> > > > > On the releases
> > > > >
> > > section: [3][5]https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
> > > one
> > > > > can find .tar.gz files with the latest (currently done by hand, I'll
> > > automate
> > > > > that soon on successful builds) Gentoo Prefix that bootstrapped. It's
> > > > > bootstrapped in /tmp/gentoo and explained how to use it as I explained in
> > > this
> > > > > email thread before.
> > > > >
> > > > > On the builds
> > > > > page: [4][6]https://dev.azure.com/12719821/12719821/_build?definitionId=2
> > > one can
> > > > > find the full logs of every step.
> > > > >
> > > > > This fulfils my immediate needs, now I'll need to spend some time doing
> > > > > something similar to emerge all the stuff I need for [5]ros_overlay and
> > > offer a
> > > > > binary repo. But I'm open to talk about what I did, improve it, maybe move
> > > it
> > > > > somewhere else... You let me know!
> > > > >
> > > > > P.S.: Most of the work, if not all, is documented in bug [6]#668940 and
> > > more
> > > > > detailed and in order in [7]this notepad originally from Olivier Huber.
> > > > >
> > > > > P.S.2: The help I got from the people in the IRC at #gentoo-prefix was
> > > great.
> > > > >
> > > > > On Tue, Nov 27, 2018 at 8:21 PM Michael Haubenwallner
> > > <[8][7][hidden email]>
> > > > > wrote:
> > > > >
> > > > > > On 11/27/2018 09:37 AM, Sam Pfeiffer wrote:
> > > > > > > On Tue, Nov 27, 2018 at 7:20 PM Fabian Groffen
> > > <[9][8][hidden email]
> > > > > > <mailto:[10][9][hidden email]>> wrote:
> > > > > > >
> > > > > > > > I don't want to depress this entire discussion, but it would be
> > > really
> > > > > > > > nice if we could somehow interact with special machines people have
> > > at
> > > > > > > > their company or at home.  Prefix needs testing on many different
> > > > > > > > machines (non-Linux) which usually don't exist in docker images.
> > > > >
> > > > > > I second this - and let me add a further aspect here:
> > > > > > What I know from buildbot setup is that the master does provide (mostly
> > > shell)
> > > > > > commands to be executed on the slave. This is fine as long as there is
> > > limited
> > > > > > visibility for the master. But when a public buildbot master is being
> > > > > > hijacked,
> > > > > > it feels too easy to execute malicious commands even on the slave
> > > machines.
> > > > >
> > > > > > So over a buildbot like setup, I would prefer a Jenkins like setup,
> > > where the
> > > > > > master does provide only trigger information to slaves. And even more
> > > > > > appealing
> > > > > > would be a standalone slave setup, where the master does just receive
> > > the
> > > > > > build
> > > > > > logs for the public, without access to slave machines at all.
> > > > >
> > > > > > > That's alright, we can use QEMU for some more esoteric hardware
> > > platforms,>
> > > > > > if it's an OS that runs on a normal amd64/x86 computer a Docker image
> > > can be
> > > > > > > built (I'm not an expert but there are images to learn how to do it).>
> > > Or in
> > > > > > the worst case we can create an old-school VM for those weird OS
> > > > > > > and automate the interaction with it (I did it for a robot by dumping
> > > all
> > > > > > > disk once and creating a VM from it, it worked ok).
> > > > >
> > > > > > Well... there's a bunch of OSs I fail to imagine the use of cloud driven
> > > > > > hardware for them, like hppa-hpux or ia64-hpux for past ones, and
> > > ppc-aix,
> > > > > > ppc-macos, sparc-solaris, arm-linux or m68k-mint for current ones.
> > > > >
> > > > > > > > That said, focussing on the (usually fast) boxes like this to catch
> > > > > > > > dependency problems and more is useful.  In the case below it looks
> > > like
> > > > > > > > the ld-wrapper is having issues.  Would it be possible to enter the
> > > > > > > > environment for that failed run?
> > > > > > >
> > > > > > > Glad you see the use of it :) Yeah as I mentioned in the previous
> > > mail,
> > > > > > > having docker installed in your machine, to access that exact
> > > environment
> > > > > > > after the failed bootstrap just do:
> > > > > > >
> > > > > > > # This will download the image to your machine (it may take a bit of
> > > time if
> > > > > > its the first time you use docker its around 1GB of data I think)
> > > > > > > docker pull awesomebytes/gentoo_prefix_latest_image
> > > > > > > # This will drop you into a shell in that environment, ready to play!
> > > > > > > docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash
> > > > > > >
> > > > > > > When you are done you can just type exit.
> > > > >
> > > > > > Nevertheless, having the breaking environment as a docker image where
> > > > > > possible (true for the major OSs we support) really is awesome!
> > > > >
> > > > > > /haubi/
> > > > >
> > > > > --
> > > > >
> > > > > Sammy Pfeiffer
> > > > > PhD Candidate at The Magic Lab within UTS.
> > > > >
> > > > >
> > > > >
> > > > >  References:
> > > > >    1. [10]https://github.com/awesomebytes/gentoo_prefix_ci
> > > > >    2. [11]https://github.com/awesomebytes/gentoo_prefix_ci_32b
> > > > >    3. [12]https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
> > > > >    4. [13]https://dev.azure.com/12719821/12719821/_build?definitionId=2
> > > > >    5. [14]https://github.com/ros/ros-overlay
> > > > >    6. [15]https://bugs.gentoo.org/668940
> > > > >    7. [16]https://pad.crans.org/p/gentoo-prefix
> > > > >    8. mailto:[17][hidden email]
> > > > >    9. mailto:[18][hidden email]
> > > > >   10. mailto:[19][hidden email]
> > > > >
> > > > > read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
> > > > --
> > > > Fabian Groffen
> > > > Gentoo on a different level
> >
> > > --
> > > Fabian Groffen
> > > Gentoo on a different level
> >
> > --
> >
> > Sammy Pfeiffer
> > PhD Candidate at The Magic Lab within UTS.
> >
> >
> >
> >  References:
> >    1. https://pad.crans.org/p/gentoo-prefix
> >    2. mailto:[hidden email]
> >    3. https://github.com/awesomebytes/gentoo_prefix_ci
> >    4. https://github.com/awesomebytes/gentoo_prefix_ci_32b
> >    5. https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
> >    6. https://dev.azure.com/12719821/12719821/_build?definitionId=2
> >    7. mailto:[hidden email]
> >    8. mailto:[hidden email]
> >    9. mailto:[hidden email]
> >   10. https://github.com/awesomebytes/gentoo_prefix_ci
> >   11. https://github.com/awesomebytes/gentoo_prefix_ci_32b
> >   12. https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
> >   13. https://dev.azure.com/12719821/12719821/_build?definitionId=2
> >   14. https://github.com/ros/ros-overlay
> >   15. https://bugs.gentoo.org/668940
> >   16. https://pad.crans.org/p/gentoo-prefix
> >   17. mailto:[hidden email]
> >   18. mailto:[hidden email]
> >   19. mailto:[hidden email]
> >
> > read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
> --
> Fabian Groffen
> Gentoo on a different level



--
Fabian Groffen
Gentoo on a different level


--

Sammy Pfeiffer
PhD Candidate at The Magic Lab within UTS.