Need software recommendation

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Need software recommendation

Khaosgrille
Hi,

i got a rockpro64 and a sopine cluster. I installed Gentoo on the rockpro64 and I would like to use the cluster as a compile cluster. Currently I am using Icecream (Icecc) which works fine but i would like to know if there is something better. (Is there an ez way to compile several packages at once? right now the load on the cluster is alway very low)

Sadly Torque is not Open Source anymore, OpenPbs might be an option, but i am not sure how well it interacts with portage.

Is there something else which i should look into?
Thanks
Khaosgrille





publickey - Khaosgrille@protonmail.com - 0xE78BC986.asc (2K) Download Attachment
signature.asc (523 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Need software recommendation

George Robb
Hope your day is going awesome.

Peek at SLURM and SPACK both were game changers for our cluster.

Hope this helps a tiny bit.  As to the distcc have you looked at distcc?

Happy computing!

On Thu, May 30, 2019 at 9:32 AM Khaosgrille <[hidden email]> wrote:
Hi,

i got a rockpro64 and a sopine cluster. I installed Gentoo on the rockpro64 and I would like to use the cluster as a compile cluster. Currently I am using Icecream (Icecc) which works fine but i would like to know if there is something better. (Is there an ez way to compile several packages at once? right now the load on the cluster is alway very low)

Sadly Torque is not Open Source anymore, OpenPbs might be an option, but i am not sure how well it interacts with portage.

Is there something else which i should look into?
Thanks
Khaosgrille




Reply | Threaded
Open this post in threaded view
|

Re: Need software recommendation

Khaosgrille
Thank you a lot!

I will take a look at them. Icecream is based on distcc but is much easier to use for example it distributes the right gcc version on its own.

SPACK looks interesting but currently I wanna keep using portage. Maybe I  figure out how to use SLURM with portage.

Thank you again

Khaosgrille



‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, May 30, 2019 4:42 PM, Robb III, George B. <[hidden email]> wrote:

Hope your day is going awesome.

Peek at SLURM and SPACK both were game changers for our cluster.

Hope this helps a tiny bit.  As to the distcc have you looked at distcc?

Happy computing!

On Thu, May 30, 2019 at 9:32 AM Khaosgrille <[hidden email]> wrote:
Hi,

i got a rockpro64 and a sopine cluster. I installed Gentoo on the rockpro64 and I would like to use the cluster as a compile cluster. Currently I am using Icecream (Icecc) which works fine but i would like to know if there is something better. (Is there an ez way to compile several packages at once? right now the load on the cluster is alway very low)

Sadly Torque is not Open Source anymore, OpenPbs might be an option, but i am not sure how well it interacts with portage.

Is there something else which i should look into?
Thanks
Khaosgrille





publickey - Khaosgrille@protonmail.com - 0xE78BC986.asc (2K) Download Attachment
signature.asc (523 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Need software recommendation

Oliver Freyermuth
In reply to this post by Khaosgrille
Hi,

Am 30.05.19 um 16:32 schrieb Khaosgrille:
> Hi,
>
> i got a rockpro64 and a sopine cluster. I installed Gentoo on the rockpro64 and I would like to use the cluster as a compile cluster. Currently I am using Icecream (Icecc) which works fine but i would like to know if there is something better. (Is there an ez way to compile several packages at once? right now the load on the cluster is alway very low)

this might be trivial:
In addition to MAKEOPTS="-jX" which applies to each package singularly, you can also use "emerge -jX"
(or put that in EMERGE_DEFAULT_OPTS) to have several packages configure and compile at the same time in parallel,
as long as the dependency tree allows for it.
You may also be interested in "--load-average=X" for emerge / "-lX" for MAKEOPTS to prevent either one of these from making the load explode on the local node.

If you are looking at a cluster workload manager, I can also recommend HTCondor. But integration with portage is something I have never done with neither SLURM nor HTCondor,
and my guess would be that icecc with high enough "-j" on both emerge and makeopts can nicely load your cluster ;-).

Cheers,
        Oliver

>
> Sadly Torque is not Open Source anymore, OpenPbs might be an option, but i am not sure how well it interacts with portage.
>
> Is there something else which i should look into?
> Thanks
> Khaosgrille
>
>
>
>


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Need software recommendation

René Neumaier
On 30.05.19 18:03, Oliver Freyermuth wrote:
> Am 30.05.19 um 16:32 schrieb Khaosgrille:
>> Hi,
>>
>> i got a rockpro64 and a sopine cluster. I installed Gentoo on the rockpro64 and I would like to use the cluster as a compile cluster. Currently I am using Icecream (Icecc) which works fine but i would like to know if there is something better. (Is there an ez way to compile several packages at once? right now the load on the cluster is alway very low)
> this might be trivial:
> In addition to MAKEOPTS="-jX" which applies to each package singularly, you can also use "emerge -jX"
> (or put that in EMERGE_DEFAULT_OPTS) to have several packages configure and compile at the same time in parallel,
> as long as the dependency tree allows for it.
> You may also be interested in "--load-average=X" for emerge / "-lX" for MAKEOPTS to prevent either one of these from making the load explode on the local node.
That's right!

@Khaosgrille
Are you sure a "Workload manager" is needed? Normally this kind of
software is just needed in case you're having different and concurrent
users or jobs on the system.
Slurm etc. ... are not for parallelism... they just implement a queue to
prevent an overcommitting of the CPUs/Mem. . But if you and your jobs
are alone... there is no benefit (except for learning purposes).
Also, the real parallelism technique (behind Slurm... etc.) is called MPI.
Have a look at: https://www.open-mpi.org/ (mpirun -np ...)

Anyway...
As far as I know distcc and others working completely different. There
is no way of combining MPI and distcc.
MPI (e.g. openmpi), OpenMP, pthreads are different concepts. And 'dist'
in 'cc' means there is already a distribution logic behind.
First, checkout what Oliver Freyermuth wrotes.


Best,
René


--
_____________________________
GPG-Fingerprint:
EC0E B6F6 B3FF 6324 B0C8 9452 EF6B 4E3C 2E59 F5AA


signature.asc (849 bytes) Download Attachment