Calculating dependencies...: Any way to make it faster?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Calculating dependencies...: Any way to make it faster?

meino.cramer
Hi,

for my embedded systems I use gentoo. Their "harddisk"s are simple
microSDcards.

When updateing or emerging especially the "Calculation
dependencies..." is a step which needs a lot of patience of the
user (me ;).

Is there any way to make it faster or (in other words): Are there
different ways to "Calculating dependencies..." and have only chossen
the slowest one...?

What can I do to spped it up?

Best regards,
Meino

PS: This is ***NO*** complain against Gentoo, the emerge-process or
any other implizit or explizit critism!!!
I *love* Gentoo -- thats why I am using it even for my embedded
systems.

PPS: Yes, I know of crosscompiling and other way to do the hard work
on another machine. I screwed up my systems more than once in the past
with my attempts to get that working. I want to natively compile on
the embedded systems therefore.



Reply | Threaded
Open this post in threaded view
|

Re: Calculating dependencies...: Any way to make it faster?

Daniel Frey-6
On 01/23/2015 08:20 PM, [hidden email] wrote:
> Hi,
>
> for my embedded systems I use gentoo. Their "harddisk"s are simple
> microSDcards.
>
> When updateing or emerging especially the "Calculation
> dependencies..." is a step which needs a lot of patience of the
> user (me ;).

I have a QX9650 and it can be a few minutes on mine, especially on a
world update. My slower CPUs (Celerons) can take more than five minutes,
I don't even want to think about embedded.

> Is there any way to make it faster or (in other words): Are there
> different ways to "Calculating dependencies..." and have only chossen
> the slowest one...?

I'd be interested as well to know as well. It used to be it did a simple
dependency check and installed packages - then revdep-rebuild could
check for packages that need rebuilds.

It's not really an issue if you only run emerge once, but if you have to
do it several times in one session it gets old really quick. It reminds
me of waiting for Windows XP checking for updates. Just give it a half
hour, it'll figure it out. :-(

Dan

Reply | Threaded
Open this post in threaded view
|

Re: Calculating dependencies...: Any way to make it faster?

William Kenworthy
On 24/01/15 13:23, Daniel Frey wrote:

> On 01/23/2015 08:20 PM, [hidden email] wrote:
>> Hi,
>>
>> for my embedded systems I use gentoo. Their "harddisk"s are simple
>> microSDcards.
>>
>> When updateing or emerging especially the "Calculation
>> dependencies..." is a step which needs a lot of patience of the
>> user (me ;).
>
> I have a QX9650 and it can be a few minutes on mine, especially on a
> world update. My slower CPUs (Celerons) can take more than five minutes,
> I don't even want to think about embedded.
>
>> Is there any way to make it faster or (in other words): Are there
>> different ways to "Calculating dependencies..." and have only chossen
>> the slowest one...?
>
> I'd be interested as well to know as well. It used to be it did a simple
> dependency check and installed packages - then revdep-rebuild could
> check for packages that need rebuilds.
>
> It's not really an issue if you only run emerge once, but if you have to
> do it several times in one session it gets old really quick. It reminds
> me of waiting for Windows XP checking for updates. Just give it a half
> hour, it'll figure it out. :-(
>
> Dan
>

distcc can make a big difference on slow machines where you have 3 or so
hosts to throw jobs at.

ccache in particular speeds up multiple passes at an emerge.

Downside is a few packages cant use ccache and exhibit seemingly random
failures to compile but if known they can be excluded using a portage
setting.

BillK



Reply | Threaded
Open this post in threaded view
|

Re: Calculating dependencies...: Any way to make it faster?

meino.cramer
Bill Kenworthy <[hidden email]> [15-01-24 06:48]:

> On 24/01/15 13:23, Daniel Frey wrote:
> > On 01/23/2015 08:20 PM, [hidden email] wrote:
> >> Hi,
> >>
> >> for my embedded systems I use gentoo. Their "harddisk"s are simple
> >> microSDcards.
> >>
> >> When updateing or emerging especially the "Calculation
> >> dependencies..." is a step which needs a lot of patience of the
> >> user (me ;).
> >
> > I have a QX9650 and it can be a few minutes on mine, especially on a
> > world update. My slower CPUs (Celerons) can take more than five minutes,
> > I don't even want to think about embedded.
> >
> >> Is there any way to make it faster or (in other words): Are there
> >> different ways to "Calculating dependencies..." and have only chossen
> >> the slowest one...?
> >
> > I'd be interested as well to know as well. It used to be it did a simple
> > dependency check and installed packages - then revdep-rebuild could
> > check for packages that need rebuilds.
> >
> > It's not really an issue if you only run emerge once, but if you have to
> > do it several times in one session it gets old really quick. It reminds
> > me of waiting for Windows XP checking for updates. Just give it a half
> > hour, it'll figure it out. :-(
> >
> > Dan
> >
>
> distcc can make a big difference on slow machines where you have 3 or so
> hosts to throw jobs at.
>
> ccache in particular speeds up multiple passes at an emerge.
>
> Downside is a few packages cant use ccache and exhibit seemingly random
> failures to compile but if known they can be excluded using a portage
> setting.
>
> BillK
>

Hi Bill,

thank you for your reply! :)

I experimented with kinds of "not compiling it natively" like distcc,
crosscompiling and such. May be me of may be a problem with the tools/
the environment/the setup or whatever: The results were corrupted
systems every time. This costed me even more time than waiting for
"Calculationg dpeendencies...".
So I am back to natively compiling that stuff...

Best regards,
Meino



Reply | Threaded
Open this post in threaded view
|

Re: Calculating dependencies...: Any way to make it faster?

Tomas Mozes
In reply to this post by meino.cramer
On 2015-01-24 05:20, [hidden email] wrote:

> Hi,
>
> for my embedded systems I use gentoo. Their "harddisk"s are simple
> microSDcards.
>
> When updateing or emerging especially the "Calculation
> dependencies..." is a step which needs a lot of patience of the
> user (me ;).
>
> Is there any way to make it faster or (in other words): Are there
> different ways to "Calculating dependencies..." and have only chossen
> the slowest one...?
>
> What can I do to spped it up?
>
> Best regards,
> Meino
>
> PS: This is ***NO*** complain against Gentoo, the emerge-process or
> any other implizit or explizit critism!!!
> I *love* Gentoo -- thats why I am using it even for my embedded
> systems.
>
> PPS: Yes, I know of crosscompiling and other way to do the hard work
> on another machine. I screwed up my systems more than once in the past
> with my attempts to get that working. I want to natively compile on
> the embedded systems therefore.

Recent change to make portage a bit faster (or to fail faster):
https://bugs.gentoo.org/show_bug.cgi?id=536926

Binary packages:
http://thread.gmane.org/gmane.linux.gentoo.devel/94176

As far as I know, there is no magical tuning that would speed up portage
(it would be enabled by default I believe ;)).

Reply | Threaded
Open this post in threaded view
|

Re: Calculating dependencies...: Any way to make it faster?

R0b0t1
If the bottleneck is reading the information from disk you might
upgrade the SD card or use a USB drive instead, which may have better
random access performance. You could also store the portage tree on
another machine with faster storage and access it over the network. If
the bottleneck is actually calculating the dependencies, you are
probably out of luck for the immediate future.

For calculating dependencies I suspect the larger bottleneck is
reading everything from disk, seeing as the machines seem fast enough
you didn't complain too much about actually compiling. In either case
you should try to revisit distcc and cross compiling as that is the
only reliable way to speed everything up. You do not necessarily need
to use distcc with a cross compiler (the configuration most likely to
cause problems, though the wiki does address this).

Reply | Threaded
Open this post in threaded view
|

Re: Calculating dependencies...: Any way to make it faster?

Neil Bothwick
In reply to this post by Tomas Mozes
On Sat, 24 Jan 2015 07:03:30 +0100, Tomas Mozes wrote:

> Binary packages:
> http://thread.gmane.org/gmane.linux.gentoo.devel/94176

In my experience, using the -k or -K option makes emerge take longer to
calculate the package list, which makes sense as it also needs to check
the availability of packages.

You could try a lower --backtrack value, but the time saved may be more
than lost again when it causes problems.


--
Neil Bothwick

Hors d'oeuvres: 3 sandwiches cut into 40 pieces.

attachment0 (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Calculating dependencies...: Any way to make it faster?

Peter Humphrey-3
In reply to this post by meino.cramer
On Saturday 24 January 2015 06:56:16 [hidden email] wrote:

> I experimented with kinds of "not compiling it natively" like distcc,
> crosscompiling and such. May be me of may be a problem with the tools/
> the environment/the setup or whatever: The results were corrupted
> systems every time. This costed me even more time than waiting for
> "Calculationg dpeendencies...".
> So I am back to natively compiling that stuff...

Have you tried exporting your packages directory over NFS to a chroot on
another box? I do this for my little Atom box and it works a treat; I know
that at some others here do the same.

You just have to make sure that /etc/portage/... files are identical between
the two machines - apart from those few things you actually want to differ,
like MAKEOPTS and buildpkg.

Before discovering this method I tried distcc and cross-compiling and had
nothing but pain, but the nfs-packages method is straightforward and easily
understood.

--
Rgds
Peter.


Reply | Threaded
Open this post in threaded view
|

Re: Calculating dependencies...: Any way to make it faster?

Marc Stuermer
In reply to this post by meino.cramer
Am 24.01.2015 um 05:20 schrieb [hidden email]:

> Is there any way to make it faster or (in other words): Are there
> different ways to "Calculating dependencies..." and have only chossen
> the slowest one...?
>
> What can I do to spped it up?

Portage is written in Python, normally running on CPython. While CPython
is the standard, it isn't the fastest way to run Python.

You could try switching over to PyPy, which uses a JIT-compiler that
CPython doesn't have. This should get quite a big performance boost, if
portage is being able to run under PyPy, that is.

Alternatively you could try a portage replacement like Paludis, which is
being written completely in C++.

Reply | Threaded
Open this post in threaded view
|

Re: Calculating dependencies...: Any way to make it faster?

Dale-46
In reply to this post by Neil Bothwick
Neil Bothwick wrote:

> On Sat, 24 Jan 2015 07:03:30 +0100, Tomas Mozes wrote:
>
>> Binary packages:
>> http://thread.gmane.org/gmane.linux.gentoo.devel/94176
> In my experience, using the -k or -K option makes emerge take longer to
> calculate the package list, which makes sense as it also needs to check
> the availability of packages.
>
> You could try a lower --backtrack value, but the time saved may be more
> than lost again when it causes problems.
>
>


Setting the backtrack to a lower value was all I could think of too.
Other than that, I think it just needs horsepower under the hood.

This topic reminds me of when I was installing Gentoo on a 133 MHz
machine with around 256MBs of ram.  It's been a while but still, it was
slow.

Dale

:-)  :-)