Hello? Thump-thump?

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Hello? Thump-thump?

Alex Schuster
Hi there!

Well... is there anybody out there? According to
http://news.gmane.org/gmane.linux.gentoo.performance , the last post is
half a year ago. Does this mean performance is already okay for everyone?
Or does no one bother to post here, because the audience seems to be low,
and uses the gentoo-user list instead?

        Just wondering,
        Wonko

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hello? Thump-thump?

Bugzilla from fritzoid@gmail.com
Hey Alex,

I'm here, and I don't really believe that performance isn't the kind of thing
that people should consider okay.  It's good to push limits. :)

Nice to meet you,
--Fritze

On Tue, Jan 19, 2010 at 10:19 AM, Alex Schuster <[hidden email]> wrote:
Hi there!

Well... is there anybody out there? According to
http://news.gmane.org/gmane.linux.gentoo.performance , the last post is
half a year ago. Does this mean performance is already okay for everyone?
Or does no one bother to post here, because the audience seems to be low,
and uses the gentoo-user list instead?

       Just wondering,
       Wonko


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hello? Thump-thump?

Nicholas J. Michalek
In reply to this post by Alex Schuster
Roll-call thread! I'm here!

Indeed, this list has tended to have periods of dormancy... like me, we probably all forgot about it until this recent resurrection.

Rather than let it die again, I say we throw out ideas for discussion. I think we should have regular posts commenting on performance improvements (or otherwise) of recent package updates, such as openrc or glibc, or anything anyone is interested in.


-Nick

On Tue, Jan 19, 2010 at 11:19, Alex Schuster <[hidden email]> wrote:
Hi there!

Well... is there anybody out there? According to
http://news.gmane.org/gmane.linux.gentoo.performance , the last post is
half a year ago. Does this mean performance is already okay for everyone?
Or does no one bother to post here, because the audience seems to be low,
and uses the gentoo-user list instead?

       Just wondering,
       Wonko


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Bad desktop performance, I think (was: Re: Hello? Thump-thump?)

Alex Schuster
Nicholas J. Michalek writes:

> Roll-call thread! I'm here!

Cool. Hi Nicholas! Hi Fritze!

> Indeed, this list has tended to have periods of dormancy... like me, we
> probably all forgot about it until this recent resurrection.

My post took awhile until it showed up, so had already filed a bug report
about this list not really existing. Now I closed it :)

> Rather than let it die again, I say we throw out ideas for discussion.
> I think we should have regular posts commenting on performance
> improvements (or otherwise) of recent package updates, such as openrc
> or glibc, or anything anyone is interested in.

Sounds good. I'll start then with what I was about to post on gentoo-user,
when I remembered that there should be this list.

Hi there!

I am a little disappointed by the performance of my system. mplayer
sometimes stutters a little during the calculation phase of emerge -DpN
@world, swfdec-player does so even more. Well, sometimes even without
emerges, I guess this flash player is not coded so well.

My system is not the fastest, silence and low power consumption was more
important for me. Still, it should be fast enough I think. I have an AMD
Athlon(tm) Dual Core Processor 4850e CPU (using -march), 4GB of memory, an
1.5 TB drive. The whole system is encrypted (aes-xts-plain) and LVMed.
While the LVM overhead should be small, encryption of course creates some
extra load. /var/tmp/portage is an unencrypted tmpfs volume though. kernel
is 2.6.31-tuxonice. I'm running KDE4 with desktop effects enabled (running
ati-drivers), X itself takes about 30-40% of CPU time according to top.
mplayer itself needs less than 20%.

PORTAGE_NICENESS is 15, PORTAGE_IONICE_COMMAND uses ionice -c 3. I thought
that with these settings emerges should not be noticeable. mplayer shows
little stutters even with a niceness of 19. This looks wrong to me.

The kernel is configured as low-latency desktop. BTW, the kernel config is
here, just in case someone wants to have a look:
http://wonkology.org/~wonko/stuff/gentoo/config-genkernel-x86-2.6.31-
tuxonice_k8

So what I am asking is:
1) What might be wrong?
2) Tuning tricks. Tweaking the schedulers, different kernel sources. mm-
sources perhaps? But they look rather old. zen-sources? I might give them
a try. But then, I like software suspend, and tuxonice is regarded to be
better than the normal software suspend, and even with tuxonice I tend to
have problems.

Any input is appreciated. It's not a big problem, though, I could live
with that. But this is Gentoo, we like to tweak things, don't we? Let's
get this list alive again.

        Wonko

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hello? Thump-thump?

Steven Lembark
In reply to this post by Nicholas J. Michalek
On Tue, 19 Jan 2010 14:56:31 -0500
"Nicholas J. Michalek" <[hidden email]> wrote:

> anything anyone is interested in.

Anyone use Xen?

I'd appreciate anything that helped performance
there (about to dive in myself).

enjoi

--
Steven Lembark                                          85-09 90th St.
Workhorse Computing                               Woodhaven, NY, 11421
[hidden email]                                    +1 888 359 3508

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bad desktop performance, I think (was: Re: Hello? Thump-thump?)

Mansour Moufid
In reply to this post by Alex Schuster
Hello all,

On Wed, Jan 20, 2010 at 6:07 AM, Alex Schuster <[hidden email]> wrote:

> I am a little disappointed by the performance of my system. mplayer
> sometimes stutters a little during the calculation phase of emerge -DpN
> @world, swfdec-player does so even more. Well, sometimes even without
> emerges, I guess this flash player is not coded so well.
>
> My system is not the fastest, silence and low power consumption was more
> important for me. Still, it should be fast enough I think. I have an AMD
> Athlon(tm) Dual Core Processor 4850e CPU (using -march), 4GB of memory, an
> 1.5 TB drive. The whole system is encrypted (aes-xts-plain) and LVMed.
> While the LVM overhead should be small, encryption of course creates some
> extra load. /var/tmp/portage is an unencrypted tmpfs volume though. kernel
> is 2.6.31-tuxonice. I'm running KDE4 with desktop effects enabled (running
> ati-drivers), X itself takes about 30-40% of CPU time according to top.
> mplayer itself needs less than 20%.

I also use LVM encryption, and mplayer plays 720p at 30 fps just fine
with only the integrated Intel graphics and 1 GB of RAM. So, I suspect
you are just not using the full potential of your CPU.

If you haven't already: grep flags /proc/cpuinfo; add any of the
{mmx,mmext,sse,sse2,ssse3,3dnow,3dnowext} flags there along with
"custom-cpuopts" to your mplayer line in /etc/portage/package.use;
then re-emerge of course.

With 4GB if RAM you should probably also set swappiness to a lower
value than default.

That's all I can think of for now, good luck!

--
Mansour Moufid

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bad desktop performance, I think

Alex Schuster
Oh, well. It's been quite a while. I wanted to reply earlier, but then
kmail crashed and I lost the mail. ThenI was quite busy with other things,
and so on.

Mansour Moufid wrote:

> Hello all,
>
> On Wed, Jan 20, 2010 at 6:07 AM, Alex Schuster <[hidden email]>
> wrote:
> > I am a little disappointed by the performance of my system. mplayer
> > sometimes stutters a little during the calculation phase of emerge
> > -DpN @world, swfdec-player does so even more. Well, sometimes even
> > without emerges, I guess this flash player is not coded so well.
> >
> > My system is not the fastest, silence and low power consumption was
> > more important for me. Still, it should be fast enough I think. I
> > have an AMD Athlon(tm) Dual Core Processor 4850e CPU (using -march),
> > 4GB of memory, an 1.5 TB drive. The whole system is encrypted
> > (aes-xts-plain) and LVMed. While the LVM overhead should be small,
> > encryption of course creates some extra load. /var/tmp/portage is an
> > unencrypted tmpfs volume though. kernel is 2.6.31-tuxonice. I'm
> > running KDE4 with desktop effects enabled (running ati-drivers), X
> > itself takes about 30-40% of CPU time according to top. mplayer
> > itself needs less than 20%.
>
> I also use LVM encryption, and mplayer plays 720p at 30 fps just fine
> with only the integrated Intel graphics and 1 GB of RAM. So, I suspect
> you are just not using the full potential of your CPU.

Yeah, playing movies without stuttering should be no problem nowadays.

> If you haven't already: grep flags /proc/cpuinfo; add any of the
> {mmx,mmext,sse,sse2,ssse3,3dnow,3dnowext} flags there along with
> "custom-cpuopts" to your mplayer line in /etc/portage/package.use;
> then re-emerge of course.

I did that, but did not see much of a difference.
Anyways, this would be a workaround only, the real problem must be
somewhere else. With movies it's only most annoying. But also amarok is
interrupted. Any playing Quake3 when updatedb kicks in is no joy, too.

> With 4GB if RAM you should probably also set swappiness to a lower
> value than default.

Thanks, that was a good idea! This made the system more responsive when
switching desktops. The drawback, however, is when starting memory-
intensive applications, this takes really really long then.

> That's all I can think of for now, good luck!

This weekend I installed Gentoo again, this time with 64bit. And all those
problems are just gone. Even with [io]nice, emerges do not make mplayer
stutter in any way. Weird, because the setup is nearly identical - same
packages, same make.conf (except for compiler flags), nearly identical
kernel .config. I just removed 2GB, with the remaining 2GB the system is
running much better than with 2.75GB and 32bits.

Whatever - I'm happy now.

        Wonko

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Horrible performance

Alex Schuster
Me again.

I wrote here about my problems with mplayer stuttering during emerges.
Then I wrote that the problem went away when I installed Gentoo again,
moving from i686 to x86_64. But the problems are back, and worse than
ever. This is driving me crazy. CRAZY!

I have an AMD Athlon(tm) Dual Core Processor 4850e CPU, on-board Radeon HD
3200 graphics, 4GB of memory, an 1.5 TB drive. Lots of LVM volumes, all
encrypted, except for /usr/src and portage stuff. The system is ~amd64,
and I have -march=k8-sse3 in my CFLAGS. Current kernel is 2.6.34-tuxonice,
but I also tried others. I'm running KDE4 with desktop effects enabled, X
itself takes about 30-40% of CPU time according to top. After system
startup and login into KDE, 3.5G of RAM are occupied. This increases after
a while, and I need swap space. Nothing to worry about I think.

Performance does not feel too bad at first. But after a while, I cannot
even play videos during emerges. The playback stutters, sometimes I have
pauses for several seconds. As long as there is no swap space occpied,
it's not so bad I think. Maybe I have a probelm with disk I/O, and things
get much worse when swapping occurs. When I look at iotop, I see
various programs like chromium and various KDE applications appear. I
guess that's normal, but should not be noticeable. Hey, there were times
when I created a 2G tmpfs for /var/tmp/portage, with only 3G on
my 32bit system. BTW, I lowered my swappiness to 10. This helps a little I
think, because the swapping occurs later, the system is more responsive.

And it feels like things get worse and worse, it's not like there was a
specific point when I thought it's slow again. Like there were some
degragation going on - fragmentation, bitrot, I don't know. It's just how
it feels to me.

I am debugging this for some days now. I tried different kernels, from
2.6.29 to 2.6.35, including the kernel I had running after the switch to
64bit, when I thought all was fine. No change. But all kernels were
configured nearly identical, so I booted a GRML live-cd and used this
kernel .config as a template. Does not feel better.

When I thought the problem was gone, I had installed the system on my 2nd
1.5 TB drive. Meanwhile I copied the partitions back to the 1st drive, so
I suspected a difference in the drives. I use the 2nd drive for backups
(using rdiff-backup), with similar partitions, so I only have to exchange
the LVM volume group names of the two drives in order to run my system
from the 2nd one. I tried this, but it did not help.

And I have similar problems when copying data between some old PATA
drives. When I copy stuff and do a mkfs on another partition, mplayer
sometimes stutters and hangs for ten seconds. No joy. Working with KDE
sucks, switching dektops sometimes takes ages, and even now I am typing
faster than kmail can display the characters. That's with am emerge of
chromium running, with PORTAGE_NICENESS=10 and using ionice -c 3. Load is
around 8, but sometimes gets even higher. And then, load suddenly drops
back to lower values, as if somthing was blocking. Some applications
swapping, maybe.

Now I am out of ideas. I really hope someone here has one. I cannot work
with this system any more when emerges are going on.

I put my kernel config, make.conf, dmesg and such stuff to
http://www.wonkology.org/gentoo/ in case someone wants to have a look at
it. Any help is GREATLY appreciated.

        Wonko

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Horrible performance

J. Roeleveld
Hi Alex,

On Wednesday 25 August 2010 03:32:40 Alex Schuster wrote:

> Me again.
>
> I have an AMD Athlon(tm) Dual Core Processor 4850e CPU, on-board Radeon HD
> 3200 graphics, 4GB of memory, an 1.5 TB drive. Lots of LVM volumes, all
> encrypted, except for /usr/src and portage stuff. The system is ~amd64,
> and I have -march=k8-sse3 in my CFLAGS. Current kernel is 2.6.34-tuxonice,
> but I also tried others. I'm running KDE4 with desktop effects enabled, X
> itself takes about 30-40% of CPU time according to top. After system
> startup and login into KDE, 3.5G of RAM are occupied. This increases after
> a while, and I need swap space. Nothing to worry about I think.

Encrypted filesystems can cause additional with activity, but I would expect
that to remain the same over a long period.
However, how is the write and read performance on those disks?

You're running KDE4, guess you went for the default and use mysql for "app-
office/akonadi-server".

I switched to using "sqlite" for this due to issues getting it to work with
mysql. I think this might help there?

> Performance does not feel too bad at first. But after a while, I cannot
> even play videos during emerges. The playback stutters, sometimes I have
> pauses for several seconds. As long as there is no swap space occpied,
> it's not so bad I think. Maybe I have a probelm with disk I/O, and things
> get much worse when swapping occurs. When I look at iotop, I see
> various programs like chromium and various KDE applications appear. I
> guess that's normal, but should not be noticeable. Hey, there were times
> when I created a 2G tmpfs for /var/tmp/portage, with only 3G on
> my 32bit system. BTW, I lowered my swappiness to 10. This helps a little I
> think, because the swapping occurs later, the system is more responsive.

Do you also encrypt swap?
Disk I/O is, in my experience, a very common cause for "freeze-ups".
Can you test with unencrypted disks to see if the issue occurs then as well?

> And I have similar problems when copying data between some old PATA
> drives. When I copy stuff and do a mkfs on another partition, mplayer
> sometimes stutters and hangs for ten seconds. No joy. Working with KDE
> sucks, switching dektops sometimes takes ages, and even now I am typing
> faster than kmail can display the characters. That's with am emerge of
> chromium running, with PORTAGE_NICENESS=10 and using ionice -c 3. Load is
> around 8, but sometimes gets even higher. And then, load suddenly drops
> back to lower values, as if somthing was blocking. Some applications
> swapping, maybe.

Very possibly, maybe an idea to check which applications are hogging the
memory. If it is the swapping of the system, then this will be caused by the
most memory-hungry processes.

Can you post the result of: "ps axu"?
This will give an indication which processes are running and using a lot of
memory.

> Now I am out of ideas. I really hope someone here has one. I cannot work
> with this system any more when emerges are going on.

Had similar issues with a desktop machine myself, managed to kill some
"features" that I wasn't using and this solved most of the problems.

> I put my kernel config, make.conf, dmesg and such stuff to
> http://www.wonkology.org/gentoo/ in case someone wants to have a look at
> it. Any help is GREATLY appreciated.

Lets see where checking for IO-speeds and memory-usage of your apps take us :)

--
Joost

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Horrible performance

Mansour Moufid
In reply to this post by Alex Schuster
On Tue, Aug 24, 2010 at 9:32 PM, Alex Schuster <[hidden email]> wrote:
> Now I am out of ideas. I really hope someone here has one. I cannot work
> with this system any more when emerges are going on.
>
> I put my kernel config, make.conf, dmesg and such stuff to
> http://www.wonkology.org/gentoo/ in case someone wants to have a look at
> it. Any help is GREATLY appreciated.

Just a thought: why -ggdb in your CFLAGS? If you have >=gcc-4.2, try:

CFLAGS="-march=native -O2 -pipe"

Then you should re-emerge gcc itself (twice?) and then world:

# emerge --oneshot binutils gcc virtual/libc
# emerge -e world

Best of luck.

--
Mansour Moufid

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Horrible performance

Kacper Kopczyński
In reply to this post by Alex Schuster
Dnia 2010-08-25, o godz. 03:32:40
Alex Schuster <[hidden email]> napisał(a):

> Me again.
>
> I wrote here about my problems with mplayer stuttering during
> emerges. Then I wrote that the problem went away when I installed
> Gentoo again, moving from i686 to x86_64. But the problems are back,
> and worse than ever. This is driving me crazy. CRAZY!
>
> I have an AMD Athlon(tm) Dual Core Processor 4850e CPU, on-board
> Radeon HD 3200 graphics, 4GB of memory, an 1.5 TB drive. Lots of LVM
> volumes, all encrypted, except for /usr/src and portage stuff. The
> system is ~amd64, and I have -march=k8-sse3 in my CFLAGS. Current
> kernel is 2.6.34-tuxonice, but I also tried others. I'm running KDE4
> with desktop effects enabled, X itself takes about 30-40% of CPU time
> according to top. After system startup and login into KDE, 3.5G of
> RAM are occupied. This increases after a while, and I need swap
> space. Nothing to worry about I think.
>
> Performance does not feel too bad at first. But after a while, I
> cannot even play videos during emerges. The playback stutters,
> sometimes I have pauses for several seconds. As long as there is no
> swap space occpied, it's not so bad I think. Maybe I have a probelm
> with disk I/O, and things get much worse when swapping occurs. When I
> look at iotop, I see various programs like chromium and various KDE
> applications appear. I guess that's normal, but should not be
> noticeable. Hey, there were times when I created a 2G tmpfs
> for /var/tmp/portage, with only 3G on my 32bit system. BTW, I lowered
> my swappiness to 10. This helps a little I think, because the
> swapping occurs later, the system is more responsive.
>
> And it feels like things get worse and worse, it's not like there was
> a specific point when I thought it's slow again. Like there were some
> degragation going on - fragmentation, bitrot, I don't know. It's just
> how it feels to me.
>
> I am debugging this for some days now. I tried different kernels,
> from 2.6.29 to 2.6.35, including the kernel I had running after the
> switch to 64bit, when I thought all was fine. No change. But all
> kernels were configured nearly identical, so I booted a GRML live-cd
> and used this kernel .config as a template. Does not feel better.
>
> When I thought the problem was gone, I had installed the system on my
> 2nd 1.5 TB drive. Meanwhile I copied the partitions back to the 1st
> drive, so I suspected a difference in the drives. I use the 2nd drive
> for backups (using rdiff-backup), with similar partitions, so I only
> have to exchange the LVM volume group names of the two drives in
> order to run my system from the 2nd one. I tried this, but it did not
> help.
>
> And I have similar problems when copying data between some old PATA
> drives. When I copy stuff and do a mkfs on another partition, mplayer
> sometimes stutters and hangs for ten seconds. No joy. Working with
> KDE sucks, switching dektops sometimes takes ages, and even now I am
> typing faster than kmail can display the characters. That's with am
> emerge of chromium running, with PORTAGE_NICENESS=10 and using ionice
> -c 3. Load is around 8, but sometimes gets even higher. And then,
> load suddenly drops back to lower values, as if somthing was
> blocking. Some applications swapping, maybe.
>
> Now I am out of ideas. I really hope someone here has one. I cannot
> work with this system any more when emerges are going on.
>
> I put my kernel config, make.conf, dmesg and such stuff to
> http://www.wonkology.org/gentoo/ in case someone wants to have a look
> at it. Any help is GREATLY appreciated.
>
> Wonko
>

Checkout configuration of your video driver - X should not take more
than 5% of cpu when iddle (with some minor effects). Perhaps kde is not
using OpenGL? Do you have OpenGL enabled?

Amount of ram you have should be sufficent to compile everything
without access to swap space.

--
Kacper Kopczyński

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Horrible performance

Alex Schuster
In reply to this post by Mansour Moufid
Mansour Moufid writes:

> On Tue, Aug 24, 2010 at 9:32 PM, Alex Schuster <[hidden email]>
> wrote:

> Just a thought: why -ggdb in your CFLAGS? If you have >=gcc-4.2, try:
>
> CFLAGS="-march=native -O2 -pipe"

I just added this some days ago in order to give debug information for a a
bug in strigi I had reported. I forgot to take it out, but so far only few
packages were compiled with this setting. But thanks for mentioning this,
I removed the debug setting now.
But should it matter? Optimization still happens. And when these
interrupts hapen, the CPU is not at 100%.

I do not use march=native because I sometimes use another host with
distcc.

        Wonko

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Horrible performance

Alex Schuster
In reply to this post by Kacper Kopczyński
Kacper Kopczyński writes:

> Checkout configuration of your video driver - X should not take more
> than 5% of cpu when iddle (with some minor effects). Perhaps kde is not
> using OpenGL? Do you have OpenGL enabled?

I'm using ati-drivers-10.7 with xorg-server-1.7.7-r1. No special settings
in xorg.conf. I never had much success with the open source radeon
drivers. First they did not run at all, I either did not get X started, or
I only got a blank screen. One month ago then I had success with xf86-
video-ati-6.13.1, but X crashed instantly when I moved the mouse cursor
onto the KDE panel. But now I have a new KDe version, so I wil give them a
try soon.

Wow, at the moment X takes 95%. When I disable desktop effects, it's down
to 5%. When enable desktop effects again, X uses 60%.

Now I went into systemsettings -> desktop effects -> all effects, and
played around with the individual settings. When I disable the blur effect
(two lines after transparency), X usage is at less than 20%. Weird, I do
not even see any difference this effect makes. Cool, much less X usage by
disabling something I did not even notice :)

Is 15-25% X usage still too much? I have several plasmoids running, so I'm
not surprised this eats some performance. BTW, at [*] there are some
screenshots of my desktop, which did not change much since I did the
screenshots.

The system already feels better now. But this cannot be the only problem,
I sometimes (especially during large emerges) turn desktop effects off,
and when I have too much load, they are turned off automatically.

> Amount of ram you have should be sufficent to compile everything
> without access to swap space.

Damn right. I used to have less memory, and ran very memory-intensive
applications, and the system was much more responsive than now.

        Wonko

[*] http://www.wonkology.org/comp/desktop/2010-06-19/

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Horrible performance

Mansour Moufid
In reply to this post by Alex Schuster
On Wed, Aug 25, 2010 at 7:09 AM, Alex Schuster <[hidden email]> wrote:
> I just added this some days ago in order to give debug information for a a
> bug in strigi I had reported. I forgot to take it out, but so far only few
> packages were compiled with this setting. But thanks for mentioning this,
> I removed the debug setting now.
> But should it matter? Optimization still happens. And when these
> interrupts hapen, the CPU is not at 100%.
>
> I do not use march=native because I sometimes use another host with
> distcc.

Oh, I thought that might have been behind the memory issue...

But actually it sounds like you don't have direct rendering? Check
with glxinfo (from x11-apps/mesa-progs):

$ glxinfo | grep -i direct

If not then check out this thread in the forums for the correct kernel
configuration:
<http://forums.gentoo.org/viewtopic-t-840613-highlight-radeon+3200.html>

If that works you can also try building the KDE libraries with -Os to
help with memory. I don't think you should be using swap at all.

--
Mansour Moufid

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Horrible performance

Alex Schuster
In reply to this post by J. Roeleveld
J. Roeleveld writes:

> On Wednesday 25 August 2010 03:32:40 Alex Schuster wrote:

> > I have an AMD Athlon(tm) Dual Core Processor 4850e CPU, on-board
> > Radeon HD 3200 graphics, 4GB of memory, an 1.5 TB drive. Lots of LVM
> > volumes, all encrypted, except for /usr/src and portage stuff. The
> > system is ~amd64, and I have -march=k8-sse3 in my CFLAGS. Current
> > kernel is 2.6.34-tuxonice, but I also tried others. I'm running KDE4
> > with desktop effects enabled, X itself takes about 30-40% of CPU
> > time according to top. After system startup and login into KDE, 3.5G
> > of RAM are occupied. This increases after a while, and I need swap
> > space. Nothing to worry about I think.
>
> Encrypted filesystems can cause additional with activity, but I would
> expect that to remain the same over a long period.

And I just moved my PORTAGE_TMPDIR to an unencrypted partition.
Can LVM create noticeable overhead? I also resized my logical volumes a
couple of times, could this lead to some LVM fragmentation?

> However, how is the write and read performance on those disks?

Here's the output of hdparm -t for all drives, 4 times.

/dev/sda: (SATA system drive)
 Timing buffered disk reads:  118 MB in  3.08 seconds =  38.37 MB/sec
 Timing buffered disk reads:  194 MB in  3.11 seconds =  62.47 MB/sec
 Timing buffered disk reads:  322 MB in  3.01 seconds = 106.82 MB/sec
 Timing buffered disk reads:  244 MB in  3.00 seconds =  81.21 MB/sec

/dev/sdb: (PATA master)
 Timing buffered disk reads:  114 MB in  3.02 seconds =  37.70 MB/sec
 Timing buffered disk reads:  114 MB in  3.00 seconds =  37.97 MB/sec
 Timing buffered disk reads:  116 MB in  3.05 seconds =  38.06 MB/sec
 Timing buffered disk reads:  116 MB in  3.05 seconds =  38.07 MB/sec

/dev/sdc: (PATA slave)
 Timing buffered disk reads:  164 MB in  3.03 seconds =  54.21 MB/sec
 Timing buffered disk reads:  166 MB in  3.02 seconds =  55.04 MB/sec
 Timing buffered disk reads:  166 MB in  3.01 seconds =  55.10 MB/sec
 Timing buffered disk reads:  158 MB in  3.01 seconds =  52.41 MB/sec

/dev/sdd: (SATA backup drive)
 Timing buffered disk reads:  314 MB in  3.00 seconds = 104.55 MB/sec
 Timing buffered disk reads:  312 MB in  3.01 seconds = 103.67 MB/sec
 Timing buffered disk reads:  308 MB in  3.01 seconds = 102.34 MB/sec
 Timing buffered disk reads:  314 MB in  3.00 seconds = 104.60 MB/sec

The system drive throughput varies a lot, depending on other I/O.

I


> You're running KDE4, guess you went for the default and use mysql for
> "app- office/akonadi-server".
>
> I switched to using "sqlite" for this due to issues getting it to work
> with mysql. I think this might help there?

So I only have to set the sqlite use flag and remove the mysql use flag
for akonadi-server? I'm doing this now. And this also gives an example for
what is going on here:

I have just logged into KDE. I did not log out since yesterday, and when  
started a VM with vmplayer, the system swapped like crazy, I could not use
it for minutes. After this, the panel did not react any more, and the
desktop background did not redraw, so I logged out and in again. The VM
started fine now. Well, could be faster, but maybe it was okay.
Then I started answering your mail, and tried to reemerge akonadi-server,
but I had a type, so portage took a long search for akomadi-server.
meanwhile the dektop became quite unresponsive, load went high, and I made
a screenshot [*]. If you look at the top right, gkrellm shows this above
'Proc'. The first increase at the left was after I started emerge, the 2nd
at the right was after I pressed the PrtSc key.


> > Performance does not feel too bad at first. But after a while, I
> > cannot even play videos during emerges. The playback stutters,
> > sometimes I have pauses for several seconds. As long as there is no
> > swap space occpied, it's not so bad I think. Maybe I have a probelm
> > with disk I/O, and things get much worse when swapping occurs. When
> > I look at iotop, I see various programs like chromium and various
> > KDE applications appear. I guess that's normal, but should not be
> > noticeable. Hey, there were times when I created a 2G tmpfs for
> > /var/tmp/portage, with only 3G on my 32bit system. BTW, I lowered my
> > swappiness to 10. This helps a little I think, because the swapping
> > occurs later, the system is more responsive.
>
> Do you also encrypt swap?

Yes.

> Disk I/O is, in my experience, a very common cause for "freeze-ups".
> Can you test with unencrypted disks to see if the issue occurs then as
> well?

Yes, I can do this. It's some work, but I tried so much, why not this. I
have some free space, and already have written a backup script that
automatically creates LVM snapshots, decrypts them, and backs it up, so I
can do this from the running system.


> > And I have similar problems when copying data between some old PATA
> > drives. When I copy stuff and do a mkfs on another partition, mplayer
> > sometimes stutters and hangs for ten seconds. No joy. Working with
> > KDE sucks, switching dektops sometimes takes ages, and even now I am
> > typing faster than kmail can display the characters. That's with am
> > emerge of chromium running, with PORTAGE_NICENESS=10 and using
> > ionice -c 3. Load is around 8, but sometimes gets even higher. And
> > then, load suddenly drops back to lower values, as if somthing was
> > blocking. Some applications swapping, maybe.
>
> Very possibly, maybe an idea to check which applications are hogging
> the memory. If it is the swapping of the system, then this will be
> caused by the most memory-hungry processes.
>
> Can you post the result of: "ps axu"?
> This will give an indication which processes are running and using a
> lot of memory.

First, here is free -m:
             total       used       free     shared    buffers     cached
Mem:          3452       3225        226          0         54        325
-/+ buffers/cache:       2844        607
Swap:         4094        935       3159

And here the output of top, sorted by memory. I think it is a similar
output to ps axu, but more condensed and better readable via mail. The
full ps aux output, sorted by memory, is in [2].

top - 21:23:03 up 1 day, 7 min, 11 users,  load average: 0.04, 0.05, 0.02
Tasks: 418 total,   1 running, 417 sleeping,   0 stopped,   0 zombie
Cpu(s): 17.8%us, 12.7%sy, 9.9%ni, 45.9%id, 13.5%wa, 0.1%hi, 0.1%si, 0.0%st
Mem:   3534936k total,  3312312k used,   222624k free,    56264k buffers
Swap:  4193272k total,   957908k used,  3235364k free,   334048k cached

  PID USER     PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                        
25993 root     20   0  847m 475m  47m S   21 13.8  49:01.11 X                                              
26553 wonko    20   0 1312m 235m  10m S    0  6.8   0:28.32 java                                            
26925 wonko    20   0  698m 112m  18m S    0  3.2   3:30.75 kontact                                        
26961 wonko    20   0  526m  79m  13m S    0  2.3   2:16.01 chrome                                          
26566 wonko    20   0 1128m  72m  10m S    0  2.1   1:02.46 amarok                                          
27056 wonko    20   0  871m  57m  19m S    0  1.7   0:05.07 chrome                                          
30324 root     30  10  195m  55m 1136 S    0  1.6   0:09.46 emerge                                          
27051 wonko    20   0  874m  50m 8668 S    0  1.5   0:14.16 chrome                                          
26126 wonko    20   0 1028m  44m  11m S    2  1.3   9:18.94 plasma-desktop                                  
27046 wonko    20   0  878m  41m 7324 S    0  1.2   0:16.59 chrome                                          
27122 wonko    20   0  871m  36m 7660 S    0  1.0   0:40.03 chrome                                          
27036 wonko    20   0  865m  29m 7492 S    0  0.8   0:00.94 chrome                                          
27198 wonko    20   0  860m  28m 6240 S    0  0.8   0:04.76 chrome                                          
26101 wonko    20   0  427m  28m 8672 S    4  0.8   5:27.48 kwin                                            
26298 wonko    20   0  396m  26m 9.9m S    0  0.8   0:02.91 knotes                                          
27766 wonko    20   0  857m  26m 5924 S    0  0.8   0:03.03 chrome                                          
26903 root     20   0 64316  25m  288 S    0  0.7   0:00.33 screen                                          
27203 wonko    20   0  865m  24m 5960 S    0  0.7   0:04.06 chrome                                          
30226 wonko    20   0  367m  23m 9232 S    0  0.7   0:03.21 gwenview                                        
26221 wonko    20   0  609m  23m 3752 S    0  0.7   0:02.55 knotify4                                        

X takes an awful lot, then comes java, which is running only for my
tvbrowser. And many many chrome processes.

> > Now I am out of ideas. I really hope someone here has one. I cannot
> > work with this system any more when emerges are going on.
>
> Had similar issues with a desktop machine myself, managed to kill some
> "features" that I wasn't using and this solved most of the problems.

I hope I can say this soon, too.

> Lets see where checking for IO-speeds and memory-usage of your apps
> take us :)

Thanks for your help! I appreciate this very much.

        Wonko

[1] http://www.wonkology.org/comp/desktop/2010-08-25_emerge-akomadi.png
[2] http://www.wonkology.org/gentoo/

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Horrible performance

Alex Schuster
In reply to this post by Mansour Moufid
Mansour Moufid writes:

> But actually it sounds like you don't have direct rendering? Check
> with glxinfo (from x11-apps/mesa-progs):
>
> $ glxinfo | grep -i direct

Thanks, but opengl is running fine. Well, it took me quite a while until I
got it working, but finally there was a version of ati-drivers that
worked, and from then on it kept working fine.


> If that works you can also try building the KDE libraries with -Os to
> help with memory. I don't think you should be using swap at all.

Me too. So -Os would only be a workaround to make the probelm less bad. I
hope I find a better solution.

        Wonko

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Horrible performance

Alex Schuster
In reply to this post by J. Roeleveld
J. Roeleveld writes:

> Can you post the result of: "ps axu"?
> This will give an indication which processes are running and using a
> lot of memory.

The fifth colums gives the memory, right? Should this add up to the total
of the 'used' column in free -m?

Because it does not:

wonko@weird ~ $ free -m
             total       used       free     shared    buffers     cached
Mem:          3452       3117        334          0         39        279
-/+ buffers/cache:       2798        653
Swap:         4094       1472       2622

wonko@weird ~ $ total=0; for rss in $( ps aux | grep -v USER | awk '{print
$6}' ); do (( total += rss )); done; echo $(( total / 1024 ))
1984

So, I get a sum of around 2 G with ps, while free -m shows 4.5 G. Whoops?
Am I missing something here, or does it look like lots of RAM is not being
freed?

        Wonko

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Horrible performance

Alex Schuster
In reply to this post by Alex Schuster
I think I just found the problem! I decided to give the open source ATI
drivers a try again, and replaced 'fglrx' by 'radeon' in my xorg.conf. And
all is fast now. Wow. I had already forgotten how fast a desktop should
be.

I'm not yet sure all is fine. On the first start, kwin crashed and all
windows were on the first desktop, this never happened before. But it's
working on the second login. I have no direct rendering, Xorg.0.log shows
(EE) RADEON(0): [dri] RADEONDRIGetVersion failed (libdri too old). I had
this before with xorg-server-1.7.7, now I'm emerging 1.8.2. Stay tuned.

        Wonko

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Horrible performance

J. Roeleveld
In reply to this post by Alex Schuster
On Wednesday 25 August 2010 21:38:10 Alex Schuster wrote:

> J. Roeleveld writes:
> > On Wednesday 25 August 2010 03:32:40 Alex Schuster wrote:
> > > I have an AMD Athlon(tm) Dual Core Processor 4850e CPU, on-board
> > > Radeon HD 3200 graphics, 4GB of memory, an 1.5 TB drive. Lots of LVM
> > > volumes, all encrypted, except for /usr/src and portage stuff. The
> > > system is ~amd64, and I have -march=k8-sse3 in my CFLAGS. Current
> > > kernel is 2.6.34-tuxonice, but I also tried others. I'm running KDE4
> > > with desktop effects enabled, X itself takes about 30-40% of CPU
> > > time according to top. After system startup and login into KDE, 3.5G
> > > of RAM are occupied. This increases after a while, and I need swap
> > > space. Nothing to worry about I think.
> >
> > Encrypted filesystems can cause additional with activity, but I would
> > expect that to remain the same over a long period.
>
> And I just moved my PORTAGE_TMPDIR to an unencrypted partition.
> Can LVM create noticeable overhead? I also resized my logical volumes a
> couple of times, could this lead to some LVM fragmentation?

Theoretically, LVM will create an additional overhead.
But I am extensibly using LVM on all my machines and haven't noticed any
significant performance drops.

LVM-fragmentation is a definite possibility.
To defragment it, have a look at the following:

http://bisqwit.iki.fi/source/lvm2defrag.html
http://www.linuxquestions.org/questions/linux-software-2/how-do-i-lvm2-defrag-
or-move-based-on-logical-volumes-689335/

I played with the first one on an older machine once and it does work quite
nicely.

> > However, how is the write and read performance on those disks?
>
> Here's the output of hdparm -t for all drives, 4 times.
>
> /dev/sda: (SATA system drive)
>  Timing buffered disk reads:  118 MB in  3.08 seconds =  38.37 MB/sec
>  Timing buffered disk reads:  194 MB in  3.11 seconds =  62.47 MB/sec
>  Timing buffered disk reads:  322 MB in  3.01 seconds = 106.82 MB/sec
>  Timing buffered disk reads:  244 MB in  3.00 seconds =  81.21 MB/sec
>
> /dev/sdb: (PATA master)
>  Timing buffered disk reads:  114 MB in  3.02 seconds =  37.70 MB/sec
>  Timing buffered disk reads:  114 MB in  3.00 seconds =  37.97 MB/sec
>  Timing buffered disk reads:  116 MB in  3.05 seconds =  38.06 MB/sec
>  Timing buffered disk reads:  116 MB in  3.05 seconds =  38.07 MB/sec
>
> /dev/sdc: (PATA slave)
>  Timing buffered disk reads:  164 MB in  3.03 seconds =  54.21 MB/sec
>  Timing buffered disk reads:  166 MB in  3.02 seconds =  55.04 MB/sec
>  Timing buffered disk reads:  166 MB in  3.01 seconds =  55.10 MB/sec
>  Timing buffered disk reads:  158 MB in  3.01 seconds =  52.41 MB/sec
>
> /dev/sdd: (SATA backup drive)
>  Timing buffered disk reads:  314 MB in  3.00 seconds = 104.55 MB/sec
>  Timing buffered disk reads:  312 MB in  3.01 seconds = 103.67 MB/sec
>  Timing buffered disk reads:  308 MB in  3.01 seconds = 102.34 MB/sec
>  Timing buffered disk reads:  314 MB in  3.00 seconds = 104.60 MB/sec
>
> The system drive throughput varies a lot, depending on other I/O.

Those look ok to me, except that I would expect SATA-drives to be faster then
PATA drives.

> > You're running KDE4, guess you went for the default and use mysql for
> > "app- office/akonadi-server".
> >
> > I switched to using "sqlite" for this due to issues getting it to work
> > with mysql. I think this might help there?
>
> So I only have to set the sqlite use flag and remove the mysql use flag
> for akonadi-server? I'm doing this now. And this also gives an example for
> what is going on here:

And unset mysql.
There is one issue that needs to be resolved manually with getting it to work
with sqlite.
See:
http://forums.gentoo.org/viewtopic-t-834883-view-
next.html?sid=3ae77f5bfba5e006e8745eedb4b6cfc4

Here is the bit that will solve the problem:
--
$ cd ~/.local/share/akonadi
$ sqlite3 akonadi.db
sqlite> INSERT INTO ResourceTable (name, isVirtual) VALUES
('akonadi_search_resource', 1);
sqlite> .exit
--

> I have just logged into KDE. I did not log out since yesterday, and when
> started a VM with vmplayer, the system swapped like crazy, I could not use
> it for minutes. After this, the panel did not react any more, and the
> desktop background did not redraw, so I logged out and in again. The VM
> started fine now. Well, could be faster, but maybe it was okay.
> Then I started answering your mail, and tried to reemerge akonadi-server,
> but I had a type, so portage took a long search for akomadi-server.
> meanwhile the dektop became quite unresponsive, load went high, and I made
> a screenshot [*]. If you look at the top right, gkrellm shows this above
> 'Proc'. The first increase at the left was after I started emerge, the 2nd
> at the right was after I pressed the PrtSc key.

VMWare allows virtual machines to use more memory then is actually available.
Also, there are settings in VMWare (possibly enabled by default) that cause
the memory to be duplicated onto disk.
This can cause issues like you are seeing.

> > > Performance does not feel too bad at first. But after a while, I
> > > cannot even play videos during emerges. The playback stutters,
> > > sometimes I have pauses for several seconds. As long as there is no
> > > swap space occpied, it's not so bad I think. Maybe I have a probelm
> > > with disk I/O, and things get much worse when swapping occurs. When
> > > I look at iotop, I see various programs like chromium and various
> > > KDE applications appear. I guess that's normal, but should not be
> > > noticeable. Hey, there were times when I created a 2G tmpfs for
> > > /var/tmp/portage, with only 3G on my 32bit system. BTW, I lowered my
> > > swappiness to 10. This helps a little I think, because the swapping
> > > occurs later, the system is more responsive.
> >
> > Do you also encrypt swap?
>
> Yes.
>
> > Disk I/O is, in my experience, a very common cause for "freeze-ups".
> > Can you test with unencrypted disks to see if the issue occurs then as
> > well?
>
> Yes, I can do this. It's some work, but I tried so much, why not this. I
> have some free space, and already have written a backup script that
> automatically creates LVM snapshots, decrypts them, and backs it up, so I
> can do this from the running system.

Ok, am interested to see if running unencrypted actually has benefits here.

> > > And I have similar problems when copying data between some old PATA
> > > drives. When I copy stuff and do a mkfs on another partition, mplayer
> > > sometimes stutters and hangs for ten seconds. No joy. Working with
> > > KDE sucks, switching dektops sometimes takes ages, and even now I am
> > > typing faster than kmail can display the characters. That's with am
> > > emerge of chromium running, with PORTAGE_NICENESS=10 and using
> > > ionice -c 3. Load is around 8, but sometimes gets even higher. And
> > > then, load suddenly drops back to lower values, as if somthing was
> > > blocking. Some applications swapping, maybe.
> >
> > Very possibly, maybe an idea to check which applications are hogging
> > the memory. If it is the swapping of the system, then this will be
> > caused by the most memory-hungry processes.
> >
> > Can you post the result of: "ps axu"?
> > This will give an indication which processes are running and using a
> > lot of memory.
>
> First, here is free -m:
>              total       used       free     shared    buffers     cached
> Mem:          3452       3225        226          0         54        325
> -/+ buffers/cache:       2844        607
> Swap:         4094        935       3159
>
> And here the output of top, sorted by memory. I think it is a similar
> output to ps axu, but more condensed and better readable via mail. The
> full ps aux output, sorted by memory, is in [2].
>
<snipped top>
>
> X takes an awful lot, then comes java, which is running only for my
> tvbrowser. And many many chrome processes.

How many web browser windows do you have open? :)

Also, do you have file indexing enabled?

> > > Now I am out of ideas. I really hope someone here has one. I cannot
> > > work with this system any more when emerges are going on.
> >
> > Had similar issues with a desktop machine myself, managed to kill some
> > "features" that I wasn't using and this solved most of the problems.
>
> I hope I can say this soon, too.

In my experience, X uses more memory when a lot of windows are open.
And yours uses about 4 times as much as mine.
But then again, I don't have much running at the moment.

> > Lets see where checking for IO-speeds and memory-usage of your apps
> > take us :)
>
> Thanks for your help! I appreciate this very much.

I'll do my best :)

--
Joost

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Horrible performance

J. Roeleveld
In reply to this post by Alex Schuster
On Thursday 26 August 2010 01:06:59 Alex Schuster wrote:

> J. Roeleveld writes:
> > Can you post the result of: "ps axu"?
> > This will give an indication which processes are running and using a
> > lot of memory.
>
> The fifth colums gives the memory, right? Should this add up to the total
> of the 'used' column in free -m?
>
> Because it does not:
>
> wonko@weird ~ $ free -m
>              total       used       free     shared    buffers     cached
> Mem:          3452       3117        334          0         39        279
> -/+ buffers/cache:       2798        653
> Swap:         4094       1472       2622
>
> wonko@weird ~ $ total=0; for rss in $( ps aux | grep -v USER | awk '{print
> $6}' ); do (( total += rss )); done; echo $(( total / 1024 ))
> 1984
>
> So, I get a sum of around 2 G with ps, while free -m shows 4.5 G. Whoops?
> Am I missing something here, or does it look like lots of RAM is not being
> freed?
>
> Wonko

Don't forget the buffers/cache.
The 2.5G you're missing is what is used in the buffers/cache line.

At least, that is how I see it.
Also, I generally look at the percentages used for the memory to find the
memory-hogs.

And your tv-viewer (java) is using quite a bit as well.
Maybe someone with more experience with tv-viewer apps can take a look and
maybe give a few pointers?

--
Joost

12
Loading...