How to determine 'Least Common Denominator' between Xen(Server) Hosts

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

How to determine 'Least Common Denominator' between Xen(Server) Hosts

Pandu Poluan

Hello list!

My company has 2 HP DL585 G5 servers and 5 Dell R... something servers. All using AMD processors. They currently are acting as XenServer hosts.

How do I determine the 'least common denominator' for Gentoo VMs (running as XenServer guests), especially for gcc flags?

I know that the (theoretical) best performance is to use -march=native , but since the processors of the HP servers are not exactly the same as the Dell's, I'm concerned that compiling with -march=native will render the VMs unable to migrate between the different hosts.

Note: Yes I know the HP servers are much older than the Dell ones, but if I go -march=native then perform an emerge when the guest is on the Dell host, the guest VM might not be able to migrate to the older HPs.

Rgds,
--

Reply | Threaded
Open this post in threaded view
|

Re: How to determine 'Least Common Denominator' between Xen(Server) Hosts

the guard
On 08/14/13 09:18, Pandu Poluan wrote:

> Hello list!
>
> My company has 2 HP DL585 G5 servers and 5 Dell R... something servers.
> All using AMD processors. They currently are acting as XenServer hosts.
>
> How do I determine the 'least common denominator' for Gentoo VMs
> (running as XenServer guests), especially for gcc flags?
>
> I know that the (theoretical) best performance is to use -march=native ,
> but since the processors of the HP servers are not exactly the same as
> the Dell's, I'm concerned that compiling with -march=native will render
> the VMs unable to migrate between the different hosts.
>
> Note: Yes I know the HP servers are much older than the Dell ones, but
> if I go -march=native then perform an emerge when the guest is on the
> Dell host, the guest VM might not be able to migrate to the older HPs.
>
> Rgds,
> --
>
-march=i386?

--
Stop talking and start compiling.
Linux user #557897

Reply | Threaded
Open this post in threaded view
|

Re: How to determine 'Least Common Denominator' between Xen(Server) Hosts

J. Roeleveld
In reply to this post by Pandu Poluan
Pandu.

I don't have the link handy. But there is a list of safe CFLAGS. Pick the one for the older CPU.
That should be safe.

--
Joost

Pandu Poluan <[hidden email]> wrote:

Hello list!

My company has 2 HP DL585 G5 servers and 5 Dell R... something servers. All using AMD processors. They currently are acting as XenServer hosts.

How do I determine the 'least common denominator' for Gentoo VMs (running as XenServer guests), especially for gcc flags?

I know that the (theoretical) best performance is to use -march=native , but since the processors of the HP servers are not exactly the same as the Dell's, I'm concerned that compiling with -march=native will render the VMs unable to migrate between the different hosts.

Note: Yes I know the HP servers are much older than the Dell ones, but if I go -march=native then perform an emerge when the guest is on the Dell host, the guest VM might not be able to migrate to the older HPs.

Rgds,
--


--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.
Reply | Threaded
Open this post in threaded view
|

Re: How to determine 'Least Common Denominator' between Xen(Server) Hosts

Helmut Jarausch
In reply to this post by Pandu Poluan
On 08/14/2013 07:18:41 AM, Pandu Poluan wrote:

> Hello list!
>
> My company has 2 HP DL585 G5 servers and 5 Dell R... something  
> servers. All
> using AMD processors. They currently are acting as XenServer hosts.
>
> How do I determine the 'least common denominator' for Gentoo VMs  
> (running
> as XenServer guests), especially for gcc flags?
>
> I know that the (theoretical) best performance is to use  
> -march=native ,
> but since the processors of the HP servers are not exactly the same  
> as the
> Dell's, I'm concerned that compiling with -march=native will render  
> the VMs
> unable to migrate between the different hosts.
>
> Note: Yes I know the HP servers are much older than the Dell ones,  
> but if I
> go -march=native then perform an emerge when the guest is on the Dell  
> host,
> the guest VM might not be able to migrate to the older HPs.
>
> Rgds,
> --

Why not compute it yourself?

Do   cat /proc/cpuinfo on all machines and compute the intersection of  
all "flags"
Then enter something like the following into /etc/portage/make.conf

CFLAGS="-O3 -pipe -msse -msse2 -msse3 -msse4a -m3dnow"

Helmut


Reply | Threaded
Open this post in threaded view
|

Re: How to determine 'Least Common Denominator' between Xen(Server) Hosts

Wang Xuerui
2013/8/14 Helmut Jarausch <[hidden email]>:
> Why not compute it yourself?
>
> Do   cat /proc/cpuinfo on all machines and compute the intersection of all
> "flags"
> Then enter something like the following into /etc/portage/make.conf
>
> CFLAGS="-O3 -pipe -msse -msse2 -msse3 -msse4a -m3dnow"


Or even better, directly intersect the GCC flags which can be obtained
in this way:

echo | gcc -O2 -pipe -march=native -E -v - 2>&1 | grep cc1

Also, according to the Gentoo Handbook going -O3 for the entire system
may cause instability and other problems. Has the situation changed
over the years?

Reply | Threaded
Open this post in threaded view
|

Re: How to determine 'Least Common Denominator' between Xen(Server) Hosts

Bruce Hill, Jr.
In reply to this post by Pandu Poluan
On Wed, Aug 14, 2013 at 12:18:41PM +0700, Pandu Poluan wrote:

> Hello list!
>
> My company has 2 HP DL585 G5 servers and 5 Dell R... something servers. All
> using AMD processors. They currently are acting as XenServer hosts.
>
> How do I determine the 'least common denominator' for Gentoo VMs (running
> as XenServer guests), especially for gcc flags?
>
> I know that the (theoretical) best performance is to use -march=native ,
> but since the processors of the HP servers are not exactly the same as the
> Dell's, I'm concerned that compiling with -march=native will render the VMs
> unable to migrate between the different hosts.
>
> Note: Yes I know the HP servers are much older than the Dell ones, but if I
> go -march=native then perform an emerge when the guest is on the Dell host,
> the guest VM might not be able to migrate to the older HPs.

To check what options CFLAGS set as -march=native would use:
gcc -march=native -E -v - </dev/null 2>&1 | sed -n 's/.* -v - //p'
(the first thing in the output is what CPU -march=native would enable)

Then you can run:
diff -u <(gcc -Q --help=target) <(gcc -march=native -Q --help=target)
to display target-specific options, versus native ones.

Assuming the 2 HP servers are identical, and the 5 Dell servers are identical,
you then only need to get the commonality of two processors (HP and Dell).
Since they're both AMD, you should have a good set of common features to help
you determine that "least common denominator", or target.
--
Happy Penguin Computers               >')
126 Fenco Drive                       ( \
Tupelo, MS 38801                       ^^
[hidden email]
662-269-2706 662-205-6424
http://happypenguincomputers.com/

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting

Reply | Threaded
Open this post in threaded view
|

Re: How to determine 'Least Common Denominator' between Xen(Server) Hosts

Paul Hartman-3
In reply to this post by Pandu Poluan

On Wed, Aug 14, 2013 at 12:18 AM, Pandu Poluan <[hidden email]> wrote:
I know that the (theoretical) best performance is to use -march=native , but since the processors of the HP servers are not exactly the same as the Dell's, I'm concerned that compiling with -march=native will render the VMs unable to migrate between the different hosts.


I use -mtune=native rather than -march=native, that way I can use some advanced processor features if they are available, but my system will still run if moved to a different host.
Reply | Threaded
Open this post in threaded view
|

Re: How to determine 'Least Common Denominator' between Xen(Server) Hosts

Pandu Poluan
In reply to this post by Wang Xuerui
Thanks, Wang Xuerui and Bruce!

That's exactly helpful.

I'm going to do some testing Real Soon.


Rgds,
--



On Wed, Aug 14, 2013 at 1:28 PM, Wang Xuerui <[hidden email]> wrote:
2013/8/14 Helmut Jarausch <[hidden email]>:
> Why not compute it yourself?
>
> Do   cat /proc/cpuinfo on all machines and compute the intersection of all
> "flags"
> Then enter something like the following into /etc/portage/make.conf
>
> CFLAGS="-O3 -pipe -msse -msse2 -msse3 -msse4a -m3dnow"


Or even better, directly intersect the GCC flags which can be obtained
in this way:

echo | gcc -O2 -pipe -march=native -E -v - 2>&1 | grep cc1

Also, according to the Gentoo Handbook going -O3 for the entire system
may cause instability and other problems. Has the situation changed
over the years?




--
FdS Pandu E Poluan
~ IT Optimizer ~

 • LOPSA Member #15248
 • Blog : http://pepoluan.tumblr.com
 • Linked-In : http://id.linkedin.com/in/pepoluan

Reply | Threaded
Open this post in threaded view
|

Re: How to determine 'Least Common Denominator' between Xen(Server) Hosts

Kerin Millar-2
In reply to this post by Bruce Hill, Jr.
On 14/08/2013 13:15, Bruce Hill wrote:

> On Wed, Aug 14, 2013 at 12:18:41PM +0700, Pandu Poluan wrote:
>> Hello list!
>>
>> My company has 2 HP DL585 G5 servers and 5 Dell R... something servers. All
>> using AMD processors. They currently are acting as XenServer hosts.
>>
>> How do I determine the 'least common denominator' for Gentoo VMs (running
>> as XenServer guests), especially for gcc flags?
>>
>> I know that the (theoretical) best performance is to use -march=native ,
>> but since the processors of the HP servers are not exactly the same as the
>> Dell's, I'm concerned that compiling with -march=native will render the VMs
>> unable to migrate between the different hosts.

A couple of points:

* The effect of setting -march=native depends on the characteristics of
   the CPU (be it virtual or otherwise)
* The characteristics of the vCPU are defined by qemu's -cpu parameter
* qemu can emulate features not implemented by the host CPU (at a cost)

One way to go about it is to start qemu with a -cpu model that exposes
features that all of your host CPUs have in common (or a subset
thereof). In that case, -march=native is fine because all of the
features that it detects as being available will be supported in
hardware on the host side.

Another way is to expose the host CPU fully with "-cpu host" and to
define your guest CFLAGS according to the most optimal subset. If you
are looking for a 'perfect' configuration then this this would be the
most effective method, if applied correctly.

Irrespective of the method, by examining /proc/cpuinfo and using the
diff technique mentioned by Bruce, you should be able to determine the
optimal configuration.

Finally, in cases where the host CPUs differ significantly - in that
native would imply a different -march value - you may choose to augment
your CFLAGS with -mtune=generic to even out performance across the
board. I don't think this would apply to you though.

>>
>> Note: Yes I know the HP servers are much older than the Dell ones, but if I
>> go -march=native then perform an emerge when the guest is on the Dell host,
>> the guest VM might not be able to migrate to the older HPs.
>
> To check what options CFLAGS set as -march=native would use:
> gcc -march=native -E -v - </dev/null 2>&1 | sed -n 's/.* -v - //p'
> (the first thing in the output is what CPU -march=native would enable)
>
> Then you can run:
> diff -u <(gcc -Q --help=target) <(gcc -march=native -Q --help=target)
> to display target-specific options, versus native ones.
> Assuming the 2 HP servers are identical, and the 5 Dell servers are identical,
> you then only need to get the commonality of two processors (HP and Dell).
> Since they're both AMD, you should have a good set of common features to help
> you determine that "least common denominator", or target.
>

Reply | Threaded
Open this post in threaded view
|

Re: How to determine 'Least Common Denominator' between Xen(Server) Hosts

Kerin Millar-2
In reply to this post by Paul Hartman-3
On 14/08/2013 16:23, Paul Hartman wrote:

>
> On Wed, Aug 14, 2013 at 12:18 AM, Pandu Poluan <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     I know that the (theoretical) best performance is to use
>     -march=native , but since the processors of the HP servers are not
>     exactly the same as the Dell's, I'm concerned that compiling with
>     -march=native will render the VMs unable to migrate between the
>     different hosts.
>
>
>
> I use -mtune=native rather than -march=native, that way I can use some
> advanced processor features if they are available, but my system will
> still run if moved to a different host.

That's not how -mtune works. If -march is unspecified, it will default
to the lowest common denominator for the platform which prevents the use
of any distinguished processor features. For an amd64 install, that
would be "-march=x86-64".

Instead, -mtune affects everything that -march doesn't. Though it
doesn't affect the instructions that *can* be used, it may effect which
of the allowed instructions are used and how. For instance, gcc includes
processor pipeline descriptions for different microarchitectures so as
to emit instructions in a way that tries to avoid pipeline hazards:

http://gcc.gnu.org/onlinedocs/gccint/Processor-pipeline-description.html

If performance matters, a better approach is to look at what
-march=native enables and manually specify all options that are common
between the hosts. Further, if the host CPU microarchitecture varies
then I would suggest adding -mtune=generic so as not to make potentially
erroneous assumptions in the course of applying that type of
optimisation. Indeed, -mtune=generic is the default if neither -march
nor -mtune are specified.

Regarding qemu, the main thing is never to use a feature that would
incur costly emulation on the host side.

--Kerin

Reply | Threaded
Open this post in threaded view
|

Re: How to determine 'Least Common Denominator' between Xen(Server) Hosts

Paul Hartman-3
On Thu, Aug 15, 2013 at 12:41 PM, Kerin Millar <[hidden email]> wrote:

>> I use -mtune=native rather than -march=native, that way I can use some
>> advanced processor features if they are available, but my system will
>> still run if moved to a different host.
>
>
> That's not how -mtune works. If -march is unspecified, it will default to
> the lowest common denominator for the platform which prevents the use of any
> distinguished processor features. For an amd64 install, that would be
> "-march=x86-64".
>
> Instead, -mtune affects everything that -march doesn't. Though it doesn't
> affect the instructions that *can* be used, it may effect which of the
> allowed instructions are used and how. For instance, gcc includes processor
> pipeline descriptions for different microarchitectures so as to emit
> instructions in a way that tries to avoid pipeline hazards:

Thanks very much for the clarification, I appreciate it.

Reply | Threaded
Open this post in threaded view
|

Re: How to determine 'Least Common Denominator' between Xen(Server) Hosts

Pandu Poluan
In reply to this post by Kerin Millar-2


On Aug 16, 2013 12:26 AM, "Kerin Millar" <[hidden email]> wrote:
>
> On 14/08/2013 13:15, Bruce Hill wrote:
>>
>> On Wed, Aug 14, 2013 at 12:18:41PM +0700, Pandu Poluan wrote:
>>>
>>> Hello list!
>>>
>>> My company has 2 HP DL585 G5 servers and 5 Dell R... something servers. All
>>> using AMD processors. They currently are acting as XenServer hosts.
>>>
>>> How do I determine the 'least common denominator' for Gentoo VMs (running
>>> as XenServer guests), especially for gcc flags?
>>>
>>> I know that the (theoretical) best performance is to use -march=native ,
>>> but since the processors of the HP servers are not exactly the same as the
>>> Dell's, I'm concerned that compiling with -march=native will render the VMs
>>> unable to migrate between the different hosts.
>
>
> A couple of points:
>
> * The effect of setting -march=native depends on the characteristics of
>   the CPU (be it virtual or otherwise)
> * The characteristics of the vCPU are defined by qemu's -cpu parameter
> * qemu can emulate features not implemented by the host CPU (at a cost)
>
> One way to go about it is to start qemu with a -cpu model that exposes features that all of your host CPUs have in common (or a subset thereof). In that case, -march=native is fine because all of the features that it detects as being available will be supported in hardware on the host side.
>
> Another way is to expose the host CPU fully with "-cpu host" and to define your guest CFLAGS according to the most optimal subset. If you are looking for a 'perfect' configuration then this this would be the most effective method, if applied correctly.
>

AFAIK, that's how XenServer configured its hosts. There's "CPU Masking" option when a heterogeneous pool of hosts were created, but I have the impression that "CPU Masking" is only being employed by the 'xe toolstack' (CloudStack) layer to determine to which host a VM can be migrated.

> Irrespective of the method, by examining /proc/cpuinfo and using the diff technique mentioned by Bruce, you should be able to determine the optimal configuration.
>
> Finally, in cases where the host CPUs differ significantly - in that native would imply a different -march value - you may choose to augment your CFLAGS with -mtune=generic to even out performance across the board. I don't think this would apply to you though.
>

Certainly doesn't apply to me. Based on tech spec I have on the servers, the processors are very similar... I just want to be doubly sure :-)

Thanks for the explanation (including the difference between 'march' and 'mtune' in your other email)!

Rgds,
--