CLVM on amd64 profile

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

CLVM on amd64 profile

Giacomo Bagnoli
Hi all,
this is my first message here, I did a quick search on google but I did
not found the answers I'm looking for.
I'm trying to setup (well, right now I am experimenting :) ) a 2 node
cluster with drbd-shared storage using 64bit profile on 2 intel xeon
servers. I've successfully setup drbd 8 in primary-primary using an lvm
LV as drbd disk with ocfs2 (that's it, drbd over lvm).
Now I'm trying to do the opposite, lvm over a drbd device, using drbd1
as a PV for a lvm VG. If I've understood the problem correctly I need
clvm to manage active-active configuration to avoid metadata corruption
when managing LVs.
This shared PV is not the system PV I use for the node OS, it's on
another disk that i plan to share directly via drbd.

What confuses me is the presence of sys-fs/clvm and sys-fs/lvm2 with
clvm and cman use flags.
I've installed sys-fs/lvm2-2.0.45, when trying to install clvm I get
blockers (device-mapper and lvm2 itself).
sys-fs/lvm2 clvm and  cman use flags are profile masked on amd64 (but
not on x86), why?
Should I need to downgrade lvm2 and install clvm or do it's safe to try
out unmasking those use flags by overriding the profile (I'm using
paludis and already have a custom profile that inherit from
amd64/2008.0/server)?

Thanks in advance,
        Giacomo


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CLVM on amd64 profile

Brian Kroth
Giacomo Bagnoli <[hidden email]> 2009-05-18 12:27:

> Hi all,
> this is my first message here, I did a quick search on google but I did
> not found the answers I'm looking for.
> I'm trying to setup (well, right now I am experimenting :) ) a 2 node
> cluster with drbd-shared storage using 64bit profile on 2 intel xeon
> servers. I've successfully setup drbd 8 in primary-primary using an lvm
> LV as drbd disk with ocfs2 (that's it, drbd over lvm).
> Now I'm trying to do the opposite, lvm over a drbd device, using drbd1
> as a PV for a lvm VG. If I've understood the problem correctly I need
> clvm to manage active-active configuration to avoid metadata corruption
> when managing LVs.
> This shared PV is not the system PV I use for the node OS, it's on
> another disk that i plan to share directly via drbd.
>
> What confuses me is the presence of sys-fs/clvm and sys-fs/lvm2 with
> clvm and cman use flags.
> I've installed sys-fs/lvm2-2.0.45, when trying to install clvm I get
> blockers (device-mapper and lvm2 itself).
> sys-fs/lvm2 clvm and  cman use flags are profile masked on amd64 (but
> not on x86), why?
> Should I need to downgrade lvm2 and install clvm or do it's safe to try
> out unmasking those use flags by overriding the profile (I'm using
> paludis and already have a custom profile that inherit from
> amd64/2008.0/server)?
>
> Thanks in advance,
> Giacomo

I remember looking at this a while ago as well and being fairly
confused.  I'm fairly certain that lvm2 with the use flags you mention
is the way to go, so you might have to unmask them in your
/etc/portage/profile/use.mask (add -clvm, -cman).

OCFS2 generally uses it's own different cluster stack than CMAN (which
uses openais).  Since I believe 2.6.26 you can use the other one, but I
haven't found it to be as stable.  It also requires quite a bit of extra
configuration for proper fencing, whereas that's built into the OCFS2
stack.  Since you're doing this with DRBD it sounds like you're trying
to do it on the cheap so proper fencing probably isn't available
anyways.

If you aren't doing anything with LVM that needs to be cluster aware
(eg: just a concatenation of multiple PVs, no striping) then it should
be just fine without CLVM.  The key is that you /must/ take all but one
node offline (umount; vgchange -an) before you do any LVM operations
such as adding more extents, else they'll have an inconsistent view of
the VGs/LVs.  This also isn't technically supported but I've run it
without problems for a year or so now.

Brian

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CLVM on amd64 profile

Giacomo Bagnoli
Brian,
first of all thanks for your answer

Il giorno lun, 18/05/2009 alle 09.29 -0500, Brian Kroth ha scritto:
> I remember looking at this a while ago as well and being fairly
> confused.  I'm fairly certain that lvm2 with the use flags you mention
> is the way to go, so you might have to unmask them in your
> /etc/portage/profile/use.mask (add -clvm, -cman).
>
> OCFS2 generally uses it's own different cluster stack than CMAN (which
> uses openais).  Since I believe 2.6.26 you can use the other one, but I
> haven't found it to be as stable.

I didn't understand here what you mean. Which one is "the other one"?
(maybe it's just my english :P )

>   It also requires quite a bit of extra
> configuration for proper fencing, whereas that's built into the OCFS2
> stack.  Since you're doing this with DRBD it sounds like you're trying
> to do it on the cheap so proper fencing probably isn't available
> anyways.

Yes, you're right. The only way I have to get a sort of STONITH (if it
can be used, I've still to check this) is IPMI which is working well so
far.

> If you aren't doing anything with LVM that needs to be cluster aware
> (eg: just a concatenation of multiple PVs, no striping) then it should
> be just fine without CLVM.  The key is that you /must/ take all but one
> node offline (umount; vgchange -an) before you do any LVM operations
> such as adding more extents, else they'll have an inconsistent view of
> the VGs/LVs.  This also isn't technically supported but I've run it
> without problems for a year or so now.

The problem is that a node will use (mount) some LVs and the other one
some others. So I can't do the fast way as you suggest, as I need to
stop all services that use data on LVs before unmounting.

I'll try to unmask those flags and re-emerge lvm2.

Thanks,
        Giacomo


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CLVM on amd64 profile

Brian Kroth
Giacomo Bagnoli <[hidden email]> 2009-05-18 17:06:

> Brian,
> first of all thanks for your answer
>
> Il giorno lun, 18/05/2009 alle 09.29 -0500, Brian Kroth ha scritto:
> > I remember looking at this a while ago as well and being fairly
> > confused.  I'm fairly certain that lvm2 with the use flags you mention
> > is the way to go, so you might have to unmask them in your
> > /etc/portage/profile/use.mask (add -clvm, -cman).
> >
> > OCFS2 generally uses it's own different cluster stack than CMAN (which
> > uses openais).  Since I believe 2.6.26 you can use the other one, but I
> > haven't found it to be as stable.
>
> I didn't understand here what you mean. Which one is "the other one"?
> (maybe it's just my english :P )

Since 2.6.26, OCFS2 has two cluster stacks available to it:
- ocfs2_stack_o2cb:  which is the original stack
- ocfs2_stack_user:  which is the one that requires cman and openais

> >   It also requires quite a bit of extra
> > configuration for proper fencing, whereas that's built into the OCFS2
> > stack.  Since you're doing this with DRBD it sounds like you're trying
> > to do it on the cheap so proper fencing probably isn't available
> > anyways.
>
> Yes, you're right. The only way I have to get a sort of STONITH (if it
> can be used, I've still to check this) is IPMI which is working well so
> far.

I believe that will work just fine.

> > If you aren't doing anything with LVM that needs to be cluster aware
> > (eg: just a concatenation of multiple PVs, no striping) then it should
> > be just fine without CLVM.  The key is that you /must/ take all but one
> > node offline (umount; vgchange -an) before you do any LVM operations
> > such as adding more extents, else they'll have an inconsistent view of
> > the VGs/LVs.  This also isn't technically supported but I've run it
> > without problems for a year or so now.
>
> The problem is that a node will use (mount) some LVs and the other one
> some others. So I can't do the fast way as you suggest, as I need to
> stop all services that use data on LVs before unmounting.

But, with OCFS2 you can (even temporarily) have a single node mount all
the LVs, move your service IPs over, and _then_ umount and stop them on
the other one while you do your changes and then remount them all.
That's the great things amount a clustered fs - more than one node can
play in the sandbox without problems.

Also, I'm gonna put a plug in for Heartbeat [1] for service management.
It's much more powerful than rgmanager in my opinion.  If you're dead
set on using the openais cluster stack you can also use pacemaker [2].

The model I typically use for Heartbeat, is to have it run a couple of
dummy monitoring scripts as resource agents clones that each set an
attribute as to the service's health on a machine.  Then using the value
of that attribute I setup rules to move service IPs to the "healthiest"
node, where your definition of "healthiest" is completely dependent upon
your monitoring scripts.

Cheers,
Brian

[1] http://www.linux-ha.org/
[2] http://clusterlabs.org/wiki/Main_Page

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CLVM on amd64 profile

Robin H. Johnson-2
In reply to this post by Giacomo Bagnoli
On Mon, May 18, 2009 at 12:27:41PM +0200, Giacomo Bagnoli wrote:

> What confuses me is the presence of sys-fs/clvm and sys-fs/lvm2 with
> clvm and cman use flags.
> I've installed sys-fs/lvm2-2.0.45, when trying to install clvm I get
> blockers (device-mapper and lvm2 itself).
> sys-fs/lvm2 clvm and  cman use flags are profile masked on amd64 (but
> not on x86), why?
> Should I need to downgrade lvm2 and install clvm or do it's safe to try
> out unmasking those use flags by overriding the profile (I'm using
> paludis and already have a custom profile that inherit from
> amd64/2008.0/server)?
I've been trying to get clvm and lvm2 merged for years, but the
clustering team seldom responds to my inquires.

The latest bug is here:
http://bugs.gentoo.org/265671

The differences between the two are very small at this point, because in
the previous work, I rebased clvm off the lvm2 ebuild that was then
current.

--
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : [hidden email]
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

attachment0 (338 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CLVM on amd64 profile

Giacomo Bagnoli
In reply to this post by Brian Kroth
On Mon, 2009-05-18 at 14:39 -0500, Brian Kroth wrote:

> Since 2.6.26, OCFS2 has two cluster stacks available to it:
> - ocfs2_stack_o2cb:  which is the original stack
> - ocfs2_stack_user:  which is the one that requires cman and openais

oh, I missed it completely.
I'm using the first one :)

> > Yes, you're right. The only way I have to get a sort of STONITH (if it
> > can be used, I've still to check this) is IPMI which is working well so
> > far.
>
> I believe that will work just fine.

hoping so, after a quick search I've seen IPMI agents for heartbeat and
for fenced.
 
> > The problem is that a node will use (mount) some LVs and the other one
> > some others. So I can't do the fast way as you suggest, as I need to
> > stop all services that use data on LVs before unmounting.
>
> But, with OCFS2 you can (even temporarily) have a single node mount all
> the LVs, move your service IPs over, and _then_ umount and stop them on
> the other one while you do your changes and then remount them all.
> That's the great things amount a clustered fs - more than one node can
> play in the sandbox without problems.

Uh, that can be an option. It seems even a scriptable work, with a bit
of effort and error-checking, but it seems doable in the end. Will check
that, thanks.

> Also, I'm gonna put a plug in for Heartbeat [1] for service management.
> It's much more powerful than rgmanager in my opinion.  If you're dead
> set on using the openais cluster stack you can also use pacemaker [2].
>The model I typically use for Heartbeat, is to have it run a couple of
> dummy monitoring scripts as resource agents clones that each set an
> attribute as to the service's health on a machine.  Then using the value
> of that attribute I setup rules to move service IPs to the "healthiest"
> node, where your definition of "healthiest" is completely dependent upon
> your monitoring scripts.
>
Again, thanks for the pointers. Expecially pacemaker of which I did not
know the existence.

        Giacomo


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CLVM on amd64 profile

Giacomo Bagnoli
In reply to this post by Robin H. Johnson-2
On Mon, 2009-05-18 at 14:22 -0700, Robin H. Johnson wrote:
> I've been trying to get clvm and lvm2 merged for years, but the
> clustering team seldom responds to my inquires.
>
> The latest bug is here:
> http://bugs.gentoo.org/265671
>
> The differences between the two are very small at this point, because in
> the previous work, I rebased clvm off the lvm2 ebuild that was then
> current.

Well, the only thing I still don't understand is why clvm and cman use
flags are masked on amd64 profile. I've unmasked them (overriding the
profile) compiled deps, configured the cluster and it seems to work,
just curious if I'm going into troubles.

        Giacomo


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CLVM on amd64 profile

Robin H. Johnson-2
On Mon, May 18, 2009 at 11:31:57PM +0200, Giacomo Bagnoli wrote:

> On Mon, 2009-05-18 at 14:22 -0700, Robin H. Johnson wrote:
> > I've been trying to get clvm and lvm2 merged for years, but the
> > clustering team seldom responds to my inquires.
> >
> > The latest bug is here:
> > http://bugs.gentoo.org/265671
> >
> > The differences between the two are very small at this point, because in
> > the previous work, I rebased clvm off the lvm2 ebuild that was then
> > current.
> Well, the only thing I still don't understand is why clvm and cman use
> flags are masked on amd64 profile. I've unmasked them (overriding the
> profile) compiled deps, configured the cluster and it seems to work,
> just curious if I'm going into troubles.
Looks like it originated prior to 2007, and hasn't been removed at all,
and instead migrated up to the global use.mask (and is reversed only for
x86).

cman and related packages are keyworded for amd64, so it should
certainly be possible to remove the use.mask on amd64. Removing it for
other architectures would require them to test cman and all the related
packages first.

--
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : [hidden email]
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

attachment0 (338 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CLVM on amd64 profile

Robin H. Johnson-2
In reply to this post by Robin H. Johnson-2
On Mon, May 18, 2009 at 02:22:24PM -0700, Robin H. Johnson wrote:
> The differences between the two are very small at this point, because in
> the previous work, I rebased clvm off the lvm2 ebuild that was then
> current.
And now the differences are nearly zero in the new lvm2-2.02.45-r2
ebuild.

I kept a single feature (ability to use --with-clvmd=none) that was in
lvm2 but not clvm, and moved over everything else from clvm into lvm2.

--
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : [hidden email]
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

attachment0 (338 bytes) Download Attachment
Loading...