Hard drive storage questions

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Hard drive storage questions

Dale-46
Howdy to all,

I have a interesting problem coming up.  Currently, I have two 3TB
drives for my /home mount point.  A lot of this is videos but some pdf
files and other documents as well plus a photo collection of family
stuff etc. 

Filesystem                                   Size  Used Avail Use%
Mounted on
/dev/mapper/Home2-Home2        5.4T  3.7T  1.8T  68% /home

I've got a little over 25% or so of usable space left.  At that point or
shortly thereafter, it could start causing some issues according to what
I've read anyway.  Either way, shortly after that, being full will
certainly be a issue.  I'm full up on my motherboard SATA ports.  Even
if I buy a larger drive or drives, I'd have to unplug one to move things
over and likely repeat that a few times.  I could do that and likely
will have to anyway but I'm trying to think a little farther ahead. 
Currently I have the more important stuff backed up to a external single
6TB USB drive, previous thread on that.  I'm trying to come up with a
plan that allows me to grow easier and without having to worry about
running out of motherboard based ports. 

I thought about a store bought enclosure with more than one hard drive
that connects by ethernet.  The ones I've found are fairly expensive. 
Doing it over USB concerns me for other reasons, USB isn't always that
stable.  So, internal isn't working out to well long term.  Ethernet
based is expensive, what I could find anyway.  USB isn't that stable. 

I'm planning to upgrade my current system.  Upgrade the CPU, memory and
maybe even the video card as well.  I thought about using a older spare
motherboard, those removed components and building a mini system sort of
thing.  I could have one small drive for a OS and then add large drives
for storage.  Then I can access those from my main system, ethernet I
would guess.  Even then, I'd still be limited to the SATA ports on the
MOBO at some point but it would be a ways into the future.  So far tho,
this is one of the better ideas.  So far.  Does anyone else have other
ideas on how to do this?  Some method that I've never heard of but
doesn't cost a lot of money to do? 

Thanks in advance for any ideas.  I can't think of anything else.  ;-) 
If you need more info, let me know. 

Dale

:-)  :-) 

Reply | Threaded
Open this post in threaded view
|

Re: Hard drive storage questions

Jack
On 2018.11.08 20:16, Dale wrote:

> Howdy to all,
>
> I have a interesting problem coming up.  Currently, I have two 3TB
> drives for my /home mount point.  A lot of this is videos but some pdf
> files and other documents as well plus a photo collection of family
> stuff etc. 
>
> Filesystem                                   Size  Used Avail Use%
> Mounted on
> /dev/mapper/Home2-Home2        5.4T  3.7T  1.8T  68% /home
>
> I've got a little over 25% or so of usable space left.  At that point  
> or
> shortly thereafter, it could start causing some issues according to  
> what
> I've read anyway.  Either way, shortly after that, being full will
> certainly be a issue.  I'm full up on my motherboard SATA ports.  Even
> if I buy a larger drive or drives, I'd have to unplug one to move  
> things
> over and likely repeat that a few times.  I could do that and likely
> will have to anyway but I'm trying to think a little farther ahead. 
> Currently I have the more important stuff backed up to a external  
> single
> 6TB USB drive, previous thread on that.  I'm trying to come up with a
> plan that allows me to grow easier and without having to worry about
> running out of motherboard based ports. 
>
> I thought about a store bought enclosure with more than one hard drive
> that connects by ethernet.  The ones I've found are fairly expensive. 
> Doing it over USB concerns me for other reasons, USB isn't always that
> stable.  So, internal isn't working out to well long term.  Ethernet
> based is expensive, what I could find anyway.  USB isn't that stable. 
>
> I'm planning to upgrade my current system.  Upgrade the CPU, memory  
> and
> maybe even the video card as well.  I thought about using a older  
> spare
> motherboard, those removed components and building a mini system sort  
> of
> thing.  I could have one small drive for a OS and then add large  
> drives
> for storage.  Then I can access those from my main system, ethernet I
> would guess.  Even then, I'd still be limited to the SATA ports on the
> MOBO at some point but it would be a ways into the future.  So far  
> tho,
> this is one of the better ideas.  So far.  Does anyone else have other
> ideas on how to do this?  Some method that I've never heard of but
> doesn't cost a lot of money to do? 
>
> Thanks in advance for any ideas.  I can't think of anything else.   
> ;-) 
> If you need more info, let me know. 
>
> Dale
>
If you have space on the mobo for another card, you should be able to  
get an additional SATA card.  I have no idea on prices, but I'd be  
surprised if it's prohibitive.

Jack
Reply | Threaded
Open this post in threaded view
|

Re: Hard drive storage questions

Dale-46
Jack wrote:

> On 2018.11.08 20:16, Dale wrote:
>> Howdy to all,
>>
>> I have a interesting problem coming up.  Currently, I have two 3TB
>> drives for my /home mount point.  A lot of this is videos but some pdf
>> files and other documents as well plus a photo collection of family
>> stuff etc. 
>>
>> Filesystem                                   Size  Used Avail Use%
>> Mounted on
>> /dev/mapper/Home2-Home2        5.4T  3.7T  1.8T  68% /home
>>
>> I've got a little over 25% or so of usable space left.  At that point or
>> shortly thereafter, it could start causing some issues according to what
>> I've read anyway.  Either way, shortly after that, being full will
>> certainly be a issue.  I'm full up on my motherboard SATA ports.  Even
>> if I buy a larger drive or drives, I'd have to unplug one to move things
>> over and likely repeat that a few times.  I could do that and likely
>> will have to anyway but I'm trying to think a little farther ahead. 
>> Currently I have the more important stuff backed up to a external single
>> 6TB USB drive, previous thread on that.  I'm trying to come up with a
>> plan that allows me to grow easier and without having to worry about
>> running out of motherboard based ports. 
>>
>> I thought about a store bought enclosure with more than one hard drive
>> that connects by ethernet.  The ones I've found are fairly expensive. 
>> Doing it over USB concerns me for other reasons, USB isn't always that
>> stable.  So, internal isn't working out to well long term.  Ethernet
>> based is expensive, what I could find anyway.  USB isn't that stable. 
>>
>> I'm planning to upgrade my current system.  Upgrade the CPU, memory and
>> maybe even the video card as well.  I thought about using a older spare
>> motherboard, those removed components and building a mini system sort of
>> thing.  I could have one small drive for a OS and then add large drives
>> for storage.  Then I can access those from my main system, ethernet I
>> would guess.  Even then, I'd still be limited to the SATA ports on the
>> MOBO at some point but it would be a ways into the future.  So far tho,
>> this is one of the better ideas.  So far.  Does anyone else have other
>> ideas on how to do this?  Some method that I've never heard of but
>> doesn't cost a lot of money to do? 
>>
>> Thanks in advance for any ideas.  I can't think of anything else.  ;-) 
>> If you need more info, let me know. 
>>
>> Dale
>>
> If you have space on the mobo for another card, you should be able to
> get an additional SATA card.  I have no idea on prices, but I'd be
> surprised if it's prohibitive.
>
> Jack
>


I hadn't thought of adding a SATA card.  I have a ethernet card and the
video card and I don't think there are any others.  I should have some
open slots there.  Well, that is one idea that I hadn't thought of. 
lol  Since I have a LARGE case, Cooler Master HAF-932, I have space for
more drives.  I think this thing holds like nine or ten pretty easy. 

Thanks.  Another option to look into. 

Dale

:-)  :-) 

Reply | Threaded
Open this post in threaded view
|

Re: Hard drive storage questions

Andrew Lowe-2
    Firstly sorry about the top post, on the phone.

    I've had the same sort of thing happen to me. I was lucky to have available sata ports so bought two WD 8TB video archive drives and attached them to the MB. The card idea from the previous post is basically the same thing.

    In turn the archive drives will spin down after about 10 minutes, you will hear this happen and is a bit disconcerting the first couple of times, and use hardly any power. In turn if you need anything from the drives it takes just a few seconds to spin up again.

    In comparison to a small NAS thingy, I'm way ahead. No purchase of bare NAS, and much lower power consumption. Much higher data transfer rates as well. Having said that, I may have to buy a NAS in the long run as a backup for other purposes.

      Andrew

Sent from my phone

-----Original Message-----
From: Dale <[hidden email]>
To: [hidden email]
Sent: Fri., 09 Nov. <a href="tel:20189">2018 9:43
Subject: Re: [gentoo-user] Hard drive storage questions

Jack wrote:
> On <a href="tel:2018110820">2018.11.08 20:16, Dale wrote:
>> Howdy to all,
>>
>> I have a interesting problem coming up.  Currently, I have two 3TB
>> drives for my /home mount point.  A lot of this is videos but some pdf
>> files and other documents as well plus a photo collection of family
>> stuff etc. 
>>
>> Filesystem                                   Size  Used Avail Use%
>> Mounted on
>> /dev/mapper/Home2-Home2        5.4T  3.7T  1.8T  68% /home
>>
>> I've got a little over 25% or so of usable space left.  At that point or
>> shortly thereafter, it could start causing some issues according to what
>> I've read anyway.  Either way, shortly after that, being full will
>> certainly be a issue.  I'm full up on my motherboard SATA ports.  Even
>> if I buy a larger drive or drives, I'd have to unplug one to move things
>> over and likely repeat that a few times.  I could do that and likely
>> will have to anyway but I'm trying to think a little farther ahead. 
>> Currently I have the more important stuff backed up to a external single
>> 6TB USB drive, previous thread on that.  I'm trying to come up with a
>> plan that allows me to grow easier and without having to worry about
>> running out of motherboard based ports. 
>>
>> I thought about a store bought enclosure with more than one hard drive
>> that connects by ethernet.  The ones I've found are fairly expensive. 
>> Doing it over USB concerns me for other reasons, USB isn't always that
>> stable.  So, internal isn't working out to well long term.  Ethernet
>> based is expensive, what I could find anyway.  USB isn't that stable. 
>>
>> I'm planning to upgrade my current system.  Upgrade the CPU, memory and
>> maybe even the video card as well.  I thought about using a older spare
>> motherboard, those removed components and building a mini system sort of
>> thing.  I could have one small drive for a OS and then add large drives
>> for storage.  Then I can access those from my main system, ethernet I
>> would guess.  Even then, I'd still be limited to the SATA ports on the
>> MOBO at some point but it would be a ways into the future.  So far tho,
>> this is one of the better ideas.  So far.  Does anyone else have other
>> ideas on how to do this?  Some method that I've never heard of but
>> doesn't cost a lot of money to do? 
>>
>> Thanks in advance for any ideas.  I can't think of anything else.  ;-) 
>> If you need more info, let me know. 
>>
>> Dale
>>
> If you have space on the mobo for another card, you should be able to
> get an additional SATA card.  I have no idea on prices, but I'd be
> surprised if it's prohibitive.
>
> Jack
>


I hadn't thought of adding a SATA card.  I have a ethernet card and the
video card and I don't think there are any others.  I should have some
open slots there.  Well, that is one idea that I hadn't thought of. 
lol  Since I have a LARGE case, Cooler Master HAF-932, I have space for
more drives.  I think this thing holds like nine or ten pretty easy. 

Thanks.  Another option to look into. 

Dale

:-)  :-) 

Reply | Threaded
Open this post in threaded view
|

Re: Hard drive storage questions

Bill Kenworthy
In reply to this post by Dale-46
On 09/11/18 09:43, Dale wrote:

> Jack wrote:
>> On 2018.11.08 20:16, Dale wrote:
>>> Howdy to all,
>>>
>>> I have a interesting problem coming up.  Currently, I have two 3TB
>>> drives for my /home mount point.  A lot of this is videos but some pdf
>>> files and other documents as well plus a photo collection of family
>>> stuff etc. 
>>>
>>> Filesystem                                   Size  Used Avail Use%
>>> Mounted on
>>> /dev/mapper/Home2-Home2        5.4T  3.7T  1.8T  68% /home
>>>
>>> I've got a little over 25% or so of usable space left.  At that point or
>>> shortly thereafter, it could start causing some issues according to what
>>> I've read anyway.  Either way, shortly after that, being full will
>>> certainly be a issue.  I'm full up on my motherboard SATA ports.  Even
>>> if I buy a larger drive or drives, I'd have to unplug one to move things
>>> over and likely repeat that a few times.  I could do that and likely
>>> will have to anyway but I'm trying to think a little farther ahead. 
>>> Currently I have the more important stuff backed up to a external single
>>> 6TB USB drive, previous thread on that.  I'm trying to come up with a
>>> plan that allows me to grow easier and without having to worry about
>>> running out of motherboard based ports. 
>>>
>>> I thought about a store bought enclosure with more than one hard drive
>>> that connects by ethernet.  The ones I've found are fairly expensive. 
>>> Doing it over USB concerns me for other reasons, USB isn't always that
>>> stable.  So, internal isn't working out to well long term.  Ethernet
>>> based is expensive, what I could find anyway.  USB isn't that stable. 
>>>
>>> I'm planning to upgrade my current system.  Upgrade the CPU, memory and
>>> maybe even the video card as well.  I thought about using a older spare
>>> motherboard, those removed components and building a mini system sort of
>>> thing.  I could have one small drive for a OS and then add large drives
>>> for storage.  Then I can access those from my main system, ethernet I
>>> would guess.  Even then, I'd still be limited to the SATA ports on the
>>> MOBO at some point but it would be a ways into the future.  So far tho,
>>> this is one of the better ideas.  So far.  Does anyone else have other
>>> ideas on how to do this?  Some method that I've never heard of but
>>> doesn't cost a lot of money to do? 
>>>
>>> Thanks in advance for any ideas.  I can't think of anything else.  ;-) 
>>> If you need more info, let me know. 
>>>
>>> Dale
>>>
>> If you have space on the mobo for another card, you should be able to
>> get an additional SATA card.  I have no idea on prices, but I'd be
>> surprised if it's prohibitive.
>>
>> Jack
>>
>
> I hadn't thought of adding a SATA card.  I have a ethernet card and the
> video card and I don't think there are any others.  I should have some
> open slots there.  Well, that is one idea that I hadn't thought of. 
> lol  Since I have a LARGE case, Cooler Master HAF-932, I have space for
> more drives.  I think this thing holds like nine or ten pretty easy. 
>
> Thanks.  Another option to look into. 
>
> Dale
>
> :-)  :-) 
>
I have used a mini-pcie board from ebay (takes two sata connections)
alongside a number of other connection types in a btrfs raid 10 for some
months as a temporary expansion - worked fine, but make sure to check
Linux compatibility first.


BillK



Reply | Threaded
Open this post in threaded view
|

Re: Hard drive storage questions

Rich Freeman
In reply to this post by Dale-46
On Thu, Nov 8, 2018 at 8:16 PM Dale <[hidden email]> wrote:
>
> I'm trying to come up with a
> plan that allows me to grow easier and without having to worry about
> running out of motherboard based ports.
>

So, this is an issue I've been changing my mind on over the years.
There are a few common approaches:

* Find ways to cram a lot of drives on one host
* Use a patchwork of NAS devices or improvised hosts sharing over
samba/nfs/etc and end up with a mess of mount points.
* Use a distributed FS

Right now I'm mainly using the first approach, and I'm trying to move
to the last.  The middle option has never appealed to me.

So, to do more of what you're doing in the most efficient way
possible, I recommend finding used LSI HBA cards.  These have mini-SAS
ports on them, and one of these can be attached to a breakout cable
that gets you 4 SATA ports.  I just picked up two of these for $20
each on ebay (used) and they have 4 mini-SAS ports each, which is
capacity for 16 SATA drives per card.  Typically these have 4x or
larger PCIe interfaces, so you'll need a large slot, or one with a
cutout.  You'd have to do the math but I suspect that if the card+MB
supports PCIe 3.0 you're not losing much if you cram it into a smaller
slot.  If most of the drives are idle most of the time then that also
demands less bandwidth.  16 fully busy hard drives obviously can put
out a lot of data if reading sequentially.

You can of course get more consumer-oriented SATA cards, but you're
lucky to get 2-4 SATA ports on a card that runs you $30.  The mini-SAS
HBAs get you a LOT more drives per PCIe slot, and your PCIe slots are
you main limiting factor assuming you have power and case space.

Oh, and those HBA cards need to be flashed into "IT" mode - they're
often sold this way, but if they support RAID you want to flash the IT
firmware that just makes them into a bunch of standalone SATA slots.
This is usually a PITA that involves DOS or whatever, but I have
noticed some of the software needed in the Gentoo repo.

If you go that route it is just like having a ton of SATA ports in
your system - they just show up as sda...sdz and so on (no idea where
it goes after that).  Software-wise you just keep doing what you're
already doing (though you should be seriously considering
mdadm/zfs/btrfs/whatever at that point).

That is the more traditional route.

Now let me talk about distributed filesystems, which is the more
scalable approach.  I'm getting tired of being limited by SATA ports,
and cases, and such.  I'm also frustrated with some of zfs's
inflexibility around removing drives.  These are constraints that make
upgrading painful, and often inefficient.  Distributed filesystems
offer a different solution.

A distributed filesystem spreads its storage across many hosts, with
an arbitrary number of drives per host (more or less).  So, you can
add more hosts, add more drives to a host, and so on.  That means
you're never forced to try to find a way to cram a few more drives in
one host.  The resulting filesystem appears as one gigantic filesystem
(unless you want to split it up), which means no mess of nfs
mountpoints and so on, and all the other headaches of nfs.  Just as
with RAID these support redundancy, except now you can lose entire
hosts without issue.  With many you can even tell it which
PDU/rack/whatever each host is plugged into, and it will make sure you
can lose all the hosts in one rack.  You can also mount the filesystem
on as many hosts as you want at the same time.

They do tend to be a bit more complex.  The big players can scale VERY
large - thousands of drives easily.  Everything seems to be moving
towards Ceph/CephFS.  If you were hosting a datacenter full of
VMs/containers/etc I'd be telling you to host it on Ceph.  However,
for small scale (which you definitely are right now), I'm not thrilled
with it.  Due to the way it allocates data (hash-based) anytime
anything changes you end up having to move all the data around in the
cluster, and all the reports I've read suggests it doesn't perform all
that great if you only have a few nodes.  Ceph storage nodes are also
RAM-hungry, and I want to run these on ARM to save power, and few ARM
boards have that kind of RAM, and they're very expensive.

Personally I'm working on deploying a cluster of a few nodes running
LizardFS, which is basically a fork/derivative of MooseFS.  While it
won't scale nearly as well, below 100 nodes should be fine, and in
particular it sounds like it works fairly well with only a few nodes.
It has its pros and cons, but for my needs it should be sufficient.
It also isn't RAM-hungry.  I'm going to be testing it on some
RockPro64s, with the LSI HBAs.

I did note that Gentoo lacks a LizardFS client.  I suspect I'll be
looking to fix that - I'm sure the moosefs ebuild would be a good
starting point.  I'm probably going to be a whimp and run the storage
nodes on Ubuntu or whatever upstream targets - they're basically
appliances as far as I'm concerned.

So, those are the two routes I'd recommend.  Just get yourself an HBA
if you only want a few more drives.  If you see your needs expanding
then consider a distributed filesystem.  The advantage of the latter
is that you can keep expanding it however you want with additional
drives/nodes/whatever.  If you're going over 20 nodes I'd use Ceph for
sure - IMO that seems to be the future of this space.

--
Rich

Reply | Threaded
Open this post in threaded view
|

Re: Hard drive storage questions

Bill Kenworthy
On 09/11/18 10:29, Rich Freeman wrote:

> On Thu, Nov 8, 2018 at 8:16 PM Dale <[hidden email]> wrote:
>> I'm trying to come up with a
>> plan that allows me to grow easier and without having to worry about
>> running out of motherboard based ports.
>>
> So, this is an issue I've been changing my mind on over the years.
> There are a few common approaches:
>
> * Find ways to cram a lot of drives on one host
> * Use a patchwork of NAS devices or improvised hosts sharing over
> samba/nfs/etc and end up with a mess of mount points.
> * Use a distributed FS
>
> Right now I'm mainly using the first approach, and I'm trying to move
> to the last.  The middle option has never appealed to me.
>
> So, to do more of what you're doing in the most efficient way
> possible, I recommend finding used LSI HBA cards.  These have mini-SAS
> ports on them, and one of these can be attached to a breakout cable
> that gets you 4 SATA ports.  I just picked up two of these for $20
> each on ebay (used) and they have 4 mini-SAS ports each, which is
> capacity for 16 SATA drives per card.  Typically these have 4x or
> larger PCIe interfaces, so you'll need a large slot, or one with a
> cutout.  You'd have to do the math but I suspect that if the card+MB
> supports PCIe 3.0 you're not losing much if you cram it into a smaller
> slot.  If most of the drives are idle most of the time then that also
> demands less bandwidth.  16 fully busy hard drives obviously can put
> out a lot of data if reading sequentially.
>
> You can of course get more consumer-oriented SATA cards, but you're
> lucky to get 2-4 SATA ports on a card that runs you $30.  The mini-SAS
> HBAs get you a LOT more drives per PCIe slot, and your PCIe slots are
> you main limiting factor assuming you have power and case space.
>
> Oh, and those HBA cards need to be flashed into "IT" mode - they're
> often sold this way, but if they support RAID you want to flash the IT
> firmware that just makes them into a bunch of standalone SATA slots.
> This is usually a PITA that involves DOS or whatever, but I have
> noticed some of the software needed in the Gentoo repo.
>
> If you go that route it is just like having a ton of SATA ports in
> your system - they just show up as sda...sdz and so on (no idea where
> it goes after that).  Software-wise you just keep doing what you're
> already doing (though you should be seriously considering
> mdadm/zfs/btrfs/whatever at that point).
>
> That is the more traditional route.
>
> Now let me talk about distributed filesystems, which is the more
> scalable approach.  I'm getting tired of being limited by SATA ports,
> and cases, and such.  I'm also frustrated with some of zfs's
> inflexibility around removing drives.  These are constraints that make
> upgrading painful, and often inefficient.  Distributed filesystems
> offer a different solution.
>
> A distributed filesystem spreads its storage across many hosts, with
> an arbitrary number of drives per host (more or less).  So, you can
> add more hosts, add more drives to a host, and so on.  That means
> you're never forced to try to find a way to cram a few more drives in
> one host.  The resulting filesystem appears as one gigantic filesystem
> (unless you want to split it up), which means no mess of nfs
> mountpoints and so on, and all the other headaches of nfs.  Just as
> with RAID these support redundancy, except now you can lose entire
> hosts without issue.  With many you can even tell it which
> PDU/rack/whatever each host is plugged into, and it will make sure you
> can lose all the hosts in one rack.  You can also mount the filesystem
> on as many hosts as you want at the same time.
>
> They do tend to be a bit more complex.  The big players can scale VERY
> large - thousands of drives easily.  Everything seems to be moving
> towards Ceph/CephFS.  If you were hosting a datacenter full of
> VMs/containers/etc I'd be telling you to host it on Ceph.  However,
> for small scale (which you definitely are right now), I'm not thrilled
> with it.  Due to the way it allocates data (hash-based) anytime
> anything changes you end up having to move all the data around in the
> cluster, and all the reports I've read suggests it doesn't perform all
> that great if you only have a few nodes.  Ceph storage nodes are also
> RAM-hungry, and I want to run these on ARM to save power, and few ARM
> boards have that kind of RAM, and they're very expensive.
>
> Personally I'm working on deploying a cluster of a few nodes running
> LizardFS, which is basically a fork/derivative of MooseFS.  While it
> won't scale nearly as well, below 100 nodes should be fine, and in
> particular it sounds like it works fairly well with only a few nodes.
> It has its pros and cons, but for my needs it should be sufficient.
> It also isn't RAM-hungry.  I'm going to be testing it on some
> RockPro64s, with the LSI HBAs.
>
> I did note that Gentoo lacks a LizardFS client.  I suspect I'll be
> looking to fix that - I'm sure the moosefs ebuild would be a good
> starting point.  I'm probably going to be a whimp and run the storage
> nodes on Ubuntu or whatever upstream targets - they're basically
> appliances as far as I'm concerned.
>
> So, those are the two routes I'd recommend.  Just get yourself an HBA
> if you only want a few more drives.  If you see your needs expanding
> then consider a distributed filesystem.  The advantage of the latter
> is that you can keep expanding it however you want with additional
> drives/nodes/whatever.  If you're going over 20 nodes I'd use Ceph for
> sure - IMO that seems to be the future of this space.
>
I'll second your comments on ceph after my experience - great idea for
large scale systems, otherwise performance is quite poor on small
systems. Needs at least GB connections with two networks as well as only
one or two drives per host to work properly.

I think I'll give lizardfs a go - an interesting read.


BillK



Reply | Threaded
Open this post in threaded view
|

Re: Hard drive storage questions

Neil Bothwick
In reply to this post by Bill Kenworthy
On Fri, 9 Nov 2018 10:07:58 +0800, Bill Kenworthy wrote:

> I have used a mini-pcie board from ebay (takes two sata connections)
> alongside a number of other connection types in a btrfs raid 10 for some
> months as a temporary expansion - worked fine, but make sure to check
> Linux compatibility first.

I have both of these in use, and they "just worked"

https://www.amazon.co.uk/Syba-SI-PEX40062-Profile-Bracket-Green/dp/B00AZ9T41M/ref=sr_1_1?ie=UTF8&qid=1541752572&sr=8-1&keywords=Syba+SI-PEX40062

https://www.amazon.co.uk/gp/product/B00UVJI3IQ/ref=oh_aui_search_detailpage?ie=UTF8&psc=1

Neither is in a particularly demanding application so I can't comment on
outright performance.


--
Neil Bothwick

For security reasons, all text in this mail is double-rot13 encrypted.

attachment0 (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Hard drive storage questions

J. Roeleveld
In reply to this post by Rich Freeman
On Friday, November 9, 2018 3:29:52 AM CET Rich Freeman wrote:

> On Thu, Nov 8, 2018 at 8:16 PM Dale <[hidden email]> wrote:
> > I'm trying to come up with a
> > plan that allows me to grow easier and without having to worry about
> > running out of motherboard based ports.
>
> So, this is an issue I've been changing my mind on over the years.
> There are a few common approaches:
>
> * Find ways to cram a lot of drives on one host
> * Use a patchwork of NAS devices or improvised hosts sharing over
> samba/nfs/etc and end up with a mess of mount points.
> * Use a distributed FS
>
> Right now I'm mainly using the first approach, and I'm trying to move
> to the last.  The middle option has never appealed to me.

I'm actually in the middle, but have a single large NAS.

> So, to do more of what you're doing in the most efficient way
> possible, I recommend finding used LSI HBA cards.  These have mini-SAS
> ports on them, and one of these can be attached to a breakout cable
> that gets you 4 SATA ports.  I just picked up two of these for $20
> each on ebay (used) and they have 4 mini-SAS ports each, which is
> capacity for 16 SATA drives per card.  Typically these have 4x or
> larger PCIe interfaces, so you'll need a large slot, or one with a
> cutout.  You'd have to do the math but I suspect that if the card+MB
> supports PCIe 3.0 you're not losing much if you cram it into a smaller
> slot.  If most of the drives are idle most of the time then that also
> demands less bandwidth.  16 fully busy hard drives obviously can put
> out a lot of data if reading sequentially.

I also recommend LSI HBA cards, they work really well and are really well
supported by Linux.

> You can of course get more consumer-oriented SATA cards, but you're
> lucky to get 2-4 SATA ports on a card that runs you $30.  The mini-SAS
> HBAs get you a LOT more drives per PCIe slot, and your PCIe slots are
> you main limiting factor assuming you have power and case space.
>
> Oh, and those HBA cards need to be flashed into "IT" mode - they're
> often sold this way, but if they support RAID you want to flash the IT
> firmware that just makes them into a bunch of standalone SATA slots.
> This is usually a PITA that involves DOS or whatever, but I have
> noticed some of the software needed in the Gentoo repo.

Even with Raid-firmware, they can be configured for JBOD.

> If you go that route it is just like having a ton of SATA ports in
> your system - they just show up as sda...sdz and so on (no idea where
> it goes after that).

I tested this once, ended up getting sdaa, sdab,...

> Software-wise you just keep doing what you're
> already doing (though you should be seriously considering
> mdadm/zfs/btrfs/whatever at that point).

I would suggest ZFS or BTRFS over mdadm. Gives you more flexibility and is a
logical follow-up to LVM.

> That is the more traditional route.
>
> Now let me talk about distributed filesystems, which is the more
> scalable approach.  I'm getting tired of being limited by SATA ports,
> and cases, and such.  I'm also frustrated with some of zfs's
> inflexibility around removing drives.

IMHO, ZFS is nice for large storage devices, not so much for regular desktops.
This is why I am hoping BTRFS will solve the resilver issues. (not kept up, is
this still not working?)

--
Joost



Reply | Threaded
Open this post in threaded view
|

Re: Hard drive storage questions

Wol's lists
In reply to this post by Dale-46
On 09/11/18 01:16, Dale wrote:

> Howdy to all,
>
> I have a interesting problem coming up.  Currently, I have two 3TB
> drives for my /home mount point.  A lot of this is videos but some pdf
> files and other documents as well plus a photo collection of family
> stuff etc.
>
> Filesystem                                   Size  Used Avail Use%
> Mounted on
> /dev/mapper/Home2-Home2        5.4T  3.7T  1.8T  68% /home
>
> I've got a little over 25% or so of usable space left.  At that point or
> shortly thereafter, it could start causing some issues according to what
> I've read anyway.  Either way, shortly after that, being full will
> certainly be a issue.  I'm full up on my motherboard SATA ports.  Even
> if I buy a larger drive or drives, I'd have to unplug one to move things
> over and likely repeat that a few times.  I could do that and likely
> will have to anyway but I'm trying to think a little farther ahead.
> Currently I have the more important stuff backed up to a external single
> 6TB USB drive, previous thread on that.  I'm trying to come up with a
> plan that allows me to grow easier and without having to worry about
> running out of motherboard based ports.
>
Rich's ideas sound good. I'm in a similar situation to you - I'm about
to outgrow my existing 3TB setup, but I've got a new 4TB lined up
waiting to go (if I can get it to POST).

Small consumer grade SATA expansions are about £35, you should be able
to get a card that does port multiplier for that money, so that's 8 SATA
drives on that card. Watch out - while I would recommend them, some
cards are advertised as "2 SATA, 2 eSATA". That's only two ports, that
can be jumper-switched between SATA or eSATA. Great for backup or
disaster recovery as well as expansion.

Then think about how you're going to lay your filesystem out. My system
currently has three partitions (/, /var and /home). Mirrored (md raid 1)
onto a second 3TB drive, giving me 6TB total. Actually, not very
conducive to future expansion.

The new system is going to have the bare-metal drives set up as one huge
partition (actually not quite true - they'll be a 3TB and a 1TB for
practical reasons). The 3TB partitions will be raided into a raid 5 with
one of the old drives. The 1TB partitions will be raid 1 (system and
home, basically).

I'll then put lvm on the two raids before actually partitioning that to
give me /, /home, and anything else I want.

At this point, I can now replace any of the drives, grow my raid, my
lvm, or my partitions, at any time and underneath a running system
(okay, I don't have hotplug so I'd have to shut down the system to
physically swap a drive over, but ...).

And I'm thinking about btrfs, but the one thing you have to be really
careful with that is snapshots running out of space.

Cheers,
Wol


Reply | Threaded
Open this post in threaded view
|

Re: Hard drive storage questions

Rich Freeman
In reply to this post by Bill Kenworthy
On Fri, Nov 9, 2018 at 3:17 AM Bill Kenworthy <[hidden email]> wrote:
>
> I'll second your comments on ceph after my experience - great idea for
> large scale systems, otherwise performance is quite poor on small
> systems. Needs at least GB connections with two networks as well as only
> one or two drives per host to work properly.
>
> I think I'll give lizardfs a go - an interesting read.
>

So, ANY distributed/NAS solution is going to want a good network
(gigabit or better), if you care about performance.  With Ceph and the
rebuilds/etc it probably makes an even bigger difference, but lizardfs
still shuttles data around.  With replication any kind of write is
multiplied so even moderate use is going to use a lot of network
bandwidth.  If you're talking about hosting OS images for VMs it is a
big deal.  If you're talking about hosting TV shows for your Myth
server or whatever, it probably isn't as big a deal unless you have 14
tuners and 12 clients.

Lizardfs isn't without its issues.  For my purposes it is fine, but it
is NOT as robust as Ceph.  Finding direct comparisons online is
difficult, but here are some of my observations (having not actually
used either, but having read up on both):

* Ceph (esp for obj store) is designed to avoid bottlenecks.  Lizardfs
has a single master server that ALL metadata requests have to go
through.  When you start getting into dozens of nodes that will start
to be a bottleneck, but it also eliminates some of the rigidity of
Ceph since clients don't have to know where all the data is.  I
imagine it adds a bit of latency to reads.

* Lizardfs defaults to acking writes after the first node receives
them, then replicates them.  Ceph defaults to acking after all
replicas are made.  For any application that takes transactions
seriously there is a HUGE data security difference, but it of course
will lower write latency for lizardfs.

* Lizardfs makes it a lot easier to tweak storage policy at the
directory/file level.  Cephfs basically does this more at the
mountpoint level.

* Ceph CRUSH maps are much more configurable than Lizardfs goals.
With Ceph you could easily say that you want 2 copies, and they have
to be on hard drives with different vendors, and in different
datacenters.  With Lizardfs combining tags like this is less
convenient, and while you could say that you want one copy in rack A
and one in rack B, you can't say that you don't care which 2 as long
as they are different.

* The lizardfs high-availability stuff (equiv of Ceph monitors) only
recently went FOSS, and probably isn't stabilized on most distros.
You can have backup masters that are ready to go, but you need your
own solution for promoting them.

* Lizardfs security seems to be non-existent.  Don't stick it on your
intranet if you are a business.  Fine for home, or for a segregated
SAN, maybe, or you could stick it all behind some kind of VPN and roll
your own security layer.  Ceph security seems pretty robust, but
watching what the ansible playbook did to set it up makes me shudder
at the thought of doing it myself.  Lots of keys that all need to be
in sync so that everything can talk to each other.  I'm not sure if
for clients whether it can outsource authentication to kerberos/etc -
not a need for me but I wouldn't be surprised if this is supported.
The key syncing makes a lot more sense within the cluster itself.

* Lizardfs is MUCH simpler to set up.  For Ceph I recommend the
ansible playbook, though if I were using it in production I'd want to
do some serious config management as it seems rather complex and it
seems like the sort of thing that could take out half a datacenter if
it had a bug.  For Lizardfs if you're willing to use the suggested
hostnames about 95% of it is auto-configuring as storage nodes just
reach out to the default master DNS name and report in, and everything
trusts everything (not just by default - I don't think you even can
lock it down unless you stick every node behind a VPN to limit who can
talk to who).

--
Rich

Reply | Threaded
Open this post in threaded view
|

Re: Hard drive storage questions

Dale-46
In reply to this post by Rich Freeman
Rich Freeman wrote:

> On Thu, Nov 8, 2018 at 8:16 PM Dale <[hidden email]> wrote:
>> I'm trying to come up with a
>> plan that allows me to grow easier and without having to worry about
>> running out of motherboard based ports.
>>
> So, this is an issue I've been changing my mind on over the years.
> There are a few common approaches:
>
> * Find ways to cram a lot of drives on one host
> * Use a patchwork of NAS devices or improvised hosts sharing over
> samba/nfs/etc and end up with a mess of mount points.
> * Use a distributed FS
>
> Right now I'm mainly using the first approach, and I'm trying to move
> to the last.  The middle option has never appealed to me.

And this is what I'm trying to avoid.  Doing one thing, realizing I
should have done it different and then having to spend even more money
to do it the right way.  I'm trying to get advice on what is the best
way forward that I can afford.  Obviously I don't need a setup like
facebook, google or something but I don't want to spend a few hundred
dollars doing something only to realize, it needs to be sold to the next
idiot on Ebay.   ROFL  You're giving me some good options to think on
here.  ;-) 

>
> So, to do more of what you're doing in the most efficient way
> possible, I recommend finding used LSI HBA cards.  These have mini-SAS
> ports on them, and one of these can be attached to a breakout cable
> that gets you 4 SATA ports.  I just picked up two of these for $20
> each on ebay (used) and they have 4 mini-SAS ports each, which is
> capacity for 16 SATA drives per card.  Typically these have 4x or
> larger PCIe interfaces, so you'll need a large slot, or one with a
> cutout.  You'd have to do the math but I suspect that if the card+MB
> supports PCIe 3.0 you're not losing much if you cram it into a smaller
> slot.  If most of the drives are idle most of the time then that also
> demands less bandwidth.  16 fully busy hard drives obviously can put
> out a lot of data if reading sequentially.
>
> You can of course get more consumer-oriented SATA cards, but you're
> lucky to get 2-4 SATA ports on a card that runs you $30.  The mini-SAS
> HBAs get you a LOT more drives per PCIe slot, and your PCIe slots are
> you main limiting factor assuming you have power and case space.
>
> Oh, and those HBA cards need to be flashed into "IT" mode - they're
> often sold this way, but if they support RAID you want to flash the IT
> firmware that just makes them into a bunch of standalone SATA slots.
> This is usually a PITA that involves DOS or whatever, but I have
> noticed some of the software needed in the Gentoo repo.
>
> If you go that route it is just like having a ton of SATA ports in
> your system - they just show up as sda...sdz and so on (no idea where
> it goes after that).  Software-wise you just keep doing what you're
> already doing (though you should be seriously considering
> mdadm/zfs/btrfs/whatever at that point).
>
> That is the more traditional route.
>
> Now let me talk about distributed filesystems, which is the more
> scalable approach.  I'm getting tired of being limited by SATA ports,
> and cases, and such.  I'm also frustrated with some of zfs's
> inflexibility around removing drives.  These are constraints that make
> upgrading painful, and often inefficient.  Distributed filesystems
> offer a different solution.
>
> A distributed filesystem spreads its storage across many hosts, with
> an arbitrary number of drives per host (more or less).  So, you can
> add more hosts, add more drives to a host, and so on.  That means
> you're never forced to try to find a way to cram a few more drives in
> one host.  The resulting filesystem appears as one gigantic filesystem
> (unless you want to split it up), which means no mess of nfs
> mountpoints and so on, and all the other headaches of nfs.  Just as
> with RAID these support redundancy, except now you can lose entire
> hosts without issue.  With many you can even tell it which
> PDU/rack/whatever each host is plugged into, and it will make sure you
> can lose all the hosts in one rack.  You can also mount the filesystem
> on as many hosts as you want at the same time.
>
> They do tend to be a bit more complex.  The big players can scale VERY
> large - thousands of drives easily.  Everything seems to be moving
> towards Ceph/CephFS.  If you were hosting a datacenter full of
> VMs/containers/etc I'd be telling you to host it on Ceph.  However,
> for small scale (which you definitely are right now), I'm not thrilled
> with it.  Due to the way it allocates data (hash-based) anytime
> anything changes you end up having to move all the data around in the
> cluster, and all the reports I've read suggests it doesn't perform all
> that great if you only have a few nodes.  Ceph storage nodes are also
> RAM-hungry, and I want to run these on ARM to save power, and few ARM
> boards have that kind of RAM, and they're very expensive.
>
> Personally I'm working on deploying a cluster of a few nodes running
> LizardFS, which is basically a fork/derivative of MooseFS.  While it
> won't scale nearly as well, below 100 nodes should be fine, and in
> particular it sounds like it works fairly well with only a few nodes.
> It has its pros and cons, but for my needs it should be sufficient.
> It also isn't RAM-hungry.  I'm going to be testing it on some
> RockPro64s, with the LSI HBAs.
>
> I did note that Gentoo lacks a LizardFS client.  I suspect I'll be
> looking to fix that - I'm sure the moosefs ebuild would be a good
> starting point.  I'm probably going to be a whimp and run the storage
> nodes on Ubuntu or whatever upstream targets - they're basically
> appliances as far as I'm concerned.
>
> So, those are the two routes I'd recommend.  Just get yourself an HBA
> if you only want a few more drives.  If you see your needs expanding
> then consider a distributed filesystem.  The advantage of the latter
> is that you can keep expanding it however you want with additional
> drives/nodes/whatever.  If you're going over 20 nodes I'd use Ceph for
> sure - IMO that seems to be the future of this space.
>

This is a lot to think on.  Money wise, and maybe even expansion wise, I
may go with the PCI SATA cards and add drives inside my case.  I have
plenty of power supply since it pulls at most 200 watts and I think my
P/S is like 700 or 800 watts.  I can also add a external SATA card or
another USB drive to do backups with as well.  At some point tho, I may
have to build one of those little tiny systems that is basically nothing
but SATA drive controllers and ethernet enabled.  Have that sitting in a
closet somewhere running some small OS.  I can always just move the
drives from my system to it if needed.

One thing is for sure, you gave a lot of info and different ways to
think on this. 

Thanks to Rich and everyone else to for their thoughts.  It's certainly
helped give me ideas I haven't thought of. 

Dale

:-)  :-)

P. S.  For those who may wonder, my Mom is home and doing pretty well. 
:-D 

Reply | Threaded
Open this post in threaded view
|

Re: Hard drive storage questions

Wol's lists
On 11/11/2018 00:45, Dale wrote:
> This is a lot to think on.  Money wise, and maybe even expansion wise, I
> may go with the PCI SATA cards and add drives inside my case.  I have
> plenty of power supply since it pulls at most 200 watts and I think my
> P/S is like 700 or 800 watts.  I can also add a external SATA card or
> another USB drive to do backups with as well.  At some point tho, I may
> have to build one of those little tiny systems that is basically nothing
> but SATA drive controllers and ethernet enabled.  Have that sitting in a
> closet somewhere running some small OS.  I can always just move the
> drives from my system to it if needed.

https://raid.wiki.kernel.org/index.php/What_is_RAID_and_why_should_you_want_it%3F

(disclaimer - I wrote it :-)

You've got a bunch of questions to ask yourself. Is this an amateur
setup (sounds a bit like it in that it appears to be a home server) or
is it a professional "money no object" setup.

Either way, if you spend good money on good disks (WD Red, Seagate
Ironwolf, etc) then most of your investment will be good to re-purpose.
My current 3TB drives are Barracudas - not a good idea for a
fault-tolerant system - which is why the replacements are Ironwolves.

Then, as that web-page makes clear, do you want your raid/volume
management to be separate from your filesystem - mdraid/lvm under ext4 -
or do you want a filesystem that is hardware-aware like zfs or xfs, or
do you want something like btrfs which tries to be the latter, but is
better used as the former.

One thing to seriously watch out for - many filesystems are aware of the
underlying layer even when you don't expect it. Not sure which
filesystem it is but I remember an email discussion where the filesystem
was aware it was running over mdraid and balanced itself for the
underlying disks. The filesystem developer didn't realise that mdraid
can add and remove disks so the underlying structure can change, and the
recommendation was "once you've set up the raid, if you want to grow
your space move it to a new raid".

At the end of the day, there is no perfect answer, and you need to ask
yourself what you are trying to achieve, and what you can afford.

Cheers,
Wol

Reply | Threaded
Open this post in threaded view
|

Re: Hard drive storage questions

Dale-46
Wol's lists wrote:

> On 11/11/2018 00:45, Dale wrote:
>> This is a lot to think on.  Money wise, and maybe even expansion wise, I
>> may go with the PCI SATA cards and add drives inside my case.  I have
>> plenty of power supply since it pulls at most 200 watts and I think my
>> P/S is like 700 or 800 watts.  I can also add a external SATA card or
>> another USB drive to do backups with as well.  At some point tho, I may
>> have to build one of those little tiny systems that is basically nothing
>> but SATA drive controllers and ethernet enabled.  Have that sitting in a
>> closet somewhere running some small OS.  I can always just move the
>> drives from my system to it if needed.
>
> https://raid.wiki.kernel.org/index.php/What_is_RAID_and_why_should_you_want_it%3F
>
>
> (disclaimer - I wrote it :-)
>
> You've got a bunch of questions to ask yourself. Is this an amateur
> setup (sounds a bit like it in that it appears to be a home server) or
> is it a professional "money no object" setup.
>
> Either way, if you spend good money on good disks (WD Red, Seagate
> Ironwolf, etc) then most of your investment will be good to
> re-purpose. My current 3TB drives are Barracudas - not a good idea for
> a fault-tolerant system - which is why the replacements are Ironwolves.
>
> Then, as that web-page makes clear, do you want your raid/volume
> management to be separate from your filesystem - mdraid/lvm under ext4
> - or do you want a filesystem that is hardware-aware like zfs or xfs,
> or do you want something like btrfs which tries to be the latter, but
> is better used as the former.
>
> One thing to seriously watch out for - many filesystems are aware of
> the underlying layer even when you don't expect it. Not sure which
> filesystem it is but I remember an email discussion where the
> filesystem was aware it was running over mdraid and balanced itself
> for the underlying disks. The filesystem developer didn't realise that
> mdraid can add and remove disks so the underlying structure can
> change, and the recommendation was "once you've set up the raid, if
> you want to grow your space move it to a new raid".
>
> At the end of the day, there is no perfect answer, and you need to ask
> yourself what you are trying to achieve, and what you can afford.
>
> Cheers,
> Wol
>


I've considered RAID before.  For what I have here, doing regular
backups is enough, I hope.  Right now, I backup family pics from my
camera, documents and all the videos, which is a LOT.  If a drive were
to fail, at least I have the backups to go to and would lose little if
anything.  Since I backup pretty regular, I could still recover most
everything that may not have made it to backup. 

I'm on a limited income and this is just a home system.  One thing I try
to do is to find a good way forward first, then do something.  That way
I don't do something that costs money, realize I did it in a bad way and
then have to spend more money doing it again and have parts that I will
likely never need or use again.  I try to skip the worthless part when I
can.  ;-) 

That said, when I do get some drives and they are installed, I'll likely
be asking questions about btrfs, zfs, xfs and anything else that may
apply.  Currently I use LVM and ext4.  I like it pretty well but that
doesn't mean there can't be something better out there. 

Dale

:-)  :-)