backing up a partition

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

backing up a partition

Philip Webb-2
I want to make a copy of a partition which I can use to replace it,
if some catastrophe damages the partition or wipes it out ;
it needs to be byte-byte identical, incl all permissions.

Can I use 'dd' ? -- eg 'dd if=/mnt/xxx of=/mnt/yyy',
where the partition has been mounted at  /mnt/xxx
& a USB stick has been mounted at  /mnt/yyy .  Will that do the job ?

There seems also to be an issue re 'bs=<some number of bytes>' :
what size is best ?  i plan to use USB 3.0 for quicker copying.

Does it matter how the USB stick is formatted ?
Can I use a raw stick with the usual default VFAT formatting ?
Might it be better to replace that with a Linux FS, eg Ext2 ?

--
========================,,============================================
SUPPORT     ___________//___,   Philip Webb
ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca


Reply | Threaded
Open this post in threaded view
|

Re: backing up a partition

Bas Zoutendijk
Dear Philip,

On Fri 24 Aug 2018 at 04:32:23 -0400, Philip Webb wrote:
> I want to make a copy of a partition which I can use to replace it,
> if some catastrophe damages the partition or wipes it out ;
> it needs to be byte-byte identical, incl all permissions.
>
> Can I use 'dd' ? -- eg 'dd if=/mnt/xxx of=/mnt/yyy',
> where the partition has been mounted at  /mnt/xxx
> & a USB stick has been mounted at  /mnt/yyy .  Will that do the job ?

  If you want  to make a byte-by-byte copy  of a partition,  the easiest
way  in   my  opinion  is   to  dd  the   (unmounted)  partition,   e.g.
if=/dev/sda1.  You  could  also do  that  with  the  entire drive,  e.g.
if=/dev/sda.

> There seems also to be an issue re 'bs=<some number of bytes>' :
> what size is best ?  i plan to use USB 3.0 for quicker copying.

  For optimal  performance the block size  should be  a multiple  of the
device’s physical  block size.  Old hard drives  had 512-B blocks,  more
recent  ones  have  larger  4-KiB  physical  blocks  that  are sometimes
software-emulated to  appear as  512-B blocks.  In  my experience  it is
faster to use even larger block sizes; I usually use bs=1M.

> Does it matter how the USB stick is formatted ?
> Can I use a raw stick with the usual default VFAT formatting ?
> Might it be better to replace that with a Linux FS, eg Ext2 ?

  The lowest overhead  is when you write the back-up  to a raw partition
of  the same  size,  e.g.  of=/dev/sdb1.  You can  also  write  it  to a
filesystem,  which  might make  it easier  to manage  multiple back-ups.
Since all permissions etc.  are stored  in the back-up file,  the choice
of file system does not really matter,  though the back-up file might be
too large for VFAT (max ~4 GiB).

                                                              Sincerely,

                                                                 Bas

--
Sebastiaan L. Zoutendijk | [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: backing up a partition

Neil Bothwick
In reply to this post by Philip Webb-2
On Fri, 24 Aug 2018 04:32:23 -0400, Philip Webb wrote:

> I want to make a copy of a partition which I can use to replace it,
> if some catastrophe damages the partition or wipes it out ;
> it needs to be byte-byte identical, incl all permissions.
>
> Can I use 'dd' ? -- eg 'dd if=/mnt/xxx of=/mnt/yyy',
> where the partition has been mounted at  /mnt/xxx
> & a USB stick has been mounted at  /mnt/yyy .  Will that do the job ?

It will, provided the USB stick is large enough.
 
> There seems also to be an issue re 'bs=<some number of bytes>' :
> what size is best ?  i plan to use USB 3.0 for quicker copying.

I usually use 4M but anything over 1M gives similar results. These days
I use dcfldd rather than dd and it gives a nice progress display and
also works out the block size for itself.
 
> Does it matter how the USB stick is formatted ?
> Can I use a raw stick with the usual default VFAT formatting ?
> Might it be better to replace that with a Linux FS, eg Ext2 ?

It doesn't matter how the stick *was* formatted because you are
overwriting it when you dd. The resulting stick won't even have a
partition table. Alternatively, format the stick and dd to a file on the
stick.

Bear in mind that dd copies the whole partition, including the bits (and
bytes) not currently in use by the filesystem, so it is inefficient in
space terms. If you just want a backup of the contents of the partition,
would tar not suffice?


--
Neil Bothwick

If Yoda so strong in force is, why words in right order he cannot put?

attachment0 (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: backing up a partition

Mick-10
In reply to this post by Philip Webb-2
On Friday, 24 August 2018 09:32:23 BST Philip Webb wrote:
> I want to make a copy of a partition which I can use to replace it,
> if some catastrophe damages the partition or wipes it out ;
> it needs to be byte-byte identical, incl all permissions.

You are looking to clone a partition.  This is a bit by bit identical copy of
the original partition.  You could clone a partition into another, so you end
up with two identical partitions including their boot record, fs and contents,
or save the cloned partition into an image file.

However, you may prefer to use clonezilla instead of dd.  The dd command will
copy each and every bit and byte of the partition whether it has data on it or
not.  It is not particularly efficient.  Clonezilla will perform better at
this task.

Personally, I would only keep a back up of the filesystem contents with e.g.
rsync, and reformat the partition and restore its contents in the case of a
disaster recovery scenario.  I would prefer a fs backup, especially in the
case of a USB stick being the backup storage media.  This is because rsync
will only copy data which has changed since the last backup, rather than
overwriting what is on the USB already.  I expect using something like dd
repeatedly will reduce considerably the life of the USB stick.


> Can I use 'dd' ? -- eg 'dd if=/mnt/xxx of=/mnt/yyy',
> where the partition has been mounted at  /mnt/xxx
> & a USB stick has been mounted at  /mnt/yyy .  Will that do the job ?

Not like that.  If you intend to create a clone of the partition, rather than
its filesystem, you need to copy the whole block device, not its mount point.

dd if=/dev/sdXx of=/media/$USER/<LABEL>/clone_of_sdXx.img bs=4096
conv=notrunc,fsync status=progress

Where /media/$USER/<LABEL>/ is assumed to be the mountpoint of your mounted
USB storage device.  The above will save an image file of the cloned partition
as 'clone_of_sdXx.img' within the existing filesystem of the USB stick.  You
could then copy this partition image to a secondary storage if you wish,
encrypt it, compress it, etc.

You'll be able to mount this image with '-o loop' and examing or use its
contents.

mount /media/$USER/<LABEL>/clone_of_sdXx.img /mnt/My_partition_image -o loop

ls -l /mnt/My_partition_image


To create a full partition clone on the USB stick rather than an image file
change the of= operand into the block device your USB stick is recognised as;
e.g.

dd if=/dev/sdXx of=/dev/sdYy bs=4096 conv=notrunc,fsync status=progress

CAUTION:  If you get /dev/sdYy wrong you *will* lose data on the device you
overwrite.

You may have problems when you try to mount the cloned USB stick with the
original partition in place, because they will both have identical filesystem
UUIDs.  I haven't tried this to know what errors it may produce.


> There seems also to be an issue re 'bs=<some number of bytes>' :
> what size is best ?  i plan to use USB 3.0 for quicker copying.

You can test what block size will transfer data faster on your particular
hardware setup. I assume in the example below the I/O bottleneck is your USB,
rather than your HDD.

dd if=/dev/zero of=/media/$USER/<LABEL>/testfile bs=1024 count=1000000

rm /media/$USER/<LABEL>/testfile && sync

dd if=/dev/zero of=/media/$USER/<LABEL>/testfile bs=2048 count=500000

rm /media/$USER/<LABEL>/testfile && sync

and so on, each time increasing the bs to 4096, 8192, ... and halving the
count to copy the same amount of data.  You will soon find the transfer speed
arrives at a peak and then reduces when an I/O bottleneck is reached.  In the
above example I have used a 1GB file, but you may want to increase this to a
larger size to make sure you saturate your buffers.


> Does it matter how the USB stick is formatted ?
> Can I use a raw stick with the usual default VFAT formatting ?
> Might it be better to replace that with a Linux FS, eg Ext2 ?

If you are using the USB stick to save an image file to it, then you will come
across the VFAT file size limit of 4GB.  So, use exFAT, or ext2.

If you are using a USB stick to create a partition clone it doesn't matter
what the USB fs type is.  It will be replaced with that of the original
partition.

HTH.
--
Regards,
Mick

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: backing up a partition

Rich Freeman
On Fri, Aug 24, 2018 at 6:09 AM Mick <[hidden email]> wrote:
>
> However, you may prefer to use clonezilla instead of dd.  The dd command will
> copy each and every bit and byte of the partition whether it has data on it or
> not.  It is not particularly efficient.  Clonezilla will perform better at
> this task.
>
> Personally, I would only keep a back up of the filesystem contents with e.g.
> rsync, and reformat the partition and restore its contents in the case of a
> disaster recovery scenario.

Just to summarize the sorts of options you have:

dd = bit level copy.  Output is the size of the partition, period,
though you could compress the output by piping it into a compression
utility/etc.  Restored partition is identical to original, including
unallocated space, file fragmentation, etc.

clonezilla/partimage/etc = sparse bit level copy.  Output is the size
of all blocks that contain useful data, and can be further compressed.
Restored partition will contain zeros in the place of free space, but
will still preserve file fragmentation, special filesystem features,
etc.  Basically these tools operate like dd at a block level, but they
first identify which blocks are used/unused.  Savings is minimal for a
full filesystem, and substantial for a near-empty one.  These tools
will fall back to dd if they can't identify free space, and can
support a wider variety of filesystems quickly because they don't have
to be able to mount/read the filesystem, just figure out which blocks
matter.  I'll also note that with clonezilla you get a fairly nice
all-in-one bootable image that can store these images remotely via
ssh/samba/etc, which makes restoring images onto bare metal very easy.

tar/rsync/etc = file level copy.  Output is the logical size of all
the files on the filesystem.  Restore partition will only contain file
contents - details like fragmentation, trailing unused space in
blocks, unused space in general, or many filesystem-specific features
like snapshots/etc will NOT be preserved.  On the other hand it is
trivial to restore this data to any filesystem of any type of any
sufficient size.  The other solutions make resizing or changing
filesystems more-or-less impossible unless you can mount the image
files and then do a subsequent file-level copy (which is no different
than doing a file level copy in the first place).

I'd toss in one other general category:

dump/send/etc - filesystem-specific serializing tools.  The tools are
specific to the filesystem, so you can't just point them at a whole
hard drive with varying partition types like you can with clonezilla.
They may or may not reproduce details like fragmentation, but they
will efficiently store the actual data and will reproduce all
filesystem-specific features (snapshots, special attributes, etc).
They may also contain features that make them more efficient
(especially for incremental backups) because they can use an algorithm
suited for the low-level data structures employed by the filesystem,
instead of doing scanning at the file/directory level.  For example,
it could just read all the metadata on the disk sequentially as it is
physically stored on the disk, instead of traversing it from root down
to leaf in the directory hierarchy which could result in lots of
seeks.  Filesystems like btrfs/zfs have data structures that make it
VERY efficient to compare two related snapshots and find just the
differences between them, including differences of one block in the
middle of a large file without having to read the whole file.
Restoration usually is flexible with regard to filesystem size, but
not type.  That is, if you have a 100GB filesystem with 20GB of data,
you could restore it to a 30GB filesystem of the same type, but not
one of a different type as with tar.

The best solution for you obviously depends on your needs.  I try to
go with the last category in general as it is far more efficient.
But, clonezilla is my general tool for replicating whole systems/etc
since it does that so well and works with anything.  For partial
backups of high-value data I use duplicity, which is file-level (and
supports various cloud/etc options for storage).

--
Rich

Reply | Threaded
Open this post in threaded view
|

Re: backing up a partition

William Kenworthy
On 24/08/18 20:53, Rich Freeman wrote:

> On Fri, Aug 24, 2018 at 6:09 AM Mick <[hidden email]> wrote:
>> However, you may prefer to use clonezilla instead of dd.  The dd command will
>> copy each and every bit and byte of the partition whether it has data on it or
>> not.  It is not particularly efficient.  Clonezilla will perform better at
>> this task.
>>
>> Personally, I would only keep a back up of the filesystem contents with e.g.
>> rsync, and reformat the partition and restore its contents in the case of a
>> disaster recovery scenario.
> Just to summarize the sorts of options you have:
>
> dd = bit level copy.  Output is the size of the partition, period,
> though you could compress the output by piping it into a compression
> utility/etc.  Restored partition is identical to original, including
> unallocated space, file fragmentation, etc.
>
> clonezilla/partimage/etc = sparse bit level copy.  Output is the size
> of all blocks that contain useful data, and can be further compressed.
> Restored partition will contain zeros in the place of free space, but
> will still preserve file fragmentation, special filesystem features,
> etc.  Basically these tools operate like dd at a block level, but they
> first identify which blocks are used/unused.  Savings is minimal for a
> full filesystem, and substantial for a near-empty one.  These tools
> will fall back to dd if they can't identify free space, and can
> support a wider variety of filesystems quickly because they don't have
> to be able to mount/read the filesystem, just figure out which blocks
> matter.  I'll also note that with clonezilla you get a fairly nice
> all-in-one bootable image that can store these images remotely via
> ssh/samba/etc, which makes restoring images onto bare metal very easy.
>
> tar/rsync/etc = file level copy.  Output is the logical size of all
> the files on the filesystem.  Restore partition will only contain file
> contents - details like fragmentation, trailing unused space in
> blocks, unused space in general, or many filesystem-specific features
> like snapshots/etc will NOT be preserved.  On the other hand it is
> trivial to restore this data to any filesystem of any type of any
> sufficient size.  The other solutions make resizing or changing
> filesystems more-or-less impossible unless you can mount the image
> files and then do a subsequent file-level copy (which is no different
> than doing a file level copy in the first place).
>
> I'd toss in one other general category:
>
> dump/send/etc - filesystem-specific serializing tools.  The tools are
> specific to the filesystem, so you can't just point them at a whole
> hard drive with varying partition types like you can with clonezilla.
> They may or may not reproduce details like fragmentation, but they
> will efficiently store the actual data and will reproduce all
> filesystem-specific features (snapshots, special attributes, etc).
> They may also contain features that make them more efficient
> (especially for incremental backups) because they can use an algorithm
> suited for the low-level data structures employed by the filesystem,
> instead of doing scanning at the file/directory level.  For example,
> it could just read all the metadata on the disk sequentially as it is
> physically stored on the disk, instead of traversing it from root down
> to leaf in the directory hierarchy which could result in lots of
> seeks.  Filesystems like btrfs/zfs have data structures that make it
> VERY efficient to compare two related snapshots and find just the
> differences between them, including differences of one block in the
> middle of a large file without having to read the whole file.
> Restoration usually is flexible with regard to filesystem size, but
> not type.  That is, if you have a 100GB filesystem with 20GB of data,
> you could restore it to a 30GB filesystem of the same type, but not
> one of a different type as with tar.
>
> The best solution for you obviously depends on your needs.  I try to
> go with the last category in general as it is far more efficient.
> But, clonezilla is my general tool for replicating whole systems/etc
> since it does that so well and works with anything.  For partial
> backups of high-value data I use duplicity, which is file-level (and
> supports various cloud/etc options for storage).
>
and another category - do a proper backup using backup software. 
Copying the raw partition has some disadvantages - difficult (but not
impossible) if you want to restore to new hardware (on failure), harder
to restore if you decide to change the underlying filesystem, takes far
longer to copy than a backup but the biggest problem is no versioning. 
Raw copies are great if you want to do an immediate restore, but hard
work or useless after a few days of changes.  Think of it this way - in
almost all cases its the data that's important, not whats holding the data.


BillK



Reply | Threaded
Open this post in threaded view
|

Re: backing up a partition

Philip Webb-2
In reply to this post by Philip Webb-2
Thanks for the replies :
it looks as if 'tar' mb adequate, but I'll think re it & make a test.

--
========================,,============================================
SUPPORT     ___________//___,   Philip Webb
ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca


Reply | Threaded
Open this post in threaded view
|

Re: backing up a partition

antlists
On 25/08/18 10:04, Philip Webb wrote:
> Thanks for the replies :
> it looks as if 'tar' mb adequate, but I'll think re it & make a test.
>
Search for the thread "Backup questions" by Dale 8/8/18.

That has a load of ideas, and depending on your host filesystem you have
other options. For example, btrfs can clone a filesystem to backup media.

Cheers,
Wol

Reply | Threaded
Open this post in threaded view
|

Re: backing up a partition

Philip Webb-2
In reply to this post by Philip Webb-2
180825 Philip Webb wrote:
> Thanks for the replies :
> it looks as if 'tar' mb adequate, but I'll think re it & make a test.

I used 'tar -a' to copy the contents of the partition to a USB stick,
then copied them back to another partition, updated Lilo
& it booted successfully into the new partition, which works the same.

--
========================,,============================================
SUPPORT     ___________//___,   Philip Webb
ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca


Reply | Threaded
Open this post in threaded view
|

Re: backing up a partition

Michael Jones
try fsarchiver

On Thu, Aug 30, 2018 at 11:53 AM, Philip Webb <[hidden email]> wrote:
180825 Philip Webb wrote:
> Thanks for the replies :
> it looks as if 'tar' mb adequate, but I'll think re it & make a test.

I used 'tar -a' to copy the contents of the partition to a USB stick,
then copied them back to another partition, updated Lilo
& it booted successfully into the new partition, which works the same.

--
========================,,============================================
SUPPORT     ___________//___,   Philip Webb
ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca



Reply | Threaded
Open this post in threaded view
|

Re: backing up a partition

Jacques Montier-3


Le ven. 31 août 2018 04:48, Michael Jones <[hidden email]> a écrit :
try fsarchiver

On Thu, Aug 30, 2018 at 11:53 AM, Philip Webb <[hidden email]> wrote:
180825 Philip Webb wrote:
> Thanks for the replies :
> it looks as if 'tar' mb adequate, but I'll think re it & make a test.

I used 'tar -a' to copy the contents of the partition to a USB stick,
then copied them back to another partition, updated Lilo
& it booted successfully into the new partition, which works the same.

--
========================,,============================================
SUPPORT     ___________//___,   Philip Webb
ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca




Hi all 

I use fsarchiver and SystemRescueCD for years.
It works fine.

Regards 

Jacques