Mirror survey re rsync --hard-link (distfiles/releases/experimental/snapshots)

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Mirror survey re rsync --hard-link (distfiles/releases/experimental/snapshots)

Robin H. Johnson-2
Hi,

TL;DR: is your mirror fetching with --hard-link/-H? Would adding it be
problematic for you? Speak up to save yourself bandwidth!

The infrastructure team is having some discussions [1] about splitting apart
the very large distfiles directory, as it now contains ~75000 files, and
this leads some issues on some file-systems.

The distfiles directory is around ~240GB presently, and we would like to
have a reasonable transition period for users, without temporarily
doubling the storage requirement on mirrors.

The best means for this is going to be hard-links, but I don't know how
many mirrors still have that enabled.

None of the rsync setup documentation has ever included it (either right
now or historically). It was strongly encouraged in this mailing list
back in 2004/2005, as part of the new release staging process, but the
most recent mention of hard-linking on the mailing list was Feb 2009.

Ideally, the mirrors that offer rsync service to users for distfiles
would ALSO offer hard-link fetching to users, but that isn't strictly
needed at this point.

To this end, I'm like to know
1. How many mirrors are already fetching from rsync with the
   --hard-link/-H option.
2. How many mirrors would NOT be able to enable rsync fetching with
   hard-links.
3. Does your mirror offer or explicitly reject hard-link data to users?

[1] https://bugs.gentoo.org/534528

--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : [hidden email]
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

signature.asc (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Mirror survey re rsync --hard-link (distfiles/releases/experimental/snapshots)

Erwin Bronkhorst - Studenten Net Twente
Hi,

> To this end, I'm like to know
> 1. How many mirrors are already fetching from rsync with the
>   --hard-link/-H option.

Our mirror at ftp.snt.utwente.nl [1][2] already uses the --hard-link option
to fetch from rsync.

> 2. How many mirrors would NOT be able to enable rsync fetching with
>   hard-links.

n/a for ftp.snt.utwente.nl.

> 3. Does your mirror offer or explicitly reject hard-link data to users?

We offer hard-link data to users.

[1] http://ftp.snt.utwente.nl/pub/linux/gentoo/
[2] http://ftp.snt.utwente.nl/pub/linux/gentoo-portage/

Regards,
Erwin Bronkhorst
Studenten Net Twente


Reply | Threaded
Open this post in threaded view
|

Re: Mirror survey re rsync --hard-link (distfiles/releases/experimental/snapshots)

Alarig Le Lay
In reply to this post by Robin H. Johnson-2
Hi,

On sam. 27 janv. 00:47:39 2018, Robin H. Johnson wrote:
> Hi,
>
> TL;DR: is your mirror fetching with --hard-link/-H? Would adding it be
> problematic for you? Speak up to save yourself bandwidth!

I just configured my mirror [1] with -H, but it doesn’t seem to have
any at this time:
mirror ~ # find /srv/gentoo/ -type f -links +1
mirror ~ #

[1] https://mirror.swordarmor.fr/gentoo/

--
alarig

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Mirror survey re rsync --hard-link (distfiles/releases/experimental/snapshots)

Carlos Carvalho-3
In reply to this post by Robin H. Johnson-2
gentoo.c3sl.ufpr.br uses hardlinks.

Reply | Threaded
Open this post in threaded view
|

Re: Mirror survey re rsync --hard-link (distfiles/releases/experimental/snapshots)

Robin H. Johnson-2
In reply to this post by Alarig Le Lay
On Sat, Jan 27, 2018 at 02:18:26PM +0100, Alarig Le Lay wrote:

> Hi,
>
> On sam. 27 janv. 00:47:39 2018, Robin H. Johnson wrote:
> > Hi,
> >
> > TL;DR: is your mirror fetching with --hard-link/-H? Would adding it be
> > problematic for you? Speak up to save yourself bandwidth!
>
> I just configured my mirror [1] with -H, but it doesn’t seem to have
> any at this time:
> mirror ~ # find /srv/gentoo/ -type f -links +1
> mirror ~ #
That's correct, there aren't any in use presently.
I added an explicit test case:
hardlink-a
hardlink-b

It should go out to mirrors in the next ~24 hours or so.

--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : [hidden email]
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

signature.asc (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Mirror survey re rsync --hard-link (distfiles/releases/experimental/snapshots)

SoEasyTo Mirrors Manager
On 2018-01-29 08:54, Robin H. Johnson wrote:

>
>> On Sat, Jan 27, 2018 at 02:18:26PM +0100, Alarig Le Lay wrote: Hi,
>>
>>> On sam. 27 janv. 00:47:39 2018, Robin H. Johnson wrote: Hi,
>>>
>>> TL;DR: is your mirror fetching with --hard-link/-H? Would adding it
>>> be
>>> problematic for you? Speak up to save yourself bandwidth!
>> I just configured my mirror [1] with -H, but it doesn't seem to have
>> any at this time:
>> mirror ~ # find /srv/gentoo/ -type f -links +1
>> mirror ~ #
> That's correct, there aren't any in use presently.
> I added an explicit test case:
> hardlink-a
> hardlink-b
>
> It should go out to mirrors in the next ~24 hours or so.

mirrors.soeasyto.com is using hard-links when syncing from master
distfiles and master portage.

There is no particular configuration to forbid them to be used by
clients.

Anyway, I did not find the test case you added on saturday on my mirror.

Reply | Threaded
Open this post in threaded view
|

Re: Mirror survey re rsync --hard-link (distfiles/releases/experimental/snapshots)

aditsu
In reply to this post by Robin H. Johnson-2
Hi, reporting for gentoo.aditsu.net

On 27/01/2018 08:47, Robin H. Johnson wrote:
> To this end, I'm like to know
> 1. How many mirrors are already fetching from rsync with the
>     --hard-link/-H option.
> 2. How many mirrors would NOT be able to enable rsync fetching with
>     hard-links.
I have added -H just now.

> 3. Does your mirror offer or explicitly reject hard-link data to users?
My mirror is http only.


On 29/01/2018 15:54, Robin H. Johnson wrote:
> I added an explicit test case:
> hardlink-a
> hardlink-b
I can see it now:

# find . -type f -links +1
./diradm-2.9.7.1.tar.bz2
./hardlink-a
./hardlink-b
# find . -samefile hardlink-a
./diradm-2.9.7.1.tar.bz2
./hardlink-a
./hardlink-b

Hope that answers all your questions.

Best regards,
Adrian

Reply | Threaded
Open this post in threaded view
|

Re: Mirror survey re rsync --hard-link (distfiles/releases/experimental/snapshots)

Alarig Le Lay
Hi,

On mar. 30 janv. 16:08:35 2018, Adrian Sándor wrote:

> I can see it now:
>
> # find . -type f -links +1
> ./diradm-2.9.7.1.tar.bz2
> ./hardlink-a
> ./hardlink-b
> # find . -samefile hardlink-a
> ./diradm-2.9.7.1.tar.bz2
> ./hardlink-a
> ./hardlink-b
>
> Hope that answers all your questions.
Does ftp.ussg.iu.edu support -H?

mirror ~ # find /srv/gentoo/ -type f -links +1
mirror ~ # ls -li /srv/gentoo/distfiles/{diradm-2.9.7.1.tar.bz2,hardlink-a,hardlink-b}
6044281 -rw-rw-r-- 1 nginx  1030 126172 Nov  9  2012 /srv/gentoo/distfiles/diradm-2.9.7.1.tar.bz2
6030536 -rw-rw-r-- 1 nginx nginx 126172 Nov  9  2012 /srv/gentoo/distfiles/hardlink-a
6031953 -rw-rw-r-- 1 nginx nginx 126172 Nov  9  2012 /srv/gentoo/distfiles/hardlink-b
mirror ~ # rm -v /srv/gentoo/distfiles/{diradm-2.9.7.1.tar.bz2,hardlink-a,hardlink-b}
removed '/srv/gentoo/distfiles/diradm-2.9.7.1.tar.bz2'
removed '/srv/gentoo/distfiles/hardlink-a'
removed '/srv/gentoo/distfiles/hardlink-b'
mirror ~ # rsync -av --delete -H ftp.ussg.iu.edu::gentoo-distfiles /srv/gentoo/
[…]
distfiles/diradm-2.9.7.1.tar.bz2
distfiles/hardlink-a
distfiles/hardlink-b
mirror ~ # find /srv/gentoo/ -type f -links +1
total size is 408,793,946,328  speedup is 23,599.52
mirror ~ # ls -li /srv/gentoo/distfiles/{diradm-2.9.7.1.tar.bz2,hardlink-a,hardlink-b}
6031953 -rw-rw-r-- 1 1030 1030 126172 Nov  9  2012 /srv/gentoo/distfiles/diradm-2.9.7.1.tar.bz2
6070537 -rw-rw-r-- 1 1030 1030 126172 Nov  9  2012 /srv/gentoo/distfiles/hardlink-a
6070547 -rw-rw-r-- 1 1030 1030 126172 Nov  9  2012 /srv/gentoo/distfiles/hardlink-b

--
alarig

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Mirror survey re rsync --hard-link (distfiles/releases/experimental/snapshots)

Robin H. Johnson-2
On Tue, Jan 30, 2018 at 10:45:35AM +0100, Alarig Le Lay wrote:

> Hi,
>
> On mar. 30 janv. 16:08:35 2018, Adrian Sándor wrote:
> > I can see it now:
> >
> > # find . -type f -links +1
> > ./diradm-2.9.7.1.tar.bz2
> > ./hardlink-a
> > ./hardlink-b
> > # find . -samefile hardlink-a
> > ./diradm-2.9.7.1.tar.bz2
> > ./hardlink-a
> > ./hardlink-b
> >
> > Hope that answers all your questions.
>
> Does ftp.ussg.iu.edu support -H?
(snip)
Hmm, they used to. I'll check with them.

For EU mirrors, hopefully I have a better source you can switch to soon
anyway.

--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : [hidden email]
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

signature.asc (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Mirror survey re rsync --hard-link (distfiles/releases/experimental/snapshots)

Alarig Le Lay
On mar. 30 janv. 18:24:32 2018, Robin H. Johnson wrote:
> For EU mirrors, hopefully I have a better source you can switch to soon
> anyway.

I’m in EU, so it sounds good :)
I just followed https://wiki.gentoo.org/wiki/Project:Infrastructure/Mirrors/Source#Initial_setup

--
alarig

signature.asc (499 bytes) Download Attachment