Re: Recent changes to gentoo-portage

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Recent changes to gentoo-portage

Jeremy Olexa-3
On Mon, Apr 2, 2012 at 7:49 AM, Carsten Otto
<[hidden email]> wrote:
> Dear Gentoo admins,
>
> it seems you made some changes to the data you distribute as
> "gentoo-portage".

Well, the infra team didn't make any changes. The gentoo-portage tree
is a living thing that adapts as the software becomes more complex.
There was a change to better protect users against timestamp oddities
(https://bugs.gentoo.org/show_bug.cgi?id=409445)

> Before the change we had to manually take care that the corresponding
> file system has enough inodes, as gentoo-portage seems to have very many
> files with only a tiny size. The recent change seems to have increased
> the number of inodes substantially. This caused our file system to run
> out of inodes and we had to manually create a fresh file system
> (extending the number of inodes is not possible without also increasing
> the size of the file system).

We don't suggest any particular filesystem. That is up to you to
implement for your own needs. I'm sorry that you ran out of inodes
(our infrastructure did too, our ramdisks became full)

> To my knowledge there was no announcement for this change. With such an
> announcement the time of having incomplete data (roughly one day in our
> case) you may have observed for our mirror (which is part of some
> round-robin DNS scheme in the gentoo.org domain) could have been lowered
> to a few minutes.

Your mirror is not part of the rsync.gentoo.org rotation. We reserve
that for officially managed Gentoo hosts only.

> Furthermore, it seems that your master mirror currently is overloaded.
> My guess is that many mirrors experience problems similar to ours and
> re-sync more often than normally.

There was just a bug that was causing metadata/md5-cache to be
resynced everytime. (https://bugs.gentoo.org/show_bug.cgi?id=410505)
This may have explained the load, but it shouldn't have been too bad,
that box that you access only transfers 15G/day.

>
> Please tell us ("us" as in "all the mirror admins") what happened and
> how to deal with the situation. Please also take care to communicate
> changes in the future.

Again, sorry for the trouble(s). This was the first time that such a
large increase in filesize happened. To be honest, we did not expect
fallout like this. Now we know for next time, if there is a next time.
Also, there is plans to reduce the footprint of files/inodes in the
short term so it should be back to previous sizes in some time.

-Jeremy

Reply | Threaded
Open this post in threaded view
|

Re: Re: Recent changes to gentoo-portage

Robin H. Johnson-2
On Mon, Apr 02, 2012 at 05:27:08PM +0200, Carsten Otto wrote:

> Hi,
>
> thank you for the quick reply.
>
> On Mon, Apr 02, 2012 at 09:57:11AM -0500, Jeremy Olexa wrote:
> > Well, the infra team didn't make any changes. The gentoo-portage tree
> > is a living thing that adapts as the software becomes more complex.
> > There was a change to better protect users against timestamp oddities
> > (https://bugs.gentoo.org/show_bug.cgi?id=409445)
> >
> > We don't suggest any particular filesystem. That is up to you to
> > implement for your own needs. I'm sorry that you ran out of inodes
> > (our infrastructure did too, our ramdisks became full)
>
> I know that I am free to pick my file system. I just think that
> increasing the amounts of inodes substantially (did you double?) is
> something worth mentioning. Furthermore, I don't really care if this is
> caused by team A or team B inside the Gentoo developer world. This is a
> change that is of huge interest to everyone mirroring portage and using
> a file system that has a restricted number of inodes (which is a huge
> percentage, I'd guess).
I'm very interested in your claim of double.

Old metadata/cache
Files: 30257
Dirs: 156
Raw Bytes: 24238169
4k blocks: 31165
Size @ 4k blocks: 124660k

New metadata/md5-cache
Files: 30256
Dirs: 155
Raw Bytes: 32650108
4k blocks: 31037
Size @ 4k blocks: 124148k

Complete tree:
Files: 159761
Dirs: 23166
Raw bytes: 263333212
4k blocks: 181851
Size @ 4k blocks: 727404k

So a 20% increase in inodes yes, but not double.

--
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : [hidden email]
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Reply | Threaded
Open this post in threaded view
|

unsubscribe

Vladimir Brik-2