[QA] New policy: 'files' directory must not be larger than 32 KiB

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[QA] New policy: 'files' directory must not be larger than 32 KiB

Michał Górny-5
Hello, everyone.

It's my pleasure to announce that with a majority vote the QA team has
accepted a new policy. The accepted wording is:

  Total size of 'files' subdirectory of a package should not be larger
  than 32 KiB. If the package needs more auxiliary files, they should
  be put into SRC_URI e.g. via tarballs.

(the total size being computed as a sum of apparent file sizes)

The relevant policy vote is finishing at bug #633758 [1]. The CI reports
 [2] were updated to report packages whose 'files' directories exceed
64 KiB, to avoid adding many new warnings at once. The limit will
be lowered down to 32 KiB as packages are fixed to comply with the new
policy.

At the same time, I would like to explicitly remind developers that
the spirit of the policy is 'do not let "files" grow large', not 'make
sure you're one byte less than 32769.' Do not argue that your package
exceeds the limit only by few bytes -- even if it gets close to the
limit, then it means it's way too large.


Motivation
==========

Repoman & pkgcheck so far checked for a single file in 'files' directory
that exceeded 20 KiB in size. This check has been criticized multiple
times, most notably because it was triggered on packages with a single
large-ish file but at the same time permitted packages to have a lot of
small-ish files, with the latter consuming much more space. Furthermore,
some developers explicitly worked around the check by splitting large
patches into two smaller parts.

The new policy aims to serve the same purpose but be more fair
by counting the total size of all files.


Rationale
=========

At this moment, syncing the repository implies fetching 'files'
directories of all packages, even though the relevant files are used
only when a ebuild referencing them is being built. This means that our
users fetch many files that they will never use -- either because they
don't need the package in question, or because the file belongs
to an old version.

For example, 'du -h app-shells/bash/files' states 232K while only three
of those files are used by the newest version, and everything else are
patches for old versions. And in case of bash, we're keeping those
versions pretty much 'forever'.

The new policy mostly targets large patchsets and files relevant to old
package versions. By removing them from the repository, we're hoping to
reduce the growth of its size a bit and reduce the amount of data
transferred via rsync.


[1]:https://bugs.gentoo.org/633758
[2]:https://qa-reports.gentoo.org/output/gentoo-ci/output.html

--
Best regards,
Michał Górny


Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

Mike Gilbert-2
On Sun, Dec 17, 2017 at 8:21 AM, Michał Górny <[hidden email]> wrote:

> Hello, everyone.
>
> It's my pleasure to announce that with a majority vote the QA team has
> accepted a new policy. The accepted wording is:
>
>   Total size of 'files' subdirectory of a package should not be larger
>   than 32 KiB. If the package needs more auxiliary files, they should
>   be put into SRC_URI e.g. via tarballs.
>
> (the total size being computed as a sum of apparent file sizes)
>
> The relevant policy vote is finishing at bug #633758 [1]. The CI reports
>  [2] were updated to report packages whose 'files' directories exceed
> 64 KiB, to avoid adding many new warnings at once. The limit will
> be lowered down to 32 KiB as packages are fixed to comply with the new
> policy.
>
> At the same time, I would like to explicitly remind developers that
> the spirit of the policy is 'do not let "files" grow large', not 'make
> sure you're one byte less than 32769.' Do not argue that your package
> exceeds the limit only by few bytes -- even if it gets close to the
> limit, then it means it's way too large.

I just want to voice my opinion on this: as a developer, this policy
is a royal pain in the ass.

I would ask the council to please increase this limit to at least 100
KiB, preferably more.

Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

Mike Gilbert-2
On Sun, Dec 17, 2017 at 1:39 PM, Mike Gilbert <[hidden email]> wrote:

> On Sun, Dec 17, 2017 at 8:21 AM, Michał Górny <[hidden email]> wrote:
>> Hello, everyone.
>>
>> It's my pleasure to announce that with a majority vote the QA team has
>> accepted a new policy. The accepted wording is:
>>
>>   Total size of 'files' subdirectory of a package should not be larger
>>   than 32 KiB. If the package needs more auxiliary files, they should
>>   be put into SRC_URI e.g. via tarballs.
>>
>> (the total size being computed as a sum of apparent file sizes)
>>
>> The relevant policy vote is finishing at bug #633758 [1]. The CI reports
>>  [2] were updated to report packages whose 'files' directories exceed
>> 64 KiB, to avoid adding many new warnings at once. The limit will
>> be lowered down to 32 KiB as packages are fixed to comply with the new
>> policy.
>>
>> At the same time, I would like to explicitly remind developers that
>> the spirit of the policy is 'do not let "files" grow large', not 'make
>> sure you're one byte less than 32769.' Do not argue that your package
>> exceeds the limit only by few bytes -- even if it gets close to the
>> limit, then it means it's way too large.
>
> I just want to voice my opinion on this: as a developer, this policy
> is a royal pain in the ass.
>
> I would ask the council to please increase this limit to at least 100
> KiB, preferably more.

Please substitute "QA team" for "council" in the above message. Thanks.

Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

Andreas K. Huettel
In reply to this post by Michał Górny-5
Am Sonntag, 17. Dezember 2017, 14:21:10 CET schrieb Michał Górny:

> Hello, everyone.
>
> It's my pleasure to announce that with a majority vote the QA team has
> accepted a new policy. The accepted wording is:
>
>   Total size of 'files' subdirectory of a package should not be larger
>   than 32 KiB. If the package needs more auxiliary files, they should
>   be put into SRC_URI e.g. via tarballs.
>
> (the total size being computed as a sum of apparent file sizes)
>
This is hard (but not impossible I guess) for toolchain stuff, even though we
have already most patches in external tarballs, since we want to keep some old
versions of packages available.

E.g.

huettel@pinacolada ~/Gentoo/gentoo/sys-libs/glibc/files $ find . -type f
./2.17/glibc-2.17-hardened-pie.patch
./2.18/glibc-2.18-gentoo-stack_chk_fail.c
./2.18/glibc-2.18-hardened-inittls-nosysenter.patch
./2.18/glibc-2.18-gentoo-chk_fail.c
./2.19/glibc-2.19-ia64-gcc-4.8-reloc-hack.patch
./2.19/glibc-2.19-hardened-configure-picdefault.patch
./2.20/glibc-2.20-gentoo-chk_fail.c
./2.20/glibc-2.20-gentoo-stack_chk_fail.c                                                                                                                                                            
./2.20/glibc-2.20-hardened-inittls-nosysenter.patch                                                                                                                                                  
./2.10/glibc-2.10-gentoo-chk_fail.c                                                                                                                                                                  
./2.10/glibc-2.10-hardened-inittls-nosysenter.patch                                                                                                                                                  
./2.10/glibc-2.10-hardened-configure-picdefault.patch
./nscd.tmpfilesd
./2.25/glibc-2.25-gentoo-chk_fail.c
./nscd.service
huettel@pinacolada ~/Gentoo/gentoo/sys-libs/glibc/files $ du -sh .
152K    .

--
Andreas K. Hüttel
[hidden email]
Gentoo Linux developer
(council, toolchain, perl, libreoffice, comrel)

signature.asc (1000 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

Lars Wendler
In reply to this post by Mike Gilbert-2
Am Sun, 17 Dec 2017 13:40:35 -0500
schrieb Mike Gilbert <[hidden email]>:

> On Sun, Dec 17, 2017 at 1:39 PM, Mike Gilbert <[hidden email]>
> wrote:
> > On Sun, Dec 17, 2017 at 8:21 AM, Michał Górny <[hidden email]>
> > wrote:  
> >> Hello, everyone.
> >>
> >> It's my pleasure to announce that with a majority vote the QA team
> >> has accepted a new policy. The accepted wording is:
> >>
> >>   Total size of 'files' subdirectory of a package should not be
> >> larger than 32 KiB. If the package needs more auxiliary files,
> >> they should be put into SRC_URI e.g. via tarballs.
> >>
> >> (the total size being computed as a sum of apparent file sizes)
> >>
> >> The relevant policy vote is finishing at bug #633758 [1]. The CI
> >> reports [2] were updated to report packages whose 'files'
> >> directories exceed 64 KiB, to avoid adding many new warnings at
> >> once. The limit will be lowered down to 32 KiB as packages are
> >> fixed to comply with the new policy.
> >>
> >> At the same time, I would like to explicitly remind developers that
> >> the spirit of the policy is 'do not let "files" grow large', not
> >> 'make sure you're one byte less than 32769.' Do not argue that
> >> your package exceeds the limit only by few bytes -- even if it
> >> gets close to the limit, then it means it's way too large.  
> >
> > I just want to voice my opinion on this: as a developer, this policy
> > is a royal pain in the ass.
> >
> > I would ask the council to please increase this limit to at least
> > 100 KiB, preferably more.  
>
> Please substitute "QA team" for "council" in the above message.
> Thanks.
>

I second this request for the exact same reason.

Lars

Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

Alexander Berntsen-2
On 17/12/17 20:35, Lars Wendler wrote:
>> On Sun, Dec 17, 2017 at 1:39 PM, Mike Gilbert <[hidden email]>
>> wrote:
>>> I just want to voice my opinion on this: as a developer, this
>>> policy is a royal pain in the ass.
>>>
>>> I would ask the council to please increase this limit to at
>>> least 100 KiB, preferably more.
> I second this request for the exact same reason.
Is there any impact survey of whether the proposed rationale is
something a substantial amount of people genuinely care a lot about? To
me, it doesn't sound like something that is worth sacrificing our
developers' asses for; but I have no such data.
--
Alexander
[hidden email]
https://secure.plaimi.net/~alexander


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

James Le Cuirot
In reply to this post by Andreas K. Huettel
On Sun, 17 Dec 2017 20:23:21 +0100
"Andreas K. Huettel" <[hidden email]> wrote:

> Am Sonntag, 17. Dezember 2017, 14:21:10 CET schrieb Michał Górny:
> > Hello, everyone.
> >
> > It's my pleasure to announce that with a majority vote the QA team has
> > accepted a new policy. The accepted wording is:
> >
> >   Total size of 'files' subdirectory of a package should not be larger
> >   than 32 KiB. If the package needs more auxiliary files, they should
> >   be put into SRC_URI e.g. via tarballs.
> >
> > (the total size being computed as a sum of apparent file sizes)
> >  
>
> This is hard (but not impossible I guess) for toolchain stuff, even though we
> have already most patches in external tarballs, since we want to keep some old
> versions of packages available.
>
> huettel@pinacolada ~/Gentoo/gentoo/sys-libs/glibc/files $ du -sh .
> 152K    .
Is this the right measure? Actual disk usage will vary depending on
block size. For me, this is only 94K. "du -bsh" gives the true byte
total as 79K.

--
James Le Cuirot (chewi)
Gentoo Linux Developer

attachment0 (1000 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

Ulrich Mueller-2
In reply to this post by Mike Gilbert-2
>>>>> On Sun, 17 Dec 2017, Mike Gilbert wrote:

> I just want to voice my opinion on this: as a developer, this policy
> is a royal pain in the ass.

> I would ask the council to please increase this limit to at least
> 100 KiB, preferably more.

It is a policy, not an absolute rule.

Ulrich

attachment0 (501 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

Mike Gilbert-2
On Sun, Dec 17, 2017 at 5:18 PM, Ulrich Mueller <[hidden email]> wrote:
>>>>>> On Sun, 17 Dec 2017, Mike Gilbert wrote:
>
>> I just want to voice my opinion on this: as a developer, this policy
>> is a royal pain in the ass.
>
>> I would ask the council to please increase this limit to at least
>> 100 KiB, preferably more.
>
> It is a policy, not an absolute rule.

What's the difference? Is QA not going to enforce the policy?

Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

Ulrich Mueller-2
>>>>> On Sun, 17 Dec 2017, Mike Gilbert wrote:

>> It is a policy, not an absolute rule.

> What's the difference? Is QA not going to enforce the policy?

It is described here:
https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Policies#General_Notes

The QA team can grant an exception if for a specific package there are
good reasons not to follow the policy.

Ulrich

attachment0 (501 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

Michał Górny-5
In reply to this post by Mike Gilbert-2
W dniu nie, 17.12.2017 o godzinie 13∶39 -0500, użytkownik Mike Gilbert
napisał:

> On Sun, Dec 17, 2017 at 8:21 AM, Michał Górny <[hidden email]> wrote:
> > Hello, everyone.
> >
> > It's my pleasure to announce that with a majority vote the QA team has
> > accepted a new policy. The accepted wording is:
> >
> >   Total size of 'files' subdirectory of a package should not be larger
> >   than 32 KiB. If the package needs more auxiliary files, they should
> >   be put into SRC_URI e.g. via tarballs.
> >
> > (the total size being computed as a sum of apparent file sizes)
> >
> > The relevant policy vote is finishing at bug #633758 [1]. The CI reports
> >  [2] were updated to report packages whose 'files' directories exceed
> > 64 KiB, to avoid adding many new warnings at once. The limit will
> > be lowered down to 32 KiB as packages are fixed to comply with the new
> > policy.
> >
> > At the same time, I would like to explicitly remind developers that
> > the spirit of the policy is 'do not let "files" grow large', not 'make
> > sure you're one byte less than 32769.' Do not argue that your package
> > exceeds the limit only by few bytes -- even if it gets close to the
> > limit, then it means it's way too large.
>
> I just want to voice my opinion on this: as a developer, this policy
> is a royal pain in the ass.

Given that at the moment of the vote there was around 70 packages not
meeting the limit, I dare believe it isn't such a big issue for any
single developer.

> I would ask the council to please increase this limit to at least 100
> KiB, preferably more.

Let's give it a few days first, ok? Let's see how it works in practice
and get more feedback. It'd be kinda silly to abolish it at the moment
of introducing it.

That said, without the 32 KiB directory size limit, the 20 KiB file size
limit makes even less sense, so we should probably drop it.

--
Best regards,
Michał Górny


Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

Francesco Riosa-3
In reply to this post by Michał Górny-5



On 12/17/17 14:21, Michał Górny wrote:
...

Rationale
=========

At this moment, syncing the repository implies fetching 'files'
directories of all packages, even though the relevant files are used
only when a ebuild referencing them is being built. This means that our
users fetch many files that they will never use -- either because they
don't need the package in question, or because the file belongs
to an old version.

For example, 'du -h app-shells/bash/files' states 232K while only three
of those files are used by the newest version, and everything else are
patches for old versions. And in case of bash, we're keeping those
versions pretty much 'forever'.

The new policy mostly targets large patchsets and files relevant to old
package versions. By removing them from the repository, we're hoping to
reduce the growth of its size a bit and reduce the amount of data
transferred via rsync.

Evaluating transfer size, since on-disk size is different and the latter will vary

The numbers are interesting:
- Total size of the tree: 224509 KiB #1
- Total size of files in files/: 27809 KiB #2
- Cumulative files/ >= 32KiB : 3289 KiB #2

Some simple math later and we discover that removing _all_ files from the offending packages would give only a 1,5% reduction in transfer size.
Removing _all_ files/ directory would spare 12,4% or 1/8

I don't have numbers for the past, but if I recall correctly currently the situation is greener than 10 years ago.
This to point that _some_ policy is _beneficial_ to avoid an explosion of the repo size.
However restricting it further IMO would give very little benefit and (looking at the packages involved) make life harder for no good reason.

It would be interesting instead to evaluate ways to remove _all_ files/ dirs from the tree, keeping ebuilds separated from data.
a different tree for files/ can seen a cleaner approach, give all ebuilds the same mechanism to personalize patches & co, remove limits in size (well not all limits)
Obviously the cost of such an operation is order of magnitude higher than putting some policies in place.

#1 obtained with: find * -type f -exec cat {} + | wc -c

#2 list obtained with:
cd $PORTDIR
for files in $(find * -type d -name files) ; do
    echo -n $(find ${files} -type f -exec cat {} + | wc -c)
    echo ",${files%/files}"
done

Best Regards,
Francesco
Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

Rich Freeman
On Mon, Dec 18, 2017 at 7:45 AM, Francesco Riosa <[hidden email]> wrote:
>
> It would be interesting instead to evaluate ways to remove _all_ files/ dirs
> from the tree, keeping ebuilds separated from data.

Arguably you could go a step further and not distribute even the
ebuilds except on demand.  Just have an index of what packages exist
and enough dependency info to avoid having to churn with ebuild
fetches in order to resolve them.

You could view ebuilds as source code - certainly important, but not
necessarily the best thing to just have sitting around on your hard
drive when not needed.

Whether we remove all files/ or the entire package dir from the repo,
I'd suggest that this become more standardized if we wanted to go down
one of these roads.  Instead of sticking something in SRC_URI and so
on, it might be best if files (or packages) be kept in a standard
mirrored location, and the package manager would just automatically
find/fetch them if they exist and extract them to a standard location.
Then any package that uses files/ can do so in a more standardized
way.

This also would fix some existing issues with non-upstream distfiles,
where they get stored in various random locations and aren't
necessarily available long-term.  This has been a topic that comes up
from time to time, especially if somebody is trying to build from an
old version of the repo.  Something like genkernel patches or whatever
would just go in files/ since the size limit is now gone and they'd
presumably be archived forever.

--
Rich

Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

Francesco Riosa-3


On 12/18/17 14:01, Rich Freeman wrote:
> On Mon, Dec 18, 2017 at 7:45 AM, Francesco Riosa <[hidden email]> wrote:
>> It would be interesting instead to evaluate ways to remove _all_ files/ dirs
>> from the tree, keeping ebuilds separated from data.
> Arguably you could go a step further and not distribute even the
> ebuilds except on demand.  Just have an index of what packages exist
> and enough dependency info to avoid having to churn with ebuild
> fetches in order to resolve them.
interesting point of view but this would require major refactoring of
portage and providing a resilient and capable infrastructure to serve
the ebuilds on demand.
Removing just files/ dirs  would require no modification to portage and
a load on infra that is probably much lighter (dependant on
implementation chosen)

> You could view ebuilds as source code - certainly important, but not
> necessarily the best thing to just have sitting around on your hard
> drive when not needed.
Personally, most of the time I do see  them exactly this way, but not a
big fuss either
>
> Whether we remove all files/ or the entire package dir from the repo,
> I'd suggest that this become more standardized if we wanted to go down
> one of these roads.  Instead of sticking something in SRC_URI and so
> on, it might be best if files (or packages) be kept in a standard
> mirrored location, and the package manager would just automatically
> find/fetch them if they exist and extract them to a standard location.
> Then any package that uses files/ can do so in a more standardized
> way.
Provided exact source of upstream files is kept near the ebuild the idea
is tantalizing.
A per repository base SRC_URI and "automatic" downloads of packages files

>
> This also would fix some existing issues with non-upstream distfiles,
> where they get stored in various random locations and aren't
> necessarily available long-term.  This has been a topic that comes up
> from time to time, especially if somebody is trying to build from an
> old version of the repo.  Something like genkernel patches or whatever
> would just go in files/ since the size limit is now gone and they'd
> presumably be archived forever.
>
another good point, it would be probably a good time to split $DISTDIR
in per package directory, big number of inodes in that dir has also been
a source of problems in the past, especially for mirror owners.




Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

Rich Freeman
On Mon, Dec 18, 2017 at 8:33 AM, Francesco Riosa <[hidden email]> wrote:

>
> On 12/18/17 14:01, Rich Freeman wrote:
>>
>> Whether we remove all files/ or the entire package dir from the repo,
>> I'd suggest that this become more standardized if we wanted to go down
>> one of these roads.  Instead of sticking something in SRC_URI and so
>> on, it might be best if files (or packages) be kept in a standard
>> mirrored location, and the package manager would just automatically
>> find/fetch them if they exist and extract them to a standard location.
>> Then any package that uses files/ can do so in a more standardized
>> way.
>
> Provided exact source of upstream files is kept near the ebuild the idea
> is tantalizing.

To be clear - I wasn't referring up upstream files.  I was talking
about Gentoo-created files, like patches/etc.  There should be one
standard place for these things vs having them scattered on various
webservers (including dev.g.o), with no guarantee of retention.

Doing this with upstream files would be more difficult.  For one
they'd use a ton of space, if we're committing to archive them
long-term (we already mirror them of course, but only for as long as
they're in the repo).  And of course we have packages set to nomirror
where this isn't even an option.

I guess there are some packages which are legal to distribute but
where we still host the files ourselves because upstream doesn't
maintain stable tarballs which would benefit from treating them the
same way as files/.  Those aren't very common.

--
Rich

Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

Matthias Maier-3
In reply to this post by Michał Górny-5

On Sun, Dec 17, 2017, at 07:21 CST, Michał Górny <[hidden email]> wrote:

> Hello, everyone.
>
> It's my pleasure to announce that with a majority vote the QA team has
> accepted a new policy. The accepted wording is:
>
>   Total size of 'files' subdirectory of a package should not be larger
>   than 32 KiB. If the package needs more auxiliary files, they should
>   be put into SRC_URI e.g. via tarballs.

> Rationale
> =========
>
> At this moment, syncing the repository implies fetching 'files'
> directories of all packages, even though the relevant files are used
> only when a ebuild referencing them is being built. This means that our
> users fetch many files that they will never use -- either because they
> don't need the package in question, or because the file belongs
> to an old version.
>
> For example, 'du -h app-shells/bash/files' states 232K while only three
> of those files are used by the newest version, and everything else are
> patches for old versions. And in case of bash, we're keeping those
> versions pretty much 'forever'.
>
> The new policy mostly targets large patchsets and files relevant to old
> package versions. By removing them from the repository, we're hoping to
> reduce the growth of its size a bit and reduce the amount of data
> transferred via rsync.


Assuming that only the true byte size of files are relevant and summing
up via

  du -sb */*/files | awk '{sum+=$1} END {print sum}'

Currently gives roughly 29MB of data in the files directory. (For
comparison, the complete portage tree is roughly 219MB.)

Removing all "32kb offenders" results in 26MB for all files
directories. More aggressively, removing all "20kb offenders" results in
22MB.

I hardly doubt that these (at most) 10MB of saving we're talking about
are really relevant for rsync or git (with clone-depths 1). I haven't
estimated the growth of the repository, though.

Best,
Matthias

Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

Patrick Lauer
In reply to this post by Mike Gilbert-2
On 12/17/17 19:39, Mike Gilbert wrote:

> On Sun, Dec 17, 2017 at 8:21 AM, Michał Górny <[hidden email]> wrote:
>> Hello, everyone.
>>
>> It's my pleasure to announce that with a majority vote the QA team has
>> accepted a new policy. The accepted wording is:
>>
>>   Total size of 'files' subdirectory of a package should not be larger
>>   than 32 KiB. If the package needs more auxiliary files, they should
>>   be put into SRC_URI e.g. via tarballs.
>>
>> (the total size being computed as a sum of apparent file sizes)
>>
>> The relevant policy vote is finishing at bug #633758 [1]. The CI reports
>>  [2] were updated to report packages whose 'files' directories exceed
>> 64 KiB, to avoid adding many new warnings at once. The limit will
>> be lowered down to 32 KiB as packages are fixed to comply with the new
>> policy.
>>
>> At the same time, I would like to explicitly remind developers that
>> the spirit of the policy is 'do not let "files" grow large', not 'make
>> sure you're one byte less than 32769.' Do not argue that your package
>> exceeds the limit only by few bytes -- even if it gets close to the
>> limit, then it means it's way too large.
>
> I just want to voice my opinion on this: as a developer, this policy
> is a royal pain in the ass.
>
> I would ask the council to please increase this limit to at least 100
> KiB, preferably more.
>
As a user I would like to ask everyone involved to stick to the 32kB
limit so that we (as in everyone) don't have to fetch megabytes of
patches we'll never use, just because someone was lazy.



Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

R0b0t1
On Tue, Dec 19, 2017 at 2:33 PM, Patrick Lauer <[hidden email]> wrote:

> On 12/17/17 19:39, Mike Gilbert wrote:
>> On Sun, Dec 17, 2017 at 8:21 AM, Michał Górny <[hidden email]> wrote:
>>> Hello, everyone.
>>>
>>> It's my pleasure to announce that with a majority vote the QA team has
>>> accepted a new policy. The accepted wording is:
>>>
>>>   Total size of 'files' subdirectory of a package should not be larger
>>>   than 32 KiB. If the package needs more auxiliary files, they should
>>>   be put into SRC_URI e.g. via tarballs.
>>>
>>> (the total size being computed as a sum of apparent file sizes)
>>>
>>> The relevant policy vote is finishing at bug #633758 [1]. The CI reports
>>>  [2] were updated to report packages whose 'files' directories exceed
>>> 64 KiB, to avoid adding many new warnings at once. The limit will
>>> be lowered down to 32 KiB as packages are fixed to comply with the new
>>> policy.
>>>
>>> At the same time, I would like to explicitly remind developers that
>>> the spirit of the policy is 'do not let "files" grow large', not 'make
>>> sure you're one byte less than 32769.' Do not argue that your package
>>> exceeds the limit only by few bytes -- even if it gets close to the
>>> limit, then it means it's way too large.
>>
>> I just want to voice my opinion on this: as a developer, this policy
>> is a royal pain in the ass.
>>
>> I would ask the council to please increase this limit to at least 100
>> KiB, preferably more.
>>
> As a user I would like to ask everyone involved to stick to the 32kB
> limit so that we (as in everyone) don't have to fetch megabytes of
> patches we'll never use, just because someone was lazy.
>

How easy is it to move patches to Gentoo infrastructure if the patches
are not provided by upstream? I am slightly uncomfortable with
everything being pushed to websites like GitHub by default.

Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

Aaron W. Swenson-2
In reply to this post by Michał Górny-5
On 2017-12-17 14:21, Michał Górny wrote:
>   Total size of 'files' subdirectory of a package should not be larger
>   than 32 KiB. If the package needs more auxiliary files, they should
>   be put into SRC_URI e.g. via tarballs.

I don’t have any strong opinions about this either way.

However, what alternative do we have to throwing the patches up in a
devspace?

Having previously done so with dev-db/postgresql, it was annoying having
to fix the SRC_URI because I wasn’t the one who did a slot bump.

signature.asc (386 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [QA] New policy: 'files' directory must not be larger than 32 KiB

Mike Gilbert-2
In reply to this post by Patrick Lauer
On Tue, Dec 19, 2017 at 3:33 PM, Patrick Lauer <[hidden email]> wrote:

> On 12/17/17 19:39, Mike Gilbert wrote:
>> On Sun, Dec 17, 2017 at 8:21 AM, Michał Górny <[hidden email]> wrote:
>>> Hello, everyone.
>>>
>>> It's my pleasure to announce that with a majority vote the QA team has
>>> accepted a new policy. The accepted wording is:
>>>
>>>   Total size of 'files' subdirectory of a package should not be larger
>>>   than 32 KiB. If the package needs more auxiliary files, they should
>>>   be put into SRC_URI e.g. via tarballs.
>>>
>>> (the total size being computed as a sum of apparent file sizes)
>>>
>>> The relevant policy vote is finishing at bug #633758 [1]. The CI reports
>>>  [2] were updated to report packages whose 'files' directories exceed
>>> 64 KiB, to avoid adding many new warnings at once. The limit will
>>> be lowered down to 32 KiB as packages are fixed to comply with the new
>>> policy.
>>>
>>> At the same time, I would like to explicitly remind developers that
>>> the spirit of the policy is 'do not let "files" grow large', not 'make
>>> sure you're one byte less than 32769.' Do not argue that your package
>>> exceeds the limit only by few bytes -- even if it gets close to the
>>> limit, then it means it's way too large.
>>
>> I just want to voice my opinion on this: as a developer, this policy
>> is a royal pain in the ass.
>>
>> I would ask the council to please increase this limit to at least 100
>> KiB, preferably more.
>>
> As a user I would like to ask everyone involved to stick to the 32kB
> limit so that we (as in everyone) don't have to fetch megabytes of
> patches we'll never use, just because someone was lazy.

According to Francesco's numbers, that represents a small fraction of
the space used by a repository. A typical user won't even notice it.

Call me lazy if you like, but I prefer to let the computer do slightly
more work if it saves me several minutes every time I want to push out
a patch.

12