Agenda (draft) for November meeting 2009-11-09

classic Classic list List threaded Threaded
34 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Agenda (draft) for November meeting 2009-11-09

Ulrich Mueller-2
Find below a proposed agenda for our next meeting.

I'll post it to gentoo-council and gentoo-dev-announce this Friday,
unless there are objections.

Ulrich


1. Intro (10 minutes, including late arrivals)
   1.1. Make sure that at least a couple people are logging the meeting.
   1.2. Roll call.
   1.3. Any volunteers to chair this meeting?

2. Approve summary of October meeting (5 minutes)
   (in case it will be ready then)

3. Follow-ups from previous meeting (15 minutes)
   3.1. EAPI 3 status (zmedico, ulm)
   3.2. Upgrade path for old systems (leio, solar)
        Should important system packages (e.g. Python) be reverted to
        EAPI 0, in order to provide a defined upgrade path?

4. Prefix support in main Portage tree (grobian) (20 minutes)
   Fabian Groffen proposes that package managers should define the
   EPREFIX, EROOT, and ED variables (see [1] and replies).
   4.1. Does the council generally support this idea?
   4.2. Can it still be included in EAPI 3, or even in a fast EAPI 2.1?

5. AOB / open floor (10 minutes)

[1] http://archives.gentoo.org/gentoo-dev/msg_2a62689c71f95e4de5699a330b8b5524.xml

Reply | Threaded
Open this post in threaded view
|

Re: Agenda (draft) for November meeting 2009-11-09

Ciaran McCreesh-4
On Tue, 3 Nov 2009 18:03:20 +0100
Ulrich Mueller <[hidden email]> wrote:
> Find below a proposed agenda for our next meeting.

Could you add in something like:

"Agree upon a wording for PMS for the mtime modification change
introduced to EAPI 3 last time"

please? Due to insufficient clarity in the proposal, the PMS team hasn't
been able to come up with a wording that wouldn't either require
changes to Portage (which appears to be against the Council's intent)
or that would permit behaviour currently seen as undesirable. As I
understand it, the issues are:

* What's to be done about sub-second timestamps? What about cases where
  the build filesystem supports them but the root filesystem doesn't?

* For which files must mtimes be preserved, and which can be modified?

* Is it the intent of this proposal to prevent package managers from
  automatically rewriting, say, #!/usr/bin/python to
  #!/opt/gentoo/bin/python if prefix is being used?

Or, a solution before the meeting would be fine too. It's just I don't
think this is something the PMS team is able to resolve on its own.

--
Ciaran McCreesh

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Agenda (draft) for November meeting 2009-11-09

Patrick Lauer
In reply to this post by Ulrich Mueller-2
On Tuesday 03 November 2009 18:03:20 Ulrich Mueller wrote:
> Find below a proposed agenda for our next meeting.
>
I'd like council to discuss what I consider a major bug in PMS -
see the discussion at
http://archives.gentoo.org/gentoo-pms/msg_95a13d880eb521b13d7090f30350c26a.xml

Since the PMS / QA ppl seemingly don't have the authority to decide it needs
to be deferred to council as far as I can tell.

Short version, PMS mandates bash 3.0, eclasses use 3.2 features (e.g. +=
assignment). No 3.0 ebuild is in the tree anymore, so the PMS requirement is
quite silly.
Practically the tree has grown beyond what PMS defines (and done so for about
a year now with eclasses and ebuilds "violating" PMS with noone caring).

I would appreciate a resolution to this issue, preferably one that doesn't
kill half our eclasses.

Thanks,

Patrick

Reply | Threaded
Open this post in threaded view
|

Re: Agenda (draft) for November meeting 2009-11-09

Denis Dupeyron
On Tue, Nov 3, 2009 at 11:06 AM, Ciaran McCreesh
<[hidden email]> wrote:
> Could you add in something like:
> "Agree upon a wording for PMS for the mtime modification change
> introduced to EAPI 3 last time"

On Tue, Nov 3, 2009 at 1:17 PM, Patrick Lauer <[hidden email]> wrote:
> I'd like council to discuss what I consider a major bug in PMS -
> see the discussion at
> http://archives.gentoo.org/gentoo-pms/msg_95a13d880eb521b13d7090f30350c26a.xml

Could the both of you please flesh out a proposal on how you'd expect
the council to solve these issues? It would best if, on top of telling
what should be done, you explained why it should be done this way.
Raising the questions is already interesting but proposing answers is
even better. You may have done that elsewhere before but summarizing
it here would help tremendously. And by the way those who know they
will disagree with the above posters are welcome to make proposals of
their own. It would be nice if we'd all get an opportunity to discuss
it here before the council meeting.

Denis.

Reply | Threaded
Open this post in threaded view
|

Re: Agenda (draft) for November meeting 2009-11-09

Ciaran McCreesh-4
On Tue, 3 Nov 2009 15:56:39 -0700
Denis Dupeyron <[hidden email]> wrote:
> Could the both of you please flesh out a proposal on how you'd expect
> the council to solve these issues? It would best if, on top of telling
> what should be done, you explained why it should be done this way.
> Raising the questions is already interesting but proposing answers is
> even better. You may have done that elsewhere before but summarizing
> it here would help tremendously.

Ok. For this:

> > "Agree upon a wording for PMS for the mtime modification change
> > introduced to EAPI 3 last time"

I honestly don't have an answer to this, nor a preferred solution. So
far as I can see, any possible solution is hit by one of:

* requiring changes to Portage.

* preventing future changes to Portage that would clearly be useful
  (for example, shebang rewriting for Prefix).

* being so vague and unspecific that it is entirely legal for package
  managers not to implement mtime preservation at all.

Obviously the Council had something else in mind when they voted the
proposal in, but the combined wisdom of the PMS team hasn't managed to
work out what that is. Thus, I'd like the Council to explain their
decision in sufficiently precise language that I can convert it to
LaTeX for PMS, and implement it in Paludis.

> > I'd like council to discuss what I consider a major bug in PMS -
> > see the discussion at
> > http://archives.gentoo.org/gentoo-pms/msg_95a13d880eb521b13d7090f30350c26a.xml 

As for this one, I was hoping Patrick would, as asked, start a
discussion on the gentoo-dev list regarding backwards compatibility and
how much developers care about having a clean upgrade path. I gather
from the previous Council meeting and from other comments that at least
some developers consider not being able to upgrade from an older Gentoo
install as being a problem.

The question, really, is whether there not being an upgrade path is
deliberate, or whether it's an accident that needs fixing. I don't
think that's a decision that should be made without a general
discussion amongst all Gentoo developers, and I certainly don't think
it's a decision that should be made arbitrarily by the PMS editors.

--
Ciaran McCreesh

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Agenda (draft) for November meeting 2009-11-09

Patrick Lauer
In reply to this post by Denis Dupeyron

> On Tue, Nov 3, 2009 at 1:17 PM, Patrick Lauer <[hidden email]> wrote:
> > I'd like council to discuss what I consider a major bug in PMS -
> > see the discussion at
> > http://archives.gentoo.org/gentoo-pms/msg_95a13d880eb521b13d7090f30350c26
> >a.xml
>
> Could the both of you please flesh out a proposal on how you'd expect
> the council to solve these issues?

Problem:
Usage of bash 3.2 features in ebuilds and eclasses where PMS forces bash 3.0
Apart from potentially breaking backwards compatibility etc. this is an
inconsistency between specification and product.

Possible solutions:
1) Forbid bash 3.2 features.
Impact: Cleanup of many eclasses, lots of work for maintainers, removes
actively used and useful functionality. Makes many people unhappy.

2) Fix PMS to require bash 3.2
Impact: one-line patch to PMS, small reduction in backwards compatibility
As bash 3.2 was added Oct 2006 and stabled May 2007 this would only affect
systems not updated for over 2 years, which is far beyond our usual support
horizon. Thus compatibility impact is negligible.

3) Ignore it
Impact: Well, it's what we've been doing for a year now. Seems to work out ok,
but it's slightly unsatisfying.

4) something else?

I strongly suggest (2) since it has a very low impact, comes at no cost to
maintainers and removes the need for endless further discussions of the topic.


> It would best if, on top of telling
> what should be done, you explained why it should be done this way.
> Raising the questions is already interesting but proposing answers is
> even better. You may have done that elsewhere before but summarizing
> it here would help tremendously. And by the way those who know they
> will disagree with the above posters are welcome to make proposals of
> their own. It would be nice if we'd all get an opportunity to discuss
> it here before the council meeting.
>
> Denis.
>

Reply | Threaded
Open this post in threaded view
|

mtime preservation (was: Agenda (draft) for November meeting 2009-11-09)

Ulrich Mueller-2
In reply to this post by Ciaran McCreesh-4
>>>>> On Tue, 3 Nov 2009, Ciaran McCreesh wrote:

> Due to insufficient clarity in the proposal, the PMS team hasn't
> been able to come up with a wording that wouldn't either require
> changes to Portage (which appears to be against the Council's
> intent) or that would permit behaviour currently seen as
> undesirable. As I understand it, the issues are:

> * What's to be done about sub-second timestamps? What about cases
>   where the build filesystem supports them but the root filesystem
>   doesn't?

Obviously we cannot guarantee anything below the seconds level because
of limitations in the underlying filesystems or software (e.g., tar
for binpkgs). But is there a reason for limiting it further, i.e. not
preserving sub-second timestamps if they are supported by both
filesystems?

> * For which files must mtimes be preserved, and which can be modified?

> * Is it the intent of this proposal to prevent package managers from
>   automatically rewriting, say, #!/usr/bin/python to
>   #!/opt/gentoo/bin/python if prefix is being used?

Part of the problem (what you call "insufficient clarity") is that the
proposal's original intention was to cover only the merge process,
i.e. what takes place after pkg_preinst. Whereas you want to extend it
to include everything that is taking place after src_install (for
Portage, prepstrip and whatnot).

If you limit it to the final merge process from D to ROOT, then the
answer is easy, namely mtimes of all regular files must be preserved.

Ulrich

Reply | Threaded
Open this post in threaded view
|

Re: Agenda (draft) for November meeting 2009-11-09

Ulrich Mueller-2
In reply to this post by Ulrich Mueller-2
I can add both points suggested by Ciaran and Patrick to the agenda.
But the amount of topics is becoming a bit unrealistic for an one-hour
meeting.

Can the meeting be extended to 90 minutes, exceptionally?

Ulrich

Reply | Threaded
Open this post in threaded view
|

Re: Agenda (draft) for November meeting 2009-11-09

Denis Dupeyron
On Wed, Nov 4, 2009 at 8:19 AM, Ulrich Mueller <[hidden email]> wrote:
> I can add both points suggested by Ciaran and Patrick to the agenda.
> But the amount of topics is becoming a bit unrealistic for an one-hour
> meeting.
>
> Can the meeting be extended to 90 minutes, exceptionally?

I don't mind at all. But if we discussed it on lists before the
meeting we would mostly have to vote and that would allow us to gain a
lot of time. Should we post something to gentoo-dev-announce to tell
people about this thread?

Denis.

Reply | Threaded
Open this post in threaded view
|

Re: mtime preservation (was: Agenda (draft) for November meeting 2009-11-09)

Ciaran McCreesh-4
In reply to this post by Ulrich Mueller-2
On Wed, 4 Nov 2009 16:07:30 +0100
Ulrich Mueller <[hidden email]> wrote:
> Obviously we cannot guarantee anything below the seconds level because
> of limitations in the underlying filesystems or software (e.g., tar
> for binpkgs). But is there a reason for limiting it further, i.e. not
> preserving sub-second timestamps if they are supported by both
> filesystems?

So far as I can see, if they're fully supported on both filesystems,
Portage sometimes preserves nanosecond-resolution timestamps and
sometimes doesn't. So, requiring nanosecond-resolution timestamp
preservation where possible will need Portage changes. But not
requiring it means we'll go through this whole thing again as packages
start using more accurate timestamps for cache validation things (and
yes, this will happen -- we've already hit make-related issues on this
front).

> > * For which files must mtimes be preserved, and which can be
> > modified?
>
> > * Is it the intent of this proposal to prevent package managers from
> >   automatically rewriting, say, #!/usr/bin/python to
> >   #!/opt/gentoo/bin/python if prefix is being used?
>
> Part of the problem (what you call "insufficient clarity") is that the
> proposal's original intention was to cover only the merge process,
> i.e. what takes place after pkg_preinst. Whereas you want to extend it
> to include everything that is taking place after src_install (for
> Portage, prepstrip and whatnot).
>
> If you limit it to the final merge process from D to ROOT, then the
> answer is easy, namely mtimes of all regular files must be preserved.
What I want is for the proposal to be sufficiently specific that it
covers exactly what the package manager can and cannot do, and what
ebuilds can and cannot rely upon happening. If you require mtime
preservation between pkg_preinst and the merge to /, the package
manager can just screw things up (by implementing reasonable features)
elsewhere. It is by no means clear to me that merely requiring mtime
preservation from after pkg_preinst to before pkg_postinst, and
allowing arbitrary mtime tinkering elsewhere, is what is desired.

As an example for the above, is it legal for a package manager to
rewrite any mtime that is before the start of the build process if it
does it after src_install but before pkg_preinst?

--
Ciaran McCreesh

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: mtime preservation

Zac Medico-2
Ciaran McCreesh wrote:

> On Wed, 4 Nov 2009 16:07:30 +0100
> Ulrich Mueller <[hidden email]> wrote:
>> Obviously we cannot guarantee anything below the seconds level because
>> of limitations in the underlying filesystems or software (e.g., tar
>> for binpkgs). But is there a reason for limiting it further, i.e. not
>> preserving sub-second timestamps if they are supported by both
>> filesystems?
>
> So far as I can see, if they're fully supported on both filesystems,
> Portage sometimes preserves nanosecond-resolution timestamps and
> sometimes doesn't. So, requiring nanosecond-resolution timestamp
> preservation where possible will need Portage changes.

I think it always preserves them, as long as you have at least
python-2.5 since that is required for floating-point mtime support.
--
Thanks,
Zac

Reply | Threaded
Open this post in threaded view
|

Re: Re: mtime preservation

Ciaran McCreesh-4
On Wed, 04 Nov 2009 13:12:37 -0800
Zac Medico <[hidden email]> wrote:
> > So far as I can see, if they're fully supported on both filesystems,
> > Portage sometimes preserves nanosecond-resolution timestamps and
> > sometimes doesn't. So, requiring nanosecond-resolution timestamp
> > preservation where possible will need Portage changes.
>
> I think it always preserves them, as long as you have at least
> python-2.5 since that is required for floating-point mtime support.

Mm, I can't see the code for that. So far as I can see, for the
non-fast case you're using stat.st_mtime and os.utime, which assuming
they correspond to the POSIX things of the same name, are
second-resolution. What am I missing?

--
Ciaran McCreesh

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: mtime preservation

Zac Medico-2
Ciaran McCreesh wrote:

> On Wed, 04 Nov 2009 13:12:37 -0800
> Zac Medico <[hidden email]> wrote:
>>> So far as I can see, if they're fully supported on both filesystems,
>>> Portage sometimes preserves nanosecond-resolution timestamps and
>>> sometimes doesn't. So, requiring nanosecond-resolution timestamp
>>> preservation where possible will need Portage changes.
>> I think it always preserves them, as long as you have at least
>> python-2.5 since that is required for floating-point mtime support.
>
> Mm, I can't see the code for that. So far as I can see, for the
> non-fast case you're using stat.st_mtime and os.utime, which assuming
> they correspond to the POSIX things of the same name, are
> second-resolution. What am I missing?
Ah, I guess you're right. The documentation led me to believe that
os.utime would provide nanosecond-resolution on platforms that
support it, but a simple test case seems to indicate that it does not.
--
Thanks,
Zac

#!/usr/bin/env python

import os
import sys
import tempfile

f = tempfile.NamedTemporaryFile()
t = 0.1
os.utime(f.name, (t, t))
sys.stdout.write(str(os.stat(f.name).st_mtime) + '\n')
f.close()
Reply | Threaded
Open this post in threaded view
|

Re: Re: mtime preservation

Ulrich Mueller-2
>>>>> On Wed, 04 Nov 2009, Zac Medico wrote:

> Ah, I guess you're right. The documentation led me to believe that
> os.utime would provide nanosecond-resolution on platforms that
> support it, but a simple test case seems to indicate that it does not.

Is there an os.utimes in Python? utime(2) is marked as obsolete in
POSIX.1-2008 (says the man page).

Ulrich

Reply | Threaded
Open this post in threaded view
|

Re: Re: mtime preservation

Zac Medico-2
Ulrich Mueller wrote:
>>>>>> On Wed, 04 Nov 2009, Zac Medico wrote:
>
>> Ah, I guess you're right. The documentation led me to believe that
>> os.utime would provide nanosecond-resolution on platforms that
>> support it, but a simple test case seems to indicate that it does not.
>
> Is there an os.utimes in Python? utime(2) is marked as obsolete in
> POSIX.1-2008 (says the man page).

There is no separate os.utimes function, but it looks like the patch
on this bug might make os.utime behave as desired:

  http://bugs.python.org/issue3425
--
Thanks,
Zac

Reply | Threaded
Open this post in threaded view
|

Re: Re: mtime preservation

Ulrich Mueller-2
>>>>> On Wed, 04 Nov 2009, Zac Medico wrote:

>>> Ah, I guess you're right. The documentation led me to believe that
>>> os.utime would provide nanosecond-resolution on platforms that
>>> support it, but a simple test case seems to indicate that it does not.
>>
>> Is there an os.utimes in Python? utime(2) is marked as obsolete in
>> POSIX.1-2008 (says the man page).

> There is no separate os.utimes function, but it looks like the patch
> on this bug might make os.utime behave as desired:

>   http://bugs.python.org/issue3425

So let's declare it as a Python bug and include sub-second timestamps
(where possible with the filesystem) in the spec?

Anyway, I see this as a very minor issue.

Ulrich

Reply | Threaded
Open this post in threaded view
|

Re: mtime preservation

Ulrich Mueller-2
In reply to this post by Ciaran McCreesh-4
>>>>> On Wed, 4 Nov 2009, Ciaran McCreesh wrote:

>> Part of the problem (what you call "insufficient clarity") is that
>> the proposal's original intention was to cover only the merge
>> process, i.e. what takes place after pkg_preinst. Whereas you want
>> to extend it to include everything that is taking place after
>> src_install (for Portage, prepstrip and whatnot).
>>
>> If you limit it to the final merge process from D to ROOT, then the
>> answer is easy, namely mtimes of all regular files must be
>> preserved.

> What I want is for the proposal to be sufficiently specific that it
> covers exactly what the package manager can and cannot do, and what
> ebuilds can and cannot rely upon happening. If you require mtime
> preservation between pkg_preinst and the merge to /, the package
> manager can just screw things up (by implementing reasonable
> features) elsewhere. It is by no means clear to me that merely
> requiring mtime preservation from after pkg_preinst to before
> pkg_postinst, and allowing arbitrary mtime tinkering elsewhere, is
> what is desired.

Can you try to find a suitable wording? Otherwise, it's not clear to
me how the council could resolve the issue during the next meeting.

(And as my suggested wording [1] caused some unfortunate discussion,
I don't feel like I should come up with a new one myself.)

> As an example for the above, is it legal for a package manager to
> rewrite any mtime that is before the start of the build process if
> it does it after src_install but before pkg_preinst?

So you really want this? ;-) My personal opinion is that it wouldn't
break anything and we could therefore declare it as an allowed QA
measure. And if it takes place before pkg_preinst then the ebuild
could override it in special cases.

But please be aware that the council (October meeting) has voted
against this sort of mtime fixup.

Ulrich

[1] <http://bugs.gentoo.org/264130#c39> and following comments

Reply | Threaded
Open this post in threaded view
|

Re: Agenda (draft) for November meeting 2009-11-09

Ulrich Mueller-2
In reply to this post by Ulrich Mueller-2
>>>>> On Tue, 3 Nov 2009, I wrote:

> Find below a proposed agenda for our next meeting.

Find an updated version below. I'll post it to gentoo-council and
gentoo-dev-announce tomorrow, unless there are objections.

Ulrich


1. Intro (10 minutes, including late arrivals)
   1.1. Make sure that at least a couple people are logging the meeting.
   1.2. Roll call.
   1.3. Does everyone agree to extend today's meeting to 90 minutes?

2. Approve summary of October meeting (5 minutes)
   (in case it will be ready then)

3. Follow-ups from previous meeting (15 minutes)
   3.1. EAPI 3 status (zmedico, ulm)
   3.2. Upgrade path for old systems (leio, solar)
        Should important system packages (e.g. Python) be reverted to
        EAPI 0, in order to provide a defined upgrade path?

4. Prefix support in main Portage tree (grobian) (20 minutes)
   Fabian Groffen proposes that package managers should define the
   EPREFIX, EROOT, and ED variables (see [1] and replies).
   4.1. Does the council generally support this idea?
   4.2. Can it still be included in EAPI 3, or even in a fast EAPI 2.1?

5. Usage of bash 3.2 features in Portage tree (patrick) (15 minutes)
   Should usage of bash 3.2 features in the Portage tree be forbidden,
   allowed, or should the issue be ignored? See [2] for details.

6. Preservation of file modification times in EAPI 3 (ciaranm) (15 minutes)
   Agree upon a wording for PMS for the mtime modification change
   introduced to EAPI 3 last time.

7. AOB / open floor (10 minutes)

[1] http://archives.gentoo.org/gentoo-dev/msg_2a62689c71f95e4de5699a330b8b5524.xml
[2] http://archives.gentoo.org/gentoo-council/msg_ed3cba3e0ded55c4e497451af46ea55b.xml

Reply | Threaded
Open this post in threaded view
|

Re: Re: mtime preservation

Ciaran McCreesh-4
In reply to this post by Ulrich Mueller-2
On Thu, 5 Nov 2009 11:24:05 +0100
Ulrich Mueller <[hidden email]> wrote:

> > What I want is for the proposal to be sufficiently specific that it
> > covers exactly what the package manager can and cannot do, and what
> > ebuilds can and cannot rely upon happening. If you require mtime
> > preservation between pkg_preinst and the merge to /, the package
> > manager can just screw things up (by implementing reasonable
> > features) elsewhere. It is by no means clear to me that merely
> > requiring mtime preservation from after pkg_preinst to before
> > pkg_postinst, and allowing arbitrary mtime tinkering elsewhere, is
> > what is desired.
>
> Can you try to find a suitable wording? Otherwise, it's not clear to
> me how the council could resolve the issue during the next meeting.
I've been thinking about this, and I honestly don't think it's
achievable without losing at least one of the aims. So far as I can
see, the Council's goals are mutually contradictory on this one.

--
Ciaran McCreesh

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: mtime preservation

Denis Dupeyron
Let the clueless dev that I am summarize the situation. Please correct
me if I'm wrong, I think that will help more people than just me. In
the end I'll propose 2 wordings for PMS.

I understand that the issue with requiring mtime preservation is that
after unpacking the tarball(s) some files can have absurd dates, like
January 1st 1970 or else. One easy solution to this would be to reset
mtimes at merge time which will take care of everything. However, I
know of a package which requires mtime preservation from src_compile
to the moment it's merged to ${ROOT}. And if there's a package like
this in my very small corner of Gentoo there must be more elsewhere,
and there probably will be more in the future.

So here are the 2 wordings I propose.

1 - All mtimes from the package's sources are preserved until files
are merged to ${ROOT}. In case some mtimes are absurd and/or
unsuitable, they're considered a bug and it's the maintainer's
responsibility to fix them and report back upstream.

2 - All mtimes from the package's sources are reset to current time
between the src_unpack phase and the next one. That is before
src_configure when EAPI<2 and before src_prepare when EAPI>=2. mtimes
are then preserved until files are merged to ${ROOT}.

How's that? Am I missing something?

Denis.

12