[PATCH 0/1] allow extra implementations of python

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

[PATCH 0/1] allow extra implementations of python

William Hubbs
There are situations in which downstream overlays need to have versions
of python which Gentoo no longer supports in the tree.

Currently, the only way to do this is for the overlay author to fork
python-utils-r1.eclass. This is highly undesirable since it creates a
very significant maintenance burden for the overlay author.

There are a couple of things we can do upstream to make this easier, and
I think we should do one of them.

The simplest way would be to apply the following patch.
In this situation, all the overlay author
would have to do is adjust the PYTHON_COMPAT_ALLOW_EXTRA_IMPLS variable.

The other option would be to move _PYTHON_ALL_IMPLS and  the
implementation of _python_impl_supported to a separate eclass, e.g.
python-impls-r1.eclass. This eclass could be forked freely downstream
without worrying about the other python eclasses.
I will volunteer to do the legwork for this option if we do not like the
first one.

I would advocate the first option however since no one has to fork
anything.

Thoughts?

William

William Hubbs (1):
  python.eclass: add PYTHON_COMPAT_ALLOW_EXTRA_IMPLS

 eclass/python-utils-r1.eclass | 2 ++
 1 file changed, 2 insertions(+)

--
2.24.1


Reply | Threaded
Open this post in threaded view
|

[PATCH 1/1] python.eclass: add PYTHON_COMPAT_ALLOW_EXTRA_IMPLS

William Hubbs
This variable is meant to be set in downstream overlays when they need python
implementations other than the ones we support in the tree.
It should be a space-separated list of extra implementations.

Signed-off-by: William Hubbs <[hidden email]>
---
 eclass/python-utils-r1.eclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index aacee5ac35a..4370c7a825f 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -43,6 +43,8 @@ _PYTHON_ALL_IMPLS=(
  python2_7
  python3_6 python3_7 python3_8
 )
+[[ -n ${PYTHON_COMPAT_ALLOW_EXTRA_IMPLS[*]} ]] &&
+ _PYTHON_ALL_IMPLS+=( ${PYTHON_COMPAT_ALLOW_EXTRA_IMPLS} )
 readonly _PYTHON_ALL_IMPLS
 
 # @ECLASS-VARIABLE: PYTHON_COMPAT_NO_STRICT
--
2.24.1


Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 1/1] python.eclass: add PYTHON_COMPAT_ALLOW_EXTRA_IMPLS

Rolf Eike Beer
> +[[ -n ${PYTHON_COMPAT_ALLOW_EXTRA_IMPLS[*]} ]] &&

Without string check:

[[ ${#PYTHON_COMPAT_ALLOW_EXTRA_IMPLS[@]} -gt 0 ]]

> + _PYTHON_ALL_IMPLS+=( ${PYTHON_COMPAT_ALLOW_EXTRA_IMPLS} )
>  readonly _PYTHON_ALL_IMPLS

Once array, always array:

_PYTHON_ALL_IMPLS+=( ${PYTHON_COMPAT_ALLOW_EXTRA_IMPLS[@]} )

signature.asc (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/1] allow extra implementations of python

David Seifert
In reply to this post by William Hubbs
On Thu, 2020-03-26 at 14:13 -0500, William Hubbs wrote:

> There are situations in which downstream overlays need to have versions
> of python which Gentoo no longer supports in the tree.
>
> Currently, the only way to do this is for the overlay author to fork
> python-utils-r1.eclass. This is highly undesirable since it creates a
> very significant maintenance burden for the overlay author.
>
> There are a couple of things we can do upstream to make this easier, and
> I think we should do one of them.
>
> The simplest way would be to apply the following patch.
> In this situation, all the overlay author
> would have to do is adjust the PYTHON_COMPAT_ALLOW_EXTRA_IMPLS variable.
>
> The other option would be to move _PYTHON_ALL_IMPLS and  the
> implementation of _python_impl_supported to a separate eclass, e.g.
> python-impls-r1.eclass. This eclass could be forked freely downstream
> without worrying about the other python eclasses.
> I will volunteer to do the legwork for this option if we do not like the
> first one.
>
> I would advocate the first option however since no one has to fork
> anything.
>
> Thoughts?
>
> William
>
> William Hubbs (1):
>   python.eclass: add PYTHON_COMPAT_ALLOW_EXTRA_IMPLS
>
>  eclass/python-utils-r1.eclass | 2 ++
>  1 file changed, 2 insertions(+)
>

How do you prevent some extra clever Gentoo developer from doing the following
in ::gentoo

  dev-python/foo/foo-1.ebuild:

  # don't have the time to port this right now, patches welcome
  PYTHON_COMPAT_ALLOW_EXTRA_IMPLS=( python3_4 )
  PYTHON_COMPAT=( python3_4 )
  inherit distutils-r1

Furthermore, defining PYTHON_COMPAT_ALLOW_EXTRA_IMPLS is going to break metadata
invariance, causing tons of cache mis-hits. How do you prevent that?

In addition, this will very quickly cause whatever overlay this is being used in
to become invalid. Imagine I have dev-python/foo::gentoo that supports python
3.4 and 3.5 and have dev-python/bar::sony supporting 3.4 and 3.5 sitting in a
hypothetical overlay, which depends on dev-python/foo::gentoo. Now we remove
python 3.4 from ::gentoo in python-utils-r1, and one week later we remove
python3.4 from dev-python/foo::gentoo's PYTHON_COMPAT. Now your dev-
python/bar::sony in conjunction with PYTHON_COMPAT_ALLOW_EXTRA_IMPLS has an
invalid depgraph. How do you wish to resolve this?

I feel like keeping up with ::gentoo is ultimately the better strategy than
trying to add backdoors to eclasses because some overlays are somewhat behind.

Regards
David


Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 1/1] python.eclass: add PYTHON_COMPAT_ALLOW_EXTRA_IMPLS

Mike Gilbert-2
In reply to this post by William Hubbs
On Thu, Mar 26, 2020 at 3:13 PM William Hubbs <[hidden email]> wrote:
>
> This variable is meant to be set in downstream overlays when they need python
> implementations other than the ones we support in the tree.
> It should be a space-separated list of extra implementations.

This new variable should be documented in the eclass, along with a
note that its use is unsupported for general use.

Do you intend this variable to be set in profiles or make.conf? In
either case, I don't think you can use bash arrays. Also, you're
mixing scalar and array variable syntax in your patch.

Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 1/1] python.eclass: add PYTHON_COMPAT_ALLOW_EXTRA_IMPLS

William Hubbs
On Thu, Mar 26, 2020 at 03:37:48PM -0400, Mike Gilbert wrote:

> On Thu, Mar 26, 2020 at 3:13 PM William Hubbs <[hidden email]> wrote:
> >
> > This variable is meant to be set in downstream overlays when they need python
> > implementations other than the ones we support in the tree.
> > It should be a space-separated list of extra implementations.
>
> This new variable should be documented in the eclass, along with a
> note that its use is unsupported for general use.
>
> Do you intend this variable to be set in profiles or make.conf? In
> either case, I don't think you can use bash arrays. Also, you're
> mixing scalar and array variable syntax in your patch.
Sure, I can do that. I'll fix up the syntax and add the documentation.

Thanks,

William


signature.asc (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 1/1] python.eclass: add PYTHON_COMPAT_ALLOW_EXTRA_IMPLS

Michał Górny-5
In reply to this post by William Hubbs
On Thu, 2020-03-26 at 14:13 -0500, William Hubbs wrote:

> This variable is meant to be set in downstream overlays when they need python
> implementations other than the ones we support in the tree.
> It should be a space-separated list of extra implementations.
>
> Signed-off-by: William Hubbs <[hidden email]>
> ---
>  eclass/python-utils-r1.eclass | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
> index aacee5ac35a..4370c7a825f 100644
> --- a/eclass/python-utils-r1.eclass
> +++ b/eclass/python-utils-r1.eclass
> @@ -43,6 +43,8 @@ _PYTHON_ALL_IMPLS=(
>   python2_7
>   python3_6 python3_7 python3_8
>  )
> +[[ -n ${PYTHON_COMPAT_ALLOW_EXTRA_IMPLS[*]} ]] &&
> + _PYTHON_ALL_IMPLS+=( ${PYTHON_COMPAT_ALLOW_EXTRA_IMPLS} )
>  readonly _PYTHON_ALL_IMPLS
>  
>  # @ECLASS-VARIABLE: PYTHON_COMPAT_NO_STRICT
How is this supposed to work, exactly?  If you scroll to the next
function you'd realize that after this change this 'extra
implementation' will most likely be simultaneously considered supported
and unsupported.

--
Best regards,
Michał Górny


signature.asc (631 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/1] allow extra implementations of python

Michał Górny-5
In reply to this post by William Hubbs
On Thu, 2020-03-26 at 14:13 -0500, William Hubbs wrote:
> There are situations in which downstream overlays need to have versions
> of python which Gentoo no longer supports in the tree.
>
> Currently, the only way to do this is for the overlay author to fork
> python-utils-r1.eclass. This is highly undesirable since it creates a
> very significant maintenance burden for the overlay author.

Is it preferable to create the maintenance burden on Gentoo developers
instead?  Is it fair that paid company developers shift the burden
on people who work on Gentoo in their free time, and already have their
plate full?

> The simplest way would be to apply the following patch.
> In this situation, all the overlay author
> would have to do is adjust the PYTHON_COMPAT_ALLOW_EXTRA_IMPLS variable.

...which solves exactly zero problems because the eclass still doesn't
support the implementation in question.  The best it could do is sweep
part of the problem under the carpet.

Even if it miraculously works right now at all, it will probably fail or
misbehave randomly.  Any eclass change might break it.  Then one cheap
hack will serve as an excuse to add one more, and another.

> The other option would be to move _PYTHON_ALL_IMPLS and  the
> implementation of _python_impl_supported to a separate eclass, e.g.
> python-impls-r1.eclass. This eclass could be forked freely downstream
> without worrying about the other python eclasses.

Again, this doesn't solve the problem.  It just moves the problem
elsewhere.  

> Thoughts?

I've already told you that if you want to fork, fork all eclasses.  Then
you wouldn't have to worry about internal API going out of sync.

Or don't autoupdate ::gentoo when eclasses change.

--
Best regards,
Michał Górny


signature.asc (631 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/1] allow extra implementations of python

Robin H. Johnson-2
On Thu, Mar 26, 2020 at 09:11:11PM +0100, Michał Górny wrote:
> I've already told you that if you want to fork, fork all eclasses.  Then
> you wouldn't have to worry about internal API going out of sync.
>
> Or don't autoupdate ::gentoo when eclasses change.
I also suggested something that is a corollary of this:
If you effectively freeze ::gentoo to different points for different
systems, then you ALSO need for generate frozen releases for your
overlay and backport critical changes to those releases.

--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : [hidden email]
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

signature.asc (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/1] allow extra implementations of python

William Hubbs
In reply to this post by David Seifert
On Thu, Mar 26, 2020 at 08:37:17PM +0100, David Seifert wrote:

*snip*

> How do you prevent some extra clever Gentoo developer from doing the following
> in ::gentoo
>
>   dev-python/foo/foo-1.ebuild:
>
>   # don't have the time to port this right now, patches welcome
>   PYTHON_COMPAT_ALLOW_EXTRA_IMPLS=( python3_4 )
>   PYTHON_COMPAT=( python3_4 )
>   inherit distutils-r1
 
 If there's a way inside an eclass to check that the ebuild inheriting
it is in ::gentoo, I will use it to die if the ebuild is in ::gentoo
and this variable is set.

> Furthermore, defining PYTHON_COMPAT_ALLOW_EXTRA_IMPLS is going to break metadata
> invariance, causing tons of cache mis-hits. How do you prevent that?
>
> In addition, this will very quickly cause whatever overlay this is being used in
> to become invalid. Imagine I have dev-python/foo::gentoo that supports python
> 3.4 and 3.5 and have dev-python/bar::sony supporting 3.4 and 3.5 sitting in a
> hypothetical overlay, which depends on dev-python/foo::gentoo. Now we remove
> python 3.4 from ::gentoo in python-utils-r1, and one week later we remove
> python3.4 from dev-python/foo::gentoo's PYTHON_COMPAT. Now your dev-
> python/bar::sony in conjunction with PYTHON_COMPAT_ALLOW_EXTRA_IMPLS has an
> invalid depgraph. How do you wish to resolve this?
These are both overlay maintenance questions that wouldn't affect
::gentoo, but the answer to the first one is we regenerate the metadata
very often so it wouldn't be an issue, and the second one is  resolved
by grabbing packages and putting them in another overlay until we don't
need them.

> I feel like keeping up with ::gentoo is ultimately the better strategy than
> trying to add backdoors to eclasses because some overlays are somewhat behind.
 
I agree with you. However, sometimes in the real world it doesn't work
that way, or it can take longer to move than Gentoo.

If we can do something minimal to allow downstream overlays to do what
they need to do to catch up, I think we should, especially if someone
from downstream is offering to do the work.

William


signature.asc (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/1] allow extra implementations of python

Patrick McLean-3
In reply to this post by Michał Górny-5
On Thu, 26 Mar 2020 21:11:11 +0100
Michał Górny <[hidden email]> wrote:

> On Thu, 2020-03-26 at 14:13 -0500, William Hubbs wrote:
> > There are situations in which downstream overlays need to have versions
> > of python which Gentoo no longer supports in the tree.
> >
> > Currently, the only way to do this is for the overlay author to fork
> > python-utils-r1.eclass. This is highly undesirable since it creates a
> > very significant maintenance burden for the overlay author.  
>
> Is it preferable to create the maintenance burden on Gentoo developers
> instead?  Is it fair that paid company developers shift the burden
> on people who work on Gentoo in their free time, and already have their
> plate full?

We have no intention of shifting maintenance burdens anywhere, we want
to make minor changes to make it easier for us to keep up. It's the
same as a Gentoo developer asking an upstream project to make minor
changes to their build system to support DESTDIR or a sandbox fix.

>
> > The simplest way would be to apply the following patch.
> > In this situation, all the overlay author
> > would have to do is adjust the PYTHON_COMPAT_ALLOW_EXTRA_IMPLS variable.  
>
> ...which solves exactly zero problems because the eclass still doesn't
> support the implementation in question.  The best it could do is sweep
> part of the problem under the carpet.

We don't care if it *actually* supports it, the ebuilds in question
aren't going to be installed on modern machines. We just want it to not
blow up in the global scope.

> Even if it miraculously works right now at all, it will probably fail or
> misbehave randomly.  Any eclass change might break it.  Then one cheap
> hack will serve as an excuse to add one more, and another.\
>
> > The other option would be to move _PYTHON_ALL_IMPLS and  the
> > implementation of _python_impl_supported to a separate eclass, e.g.
> > python-impls-r1.eclass. This eclass could be forked freely downstream
> > without worrying about the other python eclasses.  
>
> Again, this doesn't solve the problem.  It just moves the problem
> elsewhere.  
>

How does this not solve the problem? Overlays could trivially support
different implementations, without maintaining a lot of forks. We are
quite happy to do the work to split it out to a separate eclass.

> > Thoughts?  
>
> I've already told you that if you want to fork, fork all eclasses.  Then
> you wouldn't have to worry about internal API going out of sync.
>
> Or don't autoupdate ::gentoo when eclasses change.
>


Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/1] allow extra implementations of python

Michał Górny-5
On Thu, 2020-03-26 at 13:47 -0700, Patrick McLean wrote:

> On Thu, 26 Mar 2020 21:11:11 +0100
> Michał Górny <[hidden email]> wrote:
>
> > On Thu, 2020-03-26 at 14:13 -0500, William Hubbs wrote:
> > > There are situations in which downstream overlays need to have versions
> > > of python which Gentoo no longer supports in the tree.
> > >
> > > Currently, the only way to do this is for the overlay author to fork
> > > python-utils-r1.eclass. This is highly undesirable since it creates a
> > > very significant maintenance burden for the overlay author.  
> >
> > Is it preferable to create the maintenance burden on Gentoo developers
> > instead?  Is it fair that paid company developers shift the burden
> > on people who work on Gentoo in their free time, and already have their
> > plate full?
>
> We have no intention of shifting maintenance burdens anywhere, we want
> to make minor changes to make it easier for us to keep up. It's the
> same as a Gentoo developer asking an upstream project to make minor
> changes to their build system to support DESTDIR or a sandbox fix.
No, that's the exact opposite of it.  Supporting DESTDIR is the correct
course of action that benefits a lot of people.  Adding hacks to make
a broken thing to pretend to work is the exact opposite of that.

>
> > > The simplest way would be to apply the following patch.
> > > In this situation, all the overlay author
> > > would have to do is adjust the PYTHON_COMPAT_ALLOW_EXTRA_IMPLS variable.  
> >
> > ...which solves exactly zero problems because the eclass still doesn't
> > support the implementation in question.  The best it could do is sweep
> > part of the problem under the carpet.
>
> We don't care if it *actually* supports it, the ebuilds in question
> aren't going to be installed on modern machines. We just want it to not
> blow up in the global scope.
Which makes literally zero sense.  If you don't want them to work,
remove them.  Don't ask ::gentoo to provide special support to keep
ebuilds that are 100% broken.


--
Best regards,
Michał Górny


signature.asc (631 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/1] allow extra implementations of python

Ulrich Mueller-2
In reply to this post by William Hubbs
>>>>> On Thu, 26 Mar 2020, William Hubbs wrote:

>  If there's a way inside an eclass to check that the ebuild inheriting
> it is in ::gentoo, I will use it to die if the ebuild is in ::gentoo
> and this variable is set.

Oh please, not this again. An ebuild or eclass is supposed to work the
same everywhere, independent of the repo it is located in.

Why don't you simply copy the eclass to your overlay and do the changes
there?

Ulrich

signature.asc (517 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/1] allow extra implementations of python

Michał Górny-5
On Thu, 2020-03-26 at 22:12 +0100, Ulrich Mueller wrote:

> > > > > > On Thu, 26 Mar 2020, William Hubbs wrote:
> >  If there's a way inside an eclass to check that the ebuild inheriting
> > it is in ::gentoo, I will use it to die if the ebuild is in ::gentoo
> > and this variable is set.
>
> Oh please, not this again. An ebuild or eclass is supposed to work the
> same everywhere, independent of the repo it is located in.
>
> Why don't you simply copy the eclass to your overlay and do the changes
> there?
>
They did, and then they complained that every few months they have to
update it.  See the big slander mail from William.

--
Best regards,
Michał Górny


signature.asc (631 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/1] allow extra implementations of python

William Hubbs
On Fri, Mar 27, 2020 at 06:54:25AM +0100, Michał Górny wrote:

> On Thu, 2020-03-26 at 22:12 +0100, Ulrich Mueller wrote:
> > > > > > > On Thu, 26 Mar 2020, William Hubbs wrote:
> > >  If there's a way inside an eclass to check that the ebuild inheriting
> > > it is in ::gentoo, I will use it to die if the ebuild is in ::gentoo
> > > and this variable is set.
> >
> > Oh please, not this again. An ebuild or eclass is supposed to work the
> > same everywhere, independent of the repo it is located in.
> >
> > Why don't you simply copy the eclass to your overlay and do the changes
> > there?
> >
>
> They did, and then they complained that every few months they have to
> update it.  See the big slander mail from William.
Michal,

I already responded here on the list and apologized for the way I
presented the original issues.

So, I would appreciate it if you drop the slander accusation.

Thanks,

William


signature.asc (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/1] allow extra implementations of python

Michał Górny-5
On Fri, 2020-03-27 at 17:36 -0500, William Hubbs wrote:

> On Fri, Mar 27, 2020 at 06:54:25AM +0100, Michał Górny wrote:
> > On Thu, 2020-03-26 at 22:12 +0100, Ulrich Mueller wrote:
> > > > > > > > On Thu, 26 Mar 2020, William Hubbs wrote:
> > > >  If there's a way inside an eclass to check that the ebuild inheriting
> > > > it is in ::gentoo, I will use it to die if the ebuild is in ::gentoo
> > > > and this variable is set.
> > >
> > > Oh please, not this again. An ebuild or eclass is supposed to work the
> > > same everywhere, independent of the repo it is located in.
> > >
> > > Why don't you simply copy the eclass to your overlay and do the changes
> > > there?
> > >
> >
> > They did, and then they complained that every few months they have to
> > update it.  See the big slander mail from William.
>
> Michal,
>
> I already responded here on the list and apologized for the way I
> presented the original issues.
>
> So, I would appreciate it if you drop the slander accusation.
>
Apology accepted.  I withdraw it.

--
Best regards,
Michał Górny


signature.asc (631 bytes) Download Attachment