Continuous Stabilization (Regarding Post Install Test Scripts)

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Continuous Stabilization (Regarding Post Install Test Scripts)

Pallav Agarwal
This is a continuation of "[gentoo-dev] Proposal for changes for the next EAPI
version" thread. Since the former subject isn't very relevant anymore.

Target:
A place to provide scripts that can be used by an automated arch-tester bot to
stabilize Gentoo packages. These scripts can be as simple as a one liner

`python -c "import module"`

for example in case of a python module. Or a little complex, for example to test
different features in case of different USE flags.
The main target is to alleviate the problems and unreliability from ARCH Testing
and to more thoroughly and automatically test packages.

1 - Some important factors to consider from the previous thread:
a) This must be run POST-MERGE so that problems after merging can be detected
b) It should be hopefully scalable in both directions - similar tests for multiple packages
 - different tests for different versions of same package.
c) If a users want, they should be able to run these tests too (If they do, bug reports
collected will be very valuable.
d) Most common opinion was to not have an EAPI change unless this gains enough
use or demand.
e) If implemented as a part of ebuild, it can have all benefits of an ebuild syntax.
Different versions of a package can have different test scripts. Test code part can
have its own conditional runtime dependencies (only to take effect IF testing is on).
Also, it can have specific parts of test run or not dependent on USE flags enabled.

2 - Some suggested solutions
a) The original proposed solution:
To have an extra phase ci_test() or something similar that would be run ONLY when
emerge is called with ci-test option. That way rest of the ebuild doesn't have to change,
and for normal users, this wouldn't run unless they explicitly want it to.

PROS:
i) Full use of ebuild syntax
ii) Users can also use.
CONS:
i) Requires EAPI change
ii) Burdens the ebuild with more stuff
iii) Existing ebuilds need to be changed (one more revision per ebuild)

b) Having an extra global USE flag "ci_test" and putting test code in pkg_postinst
phase under `if use ci_test` condition.

PROS:
i) Full use of ebuild syntax
ii) Users can enable ci-test too
iii) Changes to current ebuild standard is negligible
CONS:
i) Burdens the ebuild with more stuff
ii) Existing ebuilds need to be changed (one more revision per ebuild)

c) Creating a Gentoo Overlay with ebuilds which do test packages. And the testing
script would be part of the pkg_postinst existing function.
For example, overlay could have a package dev-python/numpy-ci-test with:
```
DEPEND="dev-python/numpy"
pkg_postinstall() {
    python -c "import numpy; numpy.test()" || die "failed"
}
```
as it's content.

PROS:
i) No change to existing ebuilds or EAPI
ii) Can almost use all benefits of ebuild syntax
iii) Anybody can submit tests for ebuilds to the corresponding ebuild on overlay
iv) Eclasses can define CI tests for multiple packages at once
CONS:
i) Users will almost never use an overlay just for testing
ii) Maintainer will have to submit one script for the portage tree and one for the overlay.
Many maintainers would prefer to skip that.
iii) Ebuild names need to be different from the ones in portage tree, so the process isn't
transparent anymore. We first need to check if -ci-test package exists or not.


Any more suggestions/comments/methods are invited.

---
Regards,
Pallav
Reply | Threaded
Open this post in threaded view
|

Re: Continuous Stabilization (Regarding Post Install Test Scripts)

Harald Alfred Weiner
Dear Pallav,

how about using a dedicated eclass for arch-testing?

E.g., you might provide a function in some eclass
that only runs when called from pkg_postinst [1]
phase AND when some configuration option is set.

Then a maintainer / arch tester might provide
the tests inside the ebuild directory, e.g.,
inside a hidden sub-directory .arch-tests or
something like that, include your eclass inside
the ebuild and call your test-function from pkg_postinst
or something like that...

[1] https://devmanual.gentoo.org/ebuild-writing/functions/pkg_postinst/index.html


Best wishes,


Harald.
Harald Weiner
Institute of Applied Geometry

Johannes-Kepler-Universität
Altenberger Straße 69
A-4040 Linz
[hidden email]
http://www.ag.jku.at
On 05/19/2016 09:45 AM, Pallav Agarwal wrote:
This is a continuation of "[gentoo-dev] Proposal for changes for the next EAPI
version" thread. Since the former subject isn't very relevant anymore.

Target:
A place to provide scripts that can be used by an automated arch-tester bot to
stabilize Gentoo packages. These scripts can be as simple as a one liner

`python -c "import module"`

for example in case of a python module. Or a little complex, for example to test
different features in case of different USE flags.
The main target is to alleviate the problems and unreliability from ARCH Testing
and to more thoroughly and automatically test packages.

1 - Some important factors to consider from the previous thread:
a) This must be run POST-MERGE so that problems after merging can be detected
b) It should be hopefully scalable in both directions - similar tests for multiple packages
 - different tests for different versions of same package.
c) If a users want, they should be able to run these tests too (If they do, bug reports
collected will be very valuable.
d) Most common opinion was to not have an EAPI change unless this gains enough
use or demand.
e) If implemented as a part of ebuild, it can have all benefits of an ebuild syntax.
Different versions of a package can have different test scripts. Test code part can
have its own conditional runtime dependencies (only to take effect IF testing is on).
Also, it can have specific parts of test run or not dependent on USE flags enabled.

2 - Some suggested solutions
a) The original proposed solution:
To have an extra phase ci_test() or something similar that would be run ONLY when
emerge is called with ci-test option. That way rest of the ebuild doesn't have to change,
and for normal users, this wouldn't run unless they explicitly want it to.

PROS:
i) Full use of ebuild syntax
ii) Users can also use.
CONS:
i) Requires EAPI change
ii) Burdens the ebuild with more stuff
iii) Existing ebuilds need to be changed (one more revision per ebuild)

b) Having an extra global USE flag "ci_test" and putting test code in pkg_postinst
phase under `if use ci_test` condition.

PROS:
i) Full use of ebuild syntax
ii) Users can enable ci-test too
iii) Changes to current ebuild standard is negligible
CONS:
i) Burdens the ebuild with more stuff
ii) Existing ebuilds need to be changed (one more revision per ebuild)

c) Creating a Gentoo Overlay with ebuilds which do test packages. And the testing
script would be part of the pkg_postinst existing function.
For example, overlay could have a package dev-python/numpy-ci-test with:
```
DEPEND="dev-python/numpy"
pkg_postinstall() {
    python -c "import numpy; numpy.test()" || die "failed"
}
```
as it's content.

PROS:
i) No change to existing ebuilds or EAPI
ii) Can almost use all benefits of ebuild syntax
iii) Anybody can submit tests for ebuilds to the corresponding ebuild on overlay
iv) Eclasses can define CI tests for multiple packages at once
CONS:
i) Users will almost never use an overlay just for testing
ii) Maintainer will have to submit one script for the portage tree and one for the overlay.
Many maintainers would prefer to skip that.
iii) Ebuild names need to be different from the ones in portage tree, so the process isn't
transparent anymore. We first need to check if -ci-test package exists or not.


Any more suggestions/comments/methods are invited.

---
Regards,
Pallav