Add further py3* environments to Tox testing #166

New Issue

decentral1se · 2019-05-13T11:56:51Z

decentral1se commented

2019-05-13 11:56:51 +00:00

I'd like to have some more guarantees on versions of Python that this tools supports. We could add a py35/36/37 and also early 38 testing environment to the Tox configuration. What do you think?

I've opened up witten/borgmatic#165 on the way towards this.

I'd like to have some more guarantees on versions of Python that this tools supports. We could add a py35/36/37 and also early 38 testing environment to the Tox configuration. What do you think? I've opened up https://projects.torsion.org/witten/borgmatic/pulls/165 on the way towards this.

decentral1se commented

2019-05-13 12:02:49 +00:00

Also opened up witten/borgmatic#167.

Also opened up https://projects.torsion.org/witten/borgmatic/pulls/167.

decentral1se commented

2019-05-13 12:05:14 +00:00

And regarding CI, I've opened up witten/borgmatic#168 too :)

And regarding CI, I've opened up https://projects.torsion.org/witten/borgmatic/issues/168 too :)

decentral1se commented

2019-05-13 21:22:17 +00:00

Already being done!

https://projects.torsion.org/witten/borgmatic/src/branch/master/.drone.yml#L15

Already being done! https://projects.torsion.org/witten/borgmatic/src/branch/master/.drone.yml#L15

decentral1se closed this issue

2019-05-13 21:22:17 +00:00

witten commented

2019-05-13 23:47:54 +00:00

Yeah, I added Python-version-specific testing to CI rather than to the tox configuration because I wanted to easily support the use case of: A dev on a random system (which may have any random supported version of Python installed) should be able to just run tox and get useful test feedback.. without having to set up Docker or install multiple versions of Python.

And then with Drone set up for CI, we still have the backstop of CI tests running against all supported versions of Python.

Yeah, I added Python-version-specific testing to CI rather than to the tox configuration because I wanted to easily support the use case of: A dev on a random system (which may have any random supported version of Python installed) should be able to just run `tox` and get useful test feedback.. without having to set up Docker or install multiple versions of Python. And then with Drone set up for CI, we still have the backstop of CI tests running against all supported versions of Python.

decentral1se commented

2019-05-14 11:15:43 +00:00

Yeah, I added Python-version-specific testing to CI rather than to the tox configuration because I wanted to easily support the use case of: A dev on a random system (which may have any random supported version of Python installed) should be able to just run tox and get useful test feedback..

That makes sense. However, it leaves the dev experience out of touch with the CI experience. This can be a pain for reproducing issues locally. One way is to integrate everything under Tox and then tell Tox to skip missing intepreters, so the dev experience is not interrupted by missing Python requirements. It is also more resistent against your CI service going bust or being bought my Microsoft or whatever - you always rely on Tox :)

Anyway, just rambling!

> Yeah, I added Python-version-specific testing to CI rather than to the tox configuration because I wanted to easily support the use case of: A dev on a random system (which may have any random supported version of Python installed) should be able to just run tox and get useful test feedback.. That makes sense. However, it leaves the dev experience out of touch with the CI experience. This can be a pain for reproducing issues locally. One way is to integrate everything under Tox and then tell Tox to skip missing intepreters, so the dev experience is not interrupted by missing Python requirements. It is also more resistent against your CI service going bust or being bought my Microsoft or whatever - you always rely on Tox :) Anyway, just rambling!

witten commented

2019-05-14 16:40:17 +00:00

Yeah, I'm a big proponent of dev-CI parity as well. Telling Tox to skip missing interpreters is a compelling idea.. I didn't know that was an option. (Looks like it was added after I started using Tox!)

So let's say we change the tox.ini to start with:

[tox]
envlist = py35,py36,py37
skip_missing_interpreters=true

Then what do we do with the Drone config in CI? For instance, right now it's this:

pipeline:
  build:
    image: python:${PYTHON_VERSION}-alpine${ALPINE_VERSION}
  ...
matrix:
  ...
  PYTHON_VERSION:
    - 3.5
    - 3.6
    - 3.7

As far as I know, there isn't a single (official) Python image that contains multiple versions of Python. So would you suggest keeping the .drone.yml exactly as-is, and just rely on the fact that in any given image, all interpreters but one will be skipped?

Yeah, I'm a big proponent of dev-CI parity as well. Telling Tox to skip missing interpreters is a compelling idea.. I didn't know that was an option. (Looks like it was added after I started using Tox!) So let's say we change the `tox.ini` to start with: ``` [tox] envlist = py35,py36,py37 skip_missing_interpreters=true ``` Then what do we do with the Drone config in CI? For instance, right now it's this: ``` pipeline: build: image: python:${PYTHON_VERSION}-alpine${ALPINE_VERSION} ... matrix: ... PYTHON_VERSION: - 3.5 - 3.6 - 3.7 ``` As far as I know, there isn't a single (official) Python image that contains multiple versions of Python. So would you suggest keeping the `.drone.yml` exactly as-is, and just rely on the fact that in any given image, all interpreters but one will be skipped?

witten reopened this issue

2019-05-14 16:40:17 +00:00

witten commented

2019-05-14 16:50:45 +00:00

A point from another discussion about simplifications made possible by this sort of multi-interpreter tox run: witten/borgmatic#173 (comment)

A point from another discussion about simplifications made possible by this sort of multi-interpreter tox run: https://projects.torsion.org/witten/borgmatic/pulls/173#issuecomment-1457

decentral1se commented

2019-05-14 17:03:28 +00:00

As far as I know, there isn’t a single (official) Python image that contains multiple versions of Python. So would you suggest keeping the .drone.yml exactly as-is, and just rely on the fact that in any given image, all interpreters but one will be skipped?

I believe the trick to use here is to somehow get what the PYTHON_VERSION environment variable is into the TOXENV environment variable. See https://tox.readthedocs.io/en/latest/config.html#conf-envlist.

> As far as I know, there isn’t a single (official) Python image that contains multiple versions of Python. So would you suggest keeping the .drone.yml exactly as-is, and just rely on the fact that in any given image, all interpreters but one will be skipped? I believe the trick to use here is to somehow get what the `PYTHON_VERSION` environment variable is into the `TOXENV` environment variable. See https://tox.readthedocs.io/en/latest/config.html#conf-envlist.

decentral1se commented

2019-05-14 17:08:28 +00:00

See https://0-8-0.docs.drone.io/environment/ for environment wrangling.

I would also recommend checking out the "Parallel Execution" configuration from Drone. Multiple Python interpreters can be tested at once instead of one after the other: https://0-8-0.docs.drone.io/pipelines/.

See https://0-8-0.docs.drone.io/environment/ for environment wrangling. I would also recommend checking out the "Parallel Execution" configuration from Drone. Multiple Python interpreters can be tested at once instead of one after the other: https://0-8-0.docs.drone.io/pipelines/.

witten commented

2019-05-14 17:15:57 +00:00

I'm not quite following. Why would you have to plumb the PYTHON_VERSION through to the TOXENV if skipping_missing_interpreters is true? For instance, let's say you have the existing .drone.yml as it is today. When running through the build matrix and running an individual PYTHON_VERSION=3.5 build within it, that'll use a Python 3.5 Docker image, and run Tox within it. Tox will skip the missing Python 3.6 and 3.7 interpreters, and just run with Python 3.5.

As for parallel execution, I'm not sure how that interacts with matrix builds. Maybe it "just works" when adding a group: to a matrix build?

I'm not quite following. Why would you have to plumb the `PYTHON_VERSION` through to the `TOXENV` if `skipping_missing_interpreters` is `true`? For instance, let's say you have the existing `.drone.yml` as it is today. When running through the build matrix and running an individual `PYTHON_VERSION=3.5` build within it, that'll use a Python 3.5 Docker image, and run Tox within it. Tox will skip the missing Python 3.6 and 3.7 interpreters, and just run with Python 3.5. As for parallel execution, I'm not sure how that interacts with matrix builds. Maybe it "just works" when adding a `group:` to a matrix build?

decentral1se commented

2019-05-14 18:51:47 +00:00

My ideas was for saving time and more clear logs (no skips). But, I don't actually know how much work Tox does to build an environment before skipping the intepreter but it might be time consuming for multiple ones. Seems fine what you're proposing!

witten referenced this issue from a commit

2019-05-14 19:09:31 +00:00

Run tests for all installed versions of Python (#166).

witten commented

2019-05-14 19:11:42 +00:00

witten/borgmatic#176

https://projects.torsion.org/witten/borgmatic/pulls/176

witten closed this issue

2019-05-14 19:25:44 +00:00

Sign in to join this conversation.