Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

general overview of tox - documentation #867

Closed
jhagege opened this issue Jul 2, 2018 · 3 comments · Fixed by #897
Closed

general overview of tox - documentation #867

jhagege opened this issue Jul 2, 2018 · 3 comments · Fixed by #897
Assignees
@gaborbernat
Labels
area:documentation level:medium rought estimate that this might be neither easy nor hard to implement needs:work for PRs: not quite there and needs some changes pr-merged
Milestone
3.2

Comments

@jhagege
Copy link

jhagege commented Jul 2, 2018 •
edited
Loading

Hi,
I would like to understand what is the basic flow of tox (which is an amazing project by the way !):

  • I understand there is a run of setup.py in both environments (2.7, 3.6).
  • But then, regarding the tests ran with pytest (or other test suite): are they running against package in source code, or against package installed in site-packages (by setup.py install) ?

It's important in my case because my setup.py downloads a binary that I need and puts it in the site-packages directory, which is not originally in my source code package (because I need it to be cross-platform).

Thanks for help !
@obestwalter obestwalter added the type:question ❔ a question about how things work or if something is a bug or a feature label Jul 3, 2018
@obestwalter
Copy link
Member

Hi @cyberjoac, thanks for the 💐 :)

About your question: In part this depends on your configuration and in part it depends on the way the project is structured.

To get your desired outcome you need to make sure the following:

  1. do not set skipsdist=True (docs)in the global tox config
  2. Do not set usedevelope=True (docs) in your env where you want to test the package. tox will then install the package it built into the env and run commands of that env in that context.
  3. Make sure that the way you run your tests are actually against the installed package (see below for more details)

From the way how I phrased 2. you can see that you need to be aware that tox has no actual control over what the tests are run against. What tox does is create and activate the v[irtual]env, install what you asked for, cd into the dir that you asked for, pass through the env vars you asked for, etc. ... and then run an arbitrary set of commands that you have chosen. There is no way anyone can tell you what your tests are run against without having all the details of your project and your tox.ini.

I could think of a few ways to make sure that your tests run against the installed package, the way you have set up your project at the moment. The simplest would be to add a test that can only pass in a proper installation (e.g. stupidly checking the path of an imported package filefor containing "site-packages"). If that test fails in a tox run, then you know something is wrong.

There is one specific problem, when you use pytest together with tox, where I best refer you to these comprehensive articles about the problem (and the solution): Packaging a python library and Testing & Packaging.

The short answer how to make always absolutely sure to avoid that pitfall is introducing a project layout that makes it impossible to import modules from a project package when the package is not installed. The layout proposed in that article and that we also use in the tox source code is having your packages in a src folder.

@obestwalter obestwalter reopened this Jul 3, 2018
@jhagege
Copy link
Author

jhagege commented Jul 3, 2018 •
edited
Loading

Thanks much for the incredibly detailed answer, it's very helpful !!

@gaborbernat
Copy link
Member

We'll add a flow diagram to the documentation soon, and a general introduction to help new users with getting started. Will keep this issue open to track that.

tox-flow

@gaborbernat gaborbernat changed the title Understanding flow - Running tests against package in source code or against package installed at site-packages general overview of tox Jul 3, 2018
@gaborbernat gaborbernat changed the title general overview of tox general overview of tox - documentation Jul 3, 2018
@gaborbernat gaborbernat added area:documentation needs:work for PRs: not quite there and needs some changes level:medium rought estimate that this might be neither easy nor hard to implement labels Jul 3, 2018
@gaborbernat gaborbernat self-assigned this Jul 3, 2018
@gaborbernat gaborbernat removed the type:question ❔ a question about how things work or if something is a bug or a feature label Jul 3, 2018
@gaborbernat gaborbernat modified the milestones: 3.1, 3.2 Jul 3, 2018
@gaborbernat gaborbernat mentioned this issue Jul 10, 2018
Merged
@helpr helpr bot added the pr-available label Jul 10, 2018
@gaborbernat gaborbernat mentioned this issue Jul 10, 2018
Closed
gaborbernat added a commit that referenced this issue Jul 17, 2018
@gaborbernat
Add a system overview section on the landing page #867 (#897)
b6bcd27 
* Add a system overview section on the landing page #867

Includes a flow diagram of tox.

* PR review adjustment
@helpr helpr bot added pr-merged and removed pr-available labels Jul 17, 2018
guykisel referenced this issue in guykisel/inline-plz Sep 24, 2018
@pyup-bot @guykisel
Update tox to 3.4.0 (#380)
9023769 
This PR updates [tox](https://pypi.org/project/tox) from **3.1.2** to **3.4.0**.



<details>
  <summary>Changelog</summary>
  
  
   ### 3.4.0
   ```
   ------------------

Bugfixes
^^^^^^^^

- add ``--exists-action w`` to default pip flags to handle better VCS dependencies (`pip documentation on this &lt;https://pip.pypa.io/en/latest/reference/pip/exists-action-option&gt;`_) - by :user:`gaborbernat` (`503 &lt;https://github.com/tox-dev/tox/issues/503&gt;`_)
- instead of assuming the Python version from the base python name ask the interpreter to reveal the version for the ``ignore_basepython_conflict`` flag - by :user:`gaborbernat` (`908 &lt;https://github.com/tox-dev/tox/issues/908&gt;`_)
- PEP-517 packaging fails with sdist already exists, fixed via ensuring the dist folder is empty before invoking the backend and `pypa/setuptools 1481 &lt;https://github.com/pypa/setuptools/pull/1481&gt;`_ - by :user:`gaborbernat` (`1003 &lt;https://github.com/tox-dev/tox/issues/1003&gt;`_)


Features
^^^^^^^^

- add ``commands_pre`` and ``commands_post`` that run before and after running
  the ``commands`` (setup runs always, commands only if setup suceeds, teardown always - all
  run until the first failing command)  - by :user:`gaborbernat` (`167 &lt;https://github.com/tox-dev/tox/issues/167&gt;`_)
- ``pyproject.toml`` config support initially by just inline the tox.ini under ``tool.tox.legacy_tox_ini`` key; config source priority order is ``pyproject.toml``, ``tox.ini`` and then ``setup.cfg`` - by :user:`gaborbernat` (`814 &lt;https://github.com/tox-dev/tox/issues/814&gt;`_)
- use the os environment variable ``TOX_SKIP_ENV`` to filter out tox environment names from the run list (set by ``envlist``)  - by :user:`gaborbernat` (`824 &lt;https://github.com/tox-dev/tox/issues/824&gt;`_)
- always set ``PIP_USER=0`` (do not install into the user site package, but inside the virtual environment created) and ``PIP_NO_DEPS=0`` (installing without dependencies can cause broken package installations) inside tox - by :user:`gaborbernat` (`838 &lt;https://github.com/tox-dev/tox/issues/838&gt;`_)
- tox will inject some environment variables that to indicate a command is running within tox: ``TOX_WORK_DIR`` env var is set to the tox work directory,
  ``TOX_ENV_NAME`` is set to the current running tox environment name, ``TOX_ENV_DIR`` is set to the current tox environments working dir - by :user:`gaborbernat` (`847 &lt;https://github.com/tox-dev/tox/issues/847&gt;`_)
- While running tox invokes various commands (such as building the package, pip installing dependencies and so on), these were printed in case they failed as Python arrays. Changed the representation to a shell command, allowing the users to quickly replicate/debug the failure on their own - by :user:`gaborbernat` (`851 &lt;https://github.com/tox-dev/tox/issues/851&gt;`_)
- skip missing interpreters value from the config file can now be overridden via the ``--skip-missing-interpreters`` cli flag - by :user:`gaborbernat` (`903 &lt;https://github.com/tox-dev/tox/issues/903&gt;`_)
- keep additional environments config order when listing them - by :user:`gaborbernat` (`921 &lt;https://github.com/tox-dev/tox/issues/921&gt;`_)
- allow injecting config value inside the ini file dependent of the fact that we&#39;re connected to an interactive shell or not  - by :user:`gaborbernat` (`947 &lt;https://github.com/tox-dev/tox/issues/947&gt;`_)
- do not build sdist if skip install is specified for the envs to be run - by :user:`gaborbernat` (`974 &lt;https://github.com/tox-dev/tox/issues/974&gt;`_)
- when verbosity level increases above two start passing through verbosity flags to pip - by :user:`gaborbernat` (`982 &lt;https://github.com/tox-dev/tox/issues/982&gt;`_)
- when discovering the interpreter to use check if the tox host Python matches and use that if so - by :user:`gaborbernat` (`994 &lt;https://github.com/tox-dev/tox/issues/994&gt;`_)
- ``-vv`` will print out why a virtual environment is re-created whenever this operation is triggered - by :user:`gaborbernat` (`1004 &lt;https://github.com/tox-dev/tox/issues/1004&gt;`_)


Documentation
^^^^^^^^^^^^^

- clarify that ``python`` and ``pip`` refer to the virtual environments executable - by :user:`gaborbernat` (`305 &lt;https://github.com/tox-dev/tox/issues/305&gt;`_)
- add Sphinx and mkdocs example of generating documentation via tox - by :user:`gaborbernat` (`374 &lt;https://github.com/tox-dev/tox/issues/374&gt;`_)
- specify that ``setup.cfg`` tox configuration needs to be inside the ``tox:tox`` namespace - by :user:`gaborbernat` (`545 &lt;https://github.com/tox-dev/tox/issues/545&gt;`_)
   ```
   
  
  
   ### 3.3.0
   ```
   ------------------

Bugfixes
^^^^^^^^

- fix ``TOX_LIMITED_SHEBANG`` when running under python3 - by :user:`asottile` (`931 &lt;https://github.com/tox-dev/tox/issues/931&gt;`_)


Features
^^^^^^^^

- `PEP-517 &lt;https://www.python.org/dev/peps/pep-0517/&gt;`_ source distribution support (create a
  ``.package`` virtual environment to perform build operations inside) by :user:`gaborbernat` (`573 &lt;https://github.com/tox-dev/tox/issues/573&gt;`_)
- `flit &lt;https://flit.readthedocs.io&gt;`_ support via implementing ``PEP-517`` by :user:`gaborbernat` (`820 &lt;https://github.com/tox-dev/tox/issues/820&gt;`_)
- packaging now is exposed as a hook via ``tox_package(session, venv)`` - by :user:`gaborbernat` (`951 &lt;https://github.com/tox-dev/tox/issues/951&gt;`_)


Miscellaneous
^^^^^^^^^^^^^

- Updated the VSTS build YAML to use the latest jobs and pools syntax - by :user:`davidstaheli` (`955 &lt;https://github.com/tox-dev/tox/issues/955&gt;`_)
   ```
   
  
  
   ### 3.2.1
   ```
   ------------------

Bugfixes
^^^^^^^^

- ``--parallel--safe-build`` no longer cleans up its folders (``distdir``, ``distshare``, ``log``). - by :user:`gaborbernat` (`849 &lt;https://github.com/tox-dev/tox/issues/849&gt;`_)
   ```
   
  
  
   ### 3.2.0
   ```
   ------------------

Features
^^^^^^^^

- Switch pip invocations to use the module ``-m pip`` instead of direct invocation. This could help
  avoid some of the shebang limitations.  - by :user:`gaborbernat` (`935 &lt;https://github.com/tox-dev/tox/issues/935&gt;`_)
- Ability to specify package requirements for the tox run via the ``tox.ini`` (``tox`` section under key ``requires`` - PEP-508 style): can be used to specify both plugin requirements or build dependencies. - by :user:`gaborbernat` (`783 &lt;https://github.com/tox-dev/tox/issues/783&gt;`_)
- Allow to run multiple tox instances in parallel by providing the
  ``--parallel--safe-build`` flag. - by :user:`gaborbernat` (`849 &lt;https://github.com/tox-dev/tox/issues/849&gt;`_)
   ```
   
  
  
   ### 3.1.3
   ```
   ------------------

Bugfixes
^^^^^^^^

- A caching issue that caused the ``develop-inst-nodeps`` action, which
  reinstalls the package under test, to always run has been resolved. The
  ``develop-inst-noop`` action, which, as the name suggests, is a no-op, will now
  run unless there are changes to ``setup.py`` or ``setup.cfg`` files that have
  not been reflected - by stephenfin (`909 &lt;https://github.com/tox-dev/tox/issues/909&gt;`_)


Features
^^^^^^^^

- Python version testenvs are now automatically detected instead of comparing
  against a hard-coded list of supported versions.  This enables ``py38`` and
  eventually ``py39`` / ``py40`` / etc. to work without requiring an upgrade to
  ``tox``.  As such, the following public constants are now deprecated
  (and scheduled for removal in ``tox`` 4.0: ``CPYTHON_VERSION_TUPLES``,
  ``PYPY_VERSION_TUPLES``, ``OTHER_PYTHON_INTERPRETERS``, and ``DEFAULT_FACTORS`` -
  by :user:`asottile` (`914 &lt;https://github.com/tox-dev/tox/issues/914&gt;`_)


Documentation
^^^^^^^^^^^^^

- Add a system overview section on the index page that explains briefly how tox works -
  by :user:`gaborbernat`. (`867 &lt;https://github.com/tox-dev/tox/issues/867&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/tox
  - Changelog: https://pyup.io/changelogs/tox/
  - Docs: https://tox.readthedocs.org/
</details>
@tox-dev tox-dev locked and limited conversation to collaborators Jan 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@gaborbernat gaborbernat

Labels
area:documentation level:medium rought estimate that this might be neither easy nor hard to implement needs:work for PRs: not quite there and needs some changes pr-merged
Projects
None yet
Milestone
3.2
Development

Successfully merging a pull request may close this issue.

Add a system overview section on the landing page #867
3 participants
@obestwalter @gaborbernat @jhagege