Difficulty running tests without coverage #1122
Description
Is there an existing issue for this?
- I have searched the existing issues (open and closed), and could not find an existing issue
What keywords did you use to search existing issues?
coverage
Please describe why your using this option
I'm running some tests locally and want to dive deep on a particular issue. I've found I can pass -k "Metadata-Version" to pytest to select out the test I want. If, however, I pass this option through tox, the tests fail even when they pass, because the downselecting of tests fails the test coverage:
twine main @ tox -- -k 'Metadata-Version'
.pkg: _optional_hooks> python '/Users/jaraco/Library/Application Support/pipx/venvs/tox/lib/python3.12/site-packages/pyproject_api/_backend.py' True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python '/Users/jaraco/Library/Application Support/pipx/venvs/tox/lib/python3.12/site-packages/pyproject_api/_backend.py' True setuptools.build_meta
.pkg: get_requires_for_build_editable> python '/Users/jaraco/Library/Application Support/pipx/venvs/tox/lib/python3.12/site-packages/pyproject_api/_backend.py' True setuptools.build_meta
.pkg: build_sdist> python '/Users/jaraco/Library/Application Support/pipx/venvs/tox/lib/python3.12/site-packages/pyproject_api/_backend.py' True setuptools.build_meta
py: install_package> python -I -m pip install --force-reinstall --no-deps /Users/jaraco/code/pypa/twine/.tox/.tmp/package/14/twine-5.1.1.dev10+g2c4e4a1.tar.gz
py: commands[0]> python -m coverage run -m pytest -k Metadata-Version
============================================================== test session starts ===============================================================
platform darwin -- Python 3.12.4, pytest-8.2.2, pluggy-1.5.0
cachedir: .tox/py/.pytest_cache
rootdir: /Users/jaraco/code/pypa/twine
configfile: pytest.ini
plugins: socket-0.7.0
collected 202 items / 201 deselected / 1 selected
tests/test_package.py . [100%]
================================================================ warnings summary ================================================================
twine/__init__.py:40
/Users/jaraco/code/pypa/twine/twine/__init__.py:40: DeprecationWarning: Implicit None on return values is deprecated and will raise KeyErrors.
__uri__ = metadata["home-page"]
twine/__init__.py:42
/Users/jaraco/code/pypa/twine/twine/__init__.py:42: DeprecationWarning: Implicit None on return values is deprecated and will raise KeyErrors.
__author__ = metadata["author"]
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================= 1 passed, 201 deselected, 2 warnings in 0.20s ==================================================
py: commands[1]> python -m coverage html
Wrote HTML report to htmlcov/index.html
py: commands[2]> python -m coverage report --skip-covered --show-missing --fail-under 97
Name Stmts Miss Branch BrPart Cover Missing
------------------------------------------------------------------------
tests/conftest.py 34 15 12 0 63% 48-51, 56-60, 66-80, 85
tests/helpers.py 16 6 2 0 67% 41-47
tests/test_auth.py 127 95 40 0 24% 14, 18-26, 30-38, 42-44, 48-50, 54-55, 59-60, 66-89, 100, 105, 111, 117-118, 124-125, 131-142, 152-163, 172-175, 181-190, 203-212, 223-230, 237-245, 268-274
tests/test_check.py 65 43 8 0 36% 26, 29-30, 33-34, 37-38, 42-43, 58-73, 78-97, 112-131, 141-165, 177-202, 214-241, 246-272, 276-281
tests/test_cli.py 12 6 4 0 38% 22-27, 31-32
tests/test_commands.py 23 14 2 0 36% 10-19, 23-25, 29-34, 38-39, 43-51
tests/test_main.py 28 17 2 0 37% 28, 32-39, 46-66, 73-80
tests/test_package.py 175 138 36 0 25% 27-43, 47-63, 67-74, 78-85, 89-100, 104-115, 119-131, 139-149, 163-171, 176-177, 226-322, 334-337, 342-349, 355-362, 370-380, 442-447, 451-452
tests/test_register.py 55 41 24 0 23% 14, 26-41, 46-65, 70-79, 86-102, 107-136
tests/test_repository.py 109 79 36 0 26% 27, 36-41, 46-51, 56-68, 73-76, 81-84, 89, 93-98, 103-108, 113-120, 125-132, 137-150, 155-168, 173-184, 193-223, 228-263, 310-321, 326-334, 346-354
tests/test_settings.py 82 58 16 0 33% 27-28, 36-54, 59-81, 89-93, 102-107, 112-113, 119-120, 125-130, 135-137, 142-146, 152-154, 157-158, 161-166, 169-170
tests/test_upload.py 204 157 62 0 24% 34, 47, 57-59, 64-78, 86-103, 111-116, 121-148, 161-176, 181-193, 200-213, 219-232, 242-266, 277-280, 290-306, 316-337, 343-352, 364-376, 416-441, 447-458, 467-482, 566, 583, 591, 600-616, 620-649, 657-663, 675-686
tests/test_utils.py 87 52 38 1 46% 28-39, 50-58, 73-84, 94-101, 105, 121-132, 136-150, 177-181, 198, 215-219, 224-229, 235-239, 243-246, 260->exit, 269-270, 283-293, 300-304, 319-334, 343-347
tests/test_wheel.py 41 23 12 0 38% 35-36, 40, 44-45, 49-62, 67-69, 74-78, 83-88, 93-103
twine/__main__.py 22 13 2 0 38% 30-50
twine/auth.py 73 43 26 0 46% 18-19, 24-25, 29, 34-38, 48, 57, 60-71, 74-81, 84-89, 92-100, 103, 109, 117
twine/cli.py 29 15 2 0 45% 33-53, 73-81, 85, 91-123
twine/commands/__init__.py 21 14 11 0 22% 30-36, 40-54
twine/commands/check.py 76 56 22 0 20% 54-61, 64, 72-74, 81-108, 130-164, 176-194
twine/commands/register.py 25 16 4 0 31% 43-63, 72-87
twine/commands/upload.py 95 73 39 0 16% 51-61, 86-110, 133-148, 173-273, 282-298
twine/exceptions.py 28 4 6 0 82% 35-42, 69
twine/package.py 156 104 53 3 28% 60, 72-90, 96->95, 99-100, 106, 128-138, 145-203, 206-216, 221-225, 228-235, 239-252, 273-289, 292-293, 296-298, 301-302, 305-307, 310-311, 314-316, 320-324, 328
twine/repository.py 118 82 46 0 26% 48-66, 70-77, 81-87, 90, 94-101, 104-105, 108-109, 112-127, 130-179, 184-200, 207-232, 235-242, 250
twine/settings.py 82 54 18 0 40% 111-128, 135, 139-140, 144-145, 149, 154-156, 161-274, 285-288, 293-299, 304, 313-314, 322-327, 333-341
twine/utils.py 138 98 49 0 22% 57-100, 115-118, 125-133, 145-160, 172-181, 185-188, 193-200, 210-237, 268-297, 317-321, 330, 337-339, 348, 353-354
twine/wheel.py 52 25 18 0 44% 47-51, 56-57, 60-87
twine/wininst.py 38 26 16 0 26% 15-17, 21-25, 28-57
------------------------------------------------------------------------
TOTAL 2022 1367 606 4 31%
2 files skipped due to complete coverage.
Coverage failure: total of 31 is less than fail-under=97
py: exit 2 (0.14 seconds) /Users/jaraco/code/pypa/twine> python -m coverage report --skip-covered --show-missing --fail-under 97 pid=57729
py: FAIL code 2 (4.10=setup[2.45]+cmd[1.20,0.31,0.14] seconds)
evaluation failed :( (4.14 seconds)
In the skeleton approach, which uses pytest-cov to enact coverage and pytest-enabler to provide default parameters for a plugin when enabled, it becomes possible to easily disable coverage testing by disabling the plugin: tox -- -p no:cov -k 'Metadata-Version'.
Without this approach, I'm unable to use tox to run the tests without also invoking coverage (and getting the spurious failure). I have to instead run the tests manually.
I guess that's not horrible, but coming from a framework that makes running through tox easier, I thought I'd share this degraded experience.
Anything else you'd like to mention?
No response