pyTooling/Actions
1
0
Fork 0
mirror of https://github.com/pyTooling/Actions.git synced 2026-02-12 02:56:56 +08:00
Code Issues Packages Projects Releases Wiki Activity

Reworked UnitTesting.

Browse Source
This commit is contained in:
Patrick Lehmann
2025-08-31 20:42:40 +02:00
parent bb03bdcc11
commit 729e406294
11 changed files with 741 additions and 126 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
.github/workflows/UnitTesting.yml vendored
View File

@@ -85,12 +85,12 @@ on:
default: ''
type: string
tests_directory:
description: 'Path to the directory containing tests (relative to root_directory).'
description: 'Path to the directory containing tests (relative from root_directory).'
required: false
default: 'tests'
type: string
unittest_directory:
description: 'Path to the directory containing unit tests (relative to tests_directory).'
description: 'Path to the directory containing unit tests (relative from tests_directory).'
required: false
default: 'unit'
type: string
@@ -127,7 +127,7 @@ on:
coverage_report_json_filename:
description: 'Filename how the coverage report in JSON format will be named.'
required: false
default: 'report/coverage/coverage.json'
default: 'coverage.json'
type: string
coverage_report_html_directory:
description: 'Directory where the coverage report in HTML format will be generated.'
@@ -349,7 +349,7 @@ jobs:
if: ( matrix.system == 'ubuntu' || matrix.system == 'ubuntu-arm' ) && inputs.ubuntu_before_script != ''
run: ${{ inputs.ubuntu_before_script }}
# Windows before script
# TODO: Windows before script
- name: 🪟🟦 MinGW64 before scripts
if: matrix.system == 'msys2' && matrix.runtime == 'MINGW64' && inputs.mingw64_before_script != ''
@@ -361,6 +361,7 @@ jobs:
# Run pytests
# TODO: allow configuration of pytest_args
- name: ✅ Run unit tests (Ubuntu/macOS)
id: pytest_bash
if: ( matrix.system != 'windows' && matrix.system != 'windows-arm' )

97
doc/JobTemplate/AllInOne/CompletePipeline.rst
View File

@@ -167,6 +167,28 @@ It can be used for simple Python packages as well as namespace packages.
* :pypi:`tomli`
* :ref:`pyTooling/Actions/.github/workflows/UnitTesting.yml <JOBTMPL/UnitTesting>`
* :gh:`actions/checkout`
* :gh:`msys2/setup-msys2`
* :gh:`actions/setup-python`
* :gh:`pyTooling/download-artifact`
* :gh:`actions/download-artifact`
* :gh:`pyTooling/upload-artifact`
* :gh:`actions/upload-artifact`
* apt: Packages specified via :ref:`JOBTMPL/UnitTesting/Input/apt` parameter.
* homebrew: Packages specified via :ref:`JOBTMPL/UnitTesting/Input/brew` parameter.
* MSYS2: Packages specified via :ref:`JOBTMPL/UnitTesting/Input/pacboy` parameter.
* pip
* :pypi:`wheel`
* :pypi:`tomli`
* Python packages specified via :ref:`JOBTMPL/UnitTesting/Input/requirements` or
:ref:`JOBTMPL/UnitTesting/Input/mingw_requirements` parameter.
* :ref:`pyTooling/Actions/.github/workflows/ApplicationTesting.yml <JOBTMPL/ApplicationTesting>`
* :ref:`pyTooling/Actions/.github/workflows/CheckDocumentation.yml <JOBTMPL/CheckDocumentation>`
* :ref:`pyTooling/Actions/.github/workflows/StaticTypeCheck.yml <JOBTMPL/StaticTypeCheck>`
@@ -174,7 +196,40 @@ It can be used for simple Python packages as well as namespace packages.
* :ref:`pyTooling/Actions/.github/workflows/PublishTestResults.yml <JOBTMPL/PublishTestResults>`
* :ref:`pyTooling/Actions/.github/workflows/PublishCoverageResults.yml <JOBTMPL/PublishCoverageResults>`
* :ref:`pyTooling/Actions/.github/workflows/SphinxDocumentation.yml <JOBTMPL/SphinxDocumentation>`
* :gh:`actions/checkout`
* :gh:`actions/setup-python`
* :gh:`pyTooling/download-artifact`
* :gh:`actions/download-artifact`
* :gh:`pyTooling/upload-artifact`
* :gh:`actions/upload-artifact`
* apt
* `graphviz <https://graphviz.org/>`__
* pip
* :pypi:`wheel`
* Python packages specified via :ref:`JOBTMPL/SphinxDocumentation/Input/requirements` parameter.
* :ref:`pyTooling/Actions/.github/workflows/LaTeXDocumentation.yml <JOBTMPL/LaTeXDocumentation>`
* :gh:`pyTooling/download-artifact`
* :gh:`actions/download-artifact`
* :gh:`pyTooling/upload-artifact`
* :gh:`actions/upload-artifact`
* :gh:`addnab/docker-run-action`
* :dockerhub:`pytooling/miktex <pytooling/miktex:sphinx>`
* :ref:`pyTooling/Actions/.github/workflows/PublishToGitHubPages.yml <JOBTMPL/PublishToGitHubPages>`
* :ref:`pyTooling/Actions/.github/workflows/PublishOnPyPI.yml <JOBTMPL/PublishOnPyPI>`
* :ref:`pyTooling/Actions/.github/workflows/TagReleaseCommit.yml <JOBTMPL/TagReleaseCommit>`
@@ -193,7 +248,7 @@ It can be used for simple Python packages as well as namespace packages.
Instantiation
*************
The following instantiation example creates a job ``Params`` derived from job template ``Parameters`` version ``@r5``.
The following instantiation example creates a ``Params`` job derived from job template ``Parameters`` version ``@r5``.
It only requires a `name` parameter to create the artifact names.
.. code-block:: yaml
@@ -218,45 +273,45 @@ Parameter Summary
.. rubric:: Goto :ref:`input parameters <JOBTMPL/CompletePipeline/Inputs>`
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| Parameter Name | Required | Type | Default |
+=====================================================================+==========+==========+=================================================+
+=====================================================================+==========+==========+===================================================+
| :ref:`JOBTMPL/CompletePipeline/Input/package_namespace` | no | string | ``''`` |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| :ref:`JOBTMPL/CompletePipeline/Input/package_name` | yes | string | — — — — |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| :ref:`JOBTMPL/CompletePipeline/Input/unittest_python_version` | no | string | ``'3.13'`` |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| :ref:`JOBTMPL/CompletePipeline/Input/unittest_python_version_list` | no | string | ``'3.9 3.10 3.11 3.12 3.13'`` |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| :ref:`JOBTMPL/CompletePipeline/Input/unittest_system_list` | no | string | ``'ubuntu windows macos macos-arm ucrt64'`` |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| :ref:`JOBTMPL/CompletePipeline/Input/unittest_include_list` | no | string | ``''`` |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| :ref:`JOBTMPL/CompletePipeline/Input/unittest_exclude_list` | no | string | ``'windows-arm:3.9 windows-arm:3.10'`` |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| :ref:`JOBTMPL/CompletePipeline/Input/unittest_disable_list` | no | string | ``'windows-arm:pypy-3.10 windows-arm:pypy-3.11'`` |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| :ref:`JOBTMPL/CompletePipeline/Input/apptest_python_version` | no | string | ``'3.13'`` |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| :ref:`JOBTMPL/CompletePipeline/Input/apptest_python_version_list` | no | string | ``''`` |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| :ref:`JOBTMPL/CompletePipeline/Input/apptest_system_list` | no | string | ``'ubuntu windows macos macos-arm ucrt64'`` |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| :ref:`JOBTMPL/CompletePipeline/Input/apptest_include_list` | no | string | ``''`` |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| :ref:`JOBTMPL/CompletePipeline/Input/apptest_exclude_list` | no | string | ``'windows-arm:3.9 windows-arm:3.10'`` |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| :ref:`JOBTMPL/CompletePipeline/Input/apptest_disable_list` | no | string | ``'windows-arm:pypy-3.10 windows-arm:pypy-3.11'`` |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| :ref:`JOBTMPL/CompletePipeline/Input/codecov` | no | string | ``'false'`` |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| :ref:`JOBTMPL/CompletePipeline/Input/codacy` | no | string | ``'false'`` |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| :ref:`JOBTMPL/CompletePipeline/Input/dorny` | no | string | ``'false'`` |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
| :ref:`JOBTMPL/CompletePipeline/Input/cleanup` | no | string | ``'true'`` |
+---------------------------------------------------------------------+----------+----------+-------------------------------------------------+
+---------------------------------------------------------------------+----------+----------+---------------------------------------------------+
.. rubric:: Goto :ref:`secrets <JOBTMPL/CompletePipeline/Secrets>`

2
doc/JobTemplate/Documentation/LaTeXDocumentation.rst
View File

@@ -40,7 +40,7 @@ The ``LatexDocumentation`` job template ................
Instantiation
*************
The following instantiation example creates a job `Params` derived from job template `Parameters` version `r0`. It only
The following instantiation example creates a ``Params`` job derived from job template ``Parameters`` version ``@r5``. It only
requires a `name` parameter to create the artifact names.
.. code-block:: yaml

6
doc/JobTemplate/Documentation/SphinxDocumentation.rst
View File

@@ -46,7 +46,7 @@ The ``SphinxDocumentation`` job template ..........
* pip
* :pypi:`wheel`
* Python packages specified via :ref:`JOBTMPL/SphinxDocumentation/Input/requirements`.
* Python packages specified via :ref:`JOBTMPL/SphinxDocumentation/Input/requirements` parameter.
.. _JOBTMPL/SphinxDocumentation/Instantiation:
@@ -54,7 +54,7 @@ The ``SphinxDocumentation`` job template ..........
Instantiation
*************
The following instantiation example creates a job `Params` derived from job template `Parameters` version `r0`. It only
The following instantiation example creates a ``Params`` job derived from job template ``Parameters`` version ``@r5``. It only
requires a `name` parameter to create the artifact names.
.. code-block:: yaml
@@ -177,7 +177,7 @@ requirements
:Possible Values: Any valid list of parameters for ``pip install``. |br|
Either a requirements file can be referenced using ``'-r path/to/requirements.txt'``, or a list of
packages can be specified using a space separated list like ``'Sphinx sphinx_rtd_theme sphinxcontrib-mermaid'``.
:Description: Python dependencies to be installed through pip.
:Description: Python dependencies to be installed through *pip*.
.. _JOBTMPL/SphinxDocumentation/Input/doc_directory:

4
doc/JobTemplate/Setup/ExtractConfiguration.rst
View File

@@ -42,7 +42,7 @@ The ``ExtractConfiguration`` job template is a .....
Instantiation
*************
The following instantiation example creates a job ``ConfigParams`` derived from job template ``ExtractConfiguration``
The following instantiation example creates a ``ConfigParams`` job derived from job template ``ExtractConfiguration``
version ``@r5``. It only requires a :ref:`JOBTMPL/ExtractConfiguration/Input/package_name` parameter to extract unit
test (pytest) and code coverage (Coverage.py) settings.
@@ -594,7 +594,7 @@ coverage_report_json
output = "report/coverage/coverage.json"
.. _JOBTMPL/LatexDocumentation/Optimizations:
.. _JOBTMPL/ExtractConfiguration/Optimizations:
Optimizations
*************

10
doc/JobTemplate/Setup/Parameters.rst
View File

@@ -37,7 +37,7 @@ Simple Example
.. grid-item::
:columns: 5
The following instantiation example creates a job ``Params`` derived from job template ``Parameters`` version
The following instantiation example creates a ``Params`` job derived from job template ``Parameters`` version
``@r5``. It only requires a :ref:`JOBTMPL/Parameters/Input/package_name` parameter to create the artifact names.
.. grid-item::
@@ -74,15 +74,15 @@ Complex Example
The following instantiation example creates 3 jobs from the same template, but with differing input parameters.
The first job ``UnitTestingParams`` might be used to create a job matrix of unit tests. It creates the cross of
The first ``UnitTestingParams`` job might be used to create a job matrix of unit tests. It creates the cross of
default systems (Windows, Ubuntu, macOS, macOS-ARM, MinGW64, UCRT64) and the given list of Python versions
including some mypy versions. In addition a list of excludes (marked as :deletion:`deletions`) and includes
(marked as :addition:`additions`) is handed over resulting in the following combinations.
The second job ``PerformanceTestingParams`` might be used to create a job matrix for performance tests. Here a
The second ``PerformanceTestingParams`` job might be used to create a job matrix for performance tests. Here a
pipeline might be limited to the latest two Python versions on a selected list of platforms.
The third job ``PlatformTestingParams`` might be used to create a job matrix for platform compatibility tests.
The third ``PlatformTestingParams`` job might be used to create a job matrix for platform compatibility tests.
Here a pipeline might be limited to the latest Python version, but all available platforms.
.. grid-item::
@@ -742,7 +742,7 @@ params
* ``params['doc']`` |rarr| ``artifact_names['documentation_html']``
.. _JOBTMPL/LatexDocumentation/Optimizations:
.. _JOBTMPL/Parameters/Optimizations:
Optimizations
*************

4
doc/JobTemplate/Setup/PrepareJob.rst
View File

@@ -55,7 +55,7 @@ The job template generates various output parameters derived from
Instantiation
*************
The following instantiation example creates a job ``Prepare`` derived from job template ``PrepareJob`` version ``@r5``.
The following instantiation example creates a ``Prepare`` job derived from job template ``PrepareJob`` version ``@r5``.
In a default usecase, no input parameters need to be specified for the job template assuming a main-branch and
release-branch called ``main``, a development-branch called ``dev``, as well as semantic versioning for tags and
pull-request titles.
@@ -451,7 +451,7 @@ pr_number
empty string ``''``.
.. _JOBTMPL/LatexDocumentation/Optimizations:
.. _JOBTMPL/PrepareJob/Optimizations:
Optimizations
*************

684
doc/JobTemplate/Testing/UnitTesting.rst
View File

@@ -9,26 +9,105 @@ XML format is optionally uploaded as an artifact.
Configuration options to ``pytest`` should be given via section ``[tool.pytest.ini_options]`` in a ``pyproject.toml``
file.
**Behavior:**
.. topic:: Features
1. Checkout repository
2. Setup Python and install dependencies
3. Run unit tests using ``pytest``.
4. Upload junit test summary as an artifact
* Execute unit tests using `pytest <https://docs.pytest.org/en/stable/>`__.
**Dependencies:**
* Provide unit test results as JUnit XML file (pyTest XML dialect).
* Collect code coverage using `Coverage.py <https://coverage.readthedocs.io/>`__.
* Provide code coverage results as pytest SQLite database.
* Provide code coverage results as Cobertura XML file.
* Provide code coverage results as pytest JSON file.
* Provide code coverage results as HTML report.
.. topic:: Behavior:
.. todo:: UnitTesting:Behavior needs documentation.
1. Checkout repository.
2. Setup environment and install dependencies (``apt``, ``homebrew``, ``pacman``, ...).
3. Setup Python and install dependencies (``pip``).
4. Run instructions from ``*_before_script`` parameter.
5. Run unit tests using *pytest* and if enabled in combination with *Coverage.py*.
6. Convert gathered results to other formats.
7. Upload results (test reports, code coverage reports, ...) as an artifacts.
.. topic:: Job Execution
.. image:: ../../_static/pyTooling-Actions-UnitTesting.png
:width: 600px
.. topic:: Dependencies
* :gh:`actions/checkout`
* :gh:`msys2/setup-msys2`
* :gh:`actions/setup-python`
* :gh:`pyTooling/download-artifact`
* :gh:`actions/download-artifact`
* :gh:`pyTooling/upload-artifact`
* :gh:`actions/upload-artifact`
* apt: Packages specified via :ref:`JOBTMPL/UnitTesting/Input/apt` parameter.
* homebrew: Packages specified via :ref:`JOBTMPL/UnitTesting/Input/brew` parameter.
* MSYS2: Packages specified via :ref:`JOBTMPL/UnitTesting/Input/pacboy` parameter.
* pip
* :pypi:`wheel`
* :pypi:`tomli`
* Python packages specified via :ref:`JOBTMPL/UnitTesting/Input/requirements` or
:ref:`JOBTMPL/UnitTesting/Input/mingw_requirements` parameter.
.. _JOBTMPL/UnitTesting/Instantiation:
Instantiation
*************
Simple Example
==============
The following instantiation example creates a ``UnitTesting`` job derived from job template ``UnitTesting`` version
`@r5`. For providing the job matrix as a JSON string, the :ref:`JOBTMPL/Parameters` job template is used. Additionally,
the job needs configuration settings, which are stored in :file:`pyproject.toml`. Instead of duplicating these settings,
the :ref:`JOBTMPL/ExtractConfiguration` job template is used to extract these settings.
.. code-block:: yaml
name: Pipeline
on:
push:
workflow_dispatch:
jobs:
ConfigParams:
uses: pyTooling/Actions/.github/workflows/ExtractConfiguration.yml@r5
with:
package_name: myPackage
UnitTestingParams:
uses: pyTooling/Actions/.github/workflows/Parameters.yml@r5
with:
package_name: myPackage
UnitTesting:
uses: pyTooling/Actions/.github/workflows/UnitTesting.yml@r5
needs:
- ConfigParams
- UnitTestingParams
with:
jobs: ${{ needs.UnitTestingParams.outputs.python_jobs }}
requirements: "-r tests/unit/requirements.txt"
unittest_report_xml_directory: ${{ needs.ConfigParams.outputs.unittest_report_xml_directory }}
unittest_report_xml_filename: ${{ needs.ConfigParams.outputs.unittest_report_xml_filename }}
unittest_xml_artifact: ${{ fromJson(needs.UnitTestingParams.outputs.artifact_names).unittesting_xml }}
coverage_sqlite_artifact: ${{ fromJson(needs.UnitTestingParams.outputs.artifact_names).codecoverage_sqlite }}
.. code-block:: yaml
jobs:
@@ -44,116 +123,595 @@ Simple Example
artifact: ${{ fromJson(needs.Params.outputs.artifact_names).unittesting }}
Complex Example
===============
.. seealso::
.. code-block:: yaml
:ref:`JOBTMPL/Parameters`
``Parameters`` is usually used to pre-compute the job matrix as a JSON string with all system |times| environment
|times| Python version combinations.
:ref:`JOBTMPL/PublishTestResults`
``PublishTestResults`` can be used to merge all JUnit test reports into one file.
:ref:`JOBTMPL/PublishCoverageResults`
``PublishCoverageResults`` can be used to merge all code coverage reports into one file.
TBD
Parameters
**********
.. _JOBTMPL/UnitTesting/Parameters:
Parameter Summary
*****************
.. rubric:: Goto :ref:`input parameters <JOBTMPL/UnitTesting/Inputs>`
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| Parameter Name | Required | Type | Default |
+=========================================================================+==========+==========+===================================================================+
| :ref:`JOBTMPL/UnitTesting/Input/jobs` | yes | string | — — — — |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/apt` | no | string | ``''`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/brew` | no | string | ``''`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/pacboy` | no | string | ``''`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/requirements` | no | string | ``'-r tests/requirements.txt'`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/mingw_requirements` | no | string | ``''`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/macos_before_script` | no | string | ``''`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/macos_arm_before_script` | no | string | ``''`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/ubuntu_before_script` | no | string | ``''`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/mingw64_before_script` | no | string | ``''`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/ucrt64_before_script` | no | string | ``''`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/root_directory` | no | string | ``''`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/tests_directory` | no | string | ``'tests'`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/unittest_directory` | no | string | ``'unit'`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/unittest_report_xml_directory` | no | string | ``'report/unit'`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/unittest_report_xml_filename` | no | string | ``'TestReportSummary.xml'`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/coverage_config` | no | string | ``'pyproject.toml'`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/coverage_report_xml_directory` | no | string | ``'report/coverage'`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/coverage_report_xml_filename` | no | string | ``'coverage.xml'`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/coverage_report_json_directory` | no | string | ``'report/coverage'`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/coverage_report_json_filename` | no | string | ``'coverage.json'`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/coverage_report_html_directory` | no | string | ``'report/coverage/html'`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/unittest_xml_artifact` | no | string | ``''`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/unittest_html_artifact` | no | string | ``''`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/coverage_sqlite_artifact` | no | string | ``''`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/coverage_xml_artifact` | no | string | ``''`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/coverage_json_artifact` | no | string | ``''`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
| :ref:`JOBTMPL/UnitTesting/Input/coverage_html_artifact` | no | string | ``''`` |
+-------------------------------------------------------------------------+----------+----------+-------------------------------------------------------------------+
.. rubric:: Goto :ref:`secrets <JOBTMPL/UnitTesting/Secrets>`
This job template needs no secrets.
.. rubric:: Goto :ref:`output parameters <JOBTMPL/UnitTesting/Outputs>`
This job template has no output parameters.
.. _JOBTMPL/UnitTesting/Inputs:
Input Parameters
****************
.. _JOBTMPL/UnitTesting/Input/jobs:
jobs
====
+----------------+----------+----------+--------------+
| Parameter Name | Required | Type | Default |
+================+==========+==========+==============+
| jobs | yes | string | — — — — |
+----------------+----------+----------+--------------+
:Type: string
:Required: yes
:Default Value: — — — —
:Possible Values: A JSON string with an array of dictionaries with the following key-value pairs:
JSON list with environment fields, telling the system and Python versions to run tests with.
* ``sysicon`` - icon to display
* ``system`` - name of the system
* ``runs-on`` - virtual machine image and base operating system
* ``runtime`` - name of the runtime environment if not running natively on the VM image
* ``shell`` - name of the shell
* ``pyicon`` - icon for CPython or pypy
* ``python`` - Python version
* ``envname`` - full name of the selected environment
:Description: A JSON encoded job matrix to run multiple Python job variations.
requirements
============
.. _JOBTMPL/UnitTesting/Input/apt:
+----------------+----------+----------+---------------------------------+
| Parameter Name | Required | Type | Default |
+================+==========+==========+=================================+
| requirements | optional | string | ``-r tests/requirements.txt`` |
+----------------+----------+----------+---------------------------------+
apt
===
Python dependencies to be installed through pip.
:Type: string
:Required: no
:Default Value: ``''``
:Possible Values: Any valid list of parameters for ``apt install``. |br|
Packages are specified as a space separated list like ``'graphviz curl gzip'``.
:Description: Additional Ubuntu system dependencies to be installed through *apt*.
:Example:
.. code-block:: yaml
UnitTests:
...
with:
apt: >-
graphviz
curl
gzip
.. _JOBTMPL/UnitTesting/Input/brew:
brew
====
:Type: string
:Required: no
:Default Value: ``''``
:Possible Values: Any valid list of parameters for ``brew install``. |br|
Packages are specified as a space separated list.
:Description: Additional macOS system dependencies to be installed through *brew*.
.. _JOBTMPL/UnitTesting/Input/pacboy:
pacboy
======
+----------------+----------+----------+-----------+
| Parameter Name | Required | Type | Default |
+================+==========+==========+===========+
| pacboy | optional | string | ``""`` |
+----------------+----------+----------+-----------+
:Type: string
:Required: no
:Default Value: ``''``
:Possible Values: Any valid list of parameters for ``pacboy``. |br|
Packages are specified as a space separated list like ``'python-lxml:p python-numpy:p'``.
:Description: Additional MSYS2 system dependencies to be installed through *pacboy* (*pacman*). |br|
Usually, Python packages start with ``python-``. The suffix ``:p`` ensures pacboy figures out the
correct package repository prefix for MinGW64, UCRT64, ...
Additional MSYS2 dependencies to be installed through pacboy (pacman).
.. note::
Internally, a workflow step reads the requirements file for Python and compares requested packages with a list of
packages that should be installed through pacman/pacboy compared to installation via pip. These are mainly core packages
or packages with embedded C code.
Internally, a dedicated workflow step reads the :ref:`JOBTMPL/UnitTesting/Input/requirements` file
for Python and compares requested packages with a list of packages that should be installed through
*pacman*/*pacboy* compared to installation via *pip*. These are mainly core packages or packages
with embedded C code. |br|
The list of identified packages is handed over to *pacboy* for preinstallation. Otherwise *pip*
will later raise an error. |br|
The packages listed by this parameter will be installed in addition to the identified packages.
.. attention::
Ensure your Python requirements match the available version from MSYS2 packages list, otherwise
if your :file:`requirements.txt` requests a newer version then provided by MSYS2, such a dependency
will fail.
:Example:
.. code-block:: yaml
UnitTests:
...
with:
pacboy: >-
python-lxml:p
:Packages: The following list of Python packages is identified to be installed via *pacboy*:
* :ucrt64:`python-coverage` |rarr| :pypi:`coverage`
* :ucrt64:`igraph` |rarr| :pypi:`igraph`
* :ucrt64:`python-lxml` |rarr| :pypi:`lxml`
* :ucrt64:`python-markupsafe` |rarr| :pypi:`markupsafe`
* :ucrt64:`python-numpy` |rarr| :pypi:`numpy`
* :ucrt64:`python-pip` |rarr| :pypi:`pip`
* :ucrt64:`python-pyaml` |rarr| :pypi:`pyaml`
* :ucrt64:`python-ruamel-yaml` |rarr| :pypi:`ruamel-yaml`
* :ucrt64:`python-wheel` |rarr| :pypi:`wheel`
* :ucrt64:`python-tomli` |rarr| :pypi:`tomli`
* :ucrt64:`python-types-pyyaml` |rarr| :pypi:`types.pyyaml`
.. _JOBTMPL/UnitTesting/Input/requirements:
requirements
============
:Type: string
:Required: no
:Default Value: ``'-r tests/requirements.txt'``
:Possible Values: Any valid list of parameters for ``pip install``. |br|
Either a requirements file can be referenced using ``'-r path/to/requirements.txt'``, or a list of
packages can be specified using a space separated list like ``'coverage pytest'``.
:Description: Python dependencies to be installed through *pip*.
.. _JOBTMPL/UnitTesting/Input/mingw_requirements:
mingw_requirements
==================
+--------------------+----------+----------+----------+
| Parameter Name | Required | Type | Default |
+====================+==========+==========+==========+
| mingw_requirements | optional | string | ``""`` |
+--------------------+----------+----------+----------+
:Type: string
:Required: no
:Default Value: ``''``
:Possible Values: Any valid list of parameters for ``pip install``. |br|
Either a requirements file can be referenced using ``'-r path/to/requirements.txt'``, or a list of
packages can be specified using a space separated list like ``'coverage pytest'``.
:Description: Override Python dependencies to be installed through *pip* in MSYS2 (MinGW64/UCRT64) only.
Override Python dependencies to be installed through pip on MSYS2 (MINGW64) only.
.. _JOBTMPL/UnitTesting/Input/macos_before_script:
macos_before_script
===================
:Type: string
:Required: no
:Default Value: ``''``
:Possible Values: Any valid *Bash* instructions as single-line or multi-line string suitable for macOS (Intel platform).
:Description: These optional *Bash* instructions for macOS are executed after setting up the environment and
installing the platform specific dependencies and before running the unit test.
.. _JOBTMPL/UnitTesting/Input/macos_arm_before_script:
macos_arm_before_script
=======================
:Type: string
:Required: no
:Default Value: ``''``
:Possible Values: Any valid *Bash* instructions as single-line or multi-line string suitable for macOS (ARM platform).
:Description: These optional *Bash* instructions for macOS are executed after setting up the environment and
installing the platform specific dependencies and before running the unit test.
.. _JOBTMPL/UnitTesting/Input/ubuntu_before_script:
ubuntu_before_script
====================
:Type: string
:Required: no
:Default Value: ``''``
:Possible Values: Any valid *Bash* instructions as single-line or multi-line string suitable for Ubuntu.
:Description: These optional *Bash* instructions for Ubuntu are executed after setting up the environment and
installing the platform specific dependencies and before running the unit test.
.. _JOBTMPL/UnitTesting/Input/mingw64_before_script:
mingw64_before_script
=====================
:Type: string
:Required: no
:Default Value: ``''``
:Possible Values: Any valid *Bash* instructions as single-line or multi-line string suitable for MinGW64 on Windows.
:Description: These optional *Bash* instructions for MinGW64 on Windows are executed after setting up the
environment and installing the platform specific dependencies and before running the unit test.
.. _JOBTMPL/UnitTesting/Input/ucrt64_before_script:
ucrt64_before_script
====================
:Type: string
:Required: no
:Default Value: ``''``
:Possible Values: Any valid *Bash* instructions as single-line or multi-line string suitable for UCRT64 on Windows.
:Description: These optional *Bash* instructions for UCRT64 on Windows are executed after setting up the
environment and installing the platform specific dependencies and before running the unit test.
.. hint::
The next parameters allow running different test kinds (unit tests, performance tests, platform tests, ...) with the
same job template, but isolated in sub-directories, thus pytest only discovers a subset of tests. The following code
blocks showcase how the job template uses these parameters and how it relates to a proposed directory structure.
.. grid:: 3
.. grid-item::
:columns: 5
.. card:: Relation between :ref:`JOBTMPL/UnitTesting/Input/root_directory`, :ref:`JOBTMPL/UnitTesting/Input/tests_directory` and :ref:`JOBTMPL/UnitTesting/Input/unittest_directory`
.. code-block:: bash
cd <RepositoryRoot>
cd ${root_directory}
python -m \
pytest -raP \
--color=yes ..... \
"${tests_directory}/${unittest_directory}"
.. grid-item::
:columns: 3
.. card:: Directory Structure
.. code-block::
<RepositoryRoot>/
doc/
myPackage/
__init__.py
tests/
unit/
myTests.py
.. grid-item::
:columns: 3
.. card:: Example for Default Values
.. code-block:: bash
cd <RepositoryRoot>
cd .
python -m \
pytest -raP \
--color=yes ..... \
"tests/unit"
.. _JOBTMPL/UnitTesting/Input/root_directory:
root_directory
==============
:Type: string
:Required: no
:Default Value: ``''``
:Possible Values: Any valid directory or sub-directory.
:Description: Working directory for running tests. |br|
Usually, this is the repository's root directory. Tests are called relatively from here. See
:ref:`JOBTMPL/UnitTesting/Input/tests_directory` and :ref:`JOBTMPL/UnitTesting/Input/unittest_directory`.
.. _JOBTMPL/UnitTesting/Input/tests_directory:
tests_directory
===============
+-----------------+----------+----------+-----------+
| Parameter Name | Required | Type | Default |
+=================+==========+==========+===========+
| tests_directory | optional | string | ``tests`` |
+-----------------+----------+----------+-----------+
:Type: string
:Required: no
:Default Value: ``'tests'``
:Possible Values: Any valid directory or sub-directory.
:Description: Path to the directory containing tests (relative from :ref:`JOBTMPL/UnitTesting/Input/root_directory`).
Path to the directory containing tests (test working directory).
.. _JOBTMPL/UnitTesting/Input/unittest_directory:
unittest_directory
==================
+--------------------+----------+----------+----------+
| Parameter Name | Required | Type | Default |
+====================+==========+==========+==========+
| unittest_directory | optional | string | ``unit`` |
+--------------------+----------+----------+----------+
Path to the directory containing unit tests (relative to tests_directory).
:Type: string
:Required: no
:Default Value: ``'unit'``
:Possible Values: Any valid directory or sub-directory.
:Description: Path to the directory containing unit tests (relative from :ref:`JOBTMPL/UnitTesting/Input/tests_directory`).
artifact
========
.. _JOBTMPL/UnitTesting/Input/unittest_report_xml_directory:
+----------------+----------+----------+----------+
| Parameter Name | Required | Type | Default |
+================+==========+==========+==========+
| artifact | optional | string | ``""`` |
+----------------+----------+----------+----------+
unittest_report_xml_directory
=============================
Generate unit test report with junitxml and upload results as an artifact.
:Type: string
:Required: no
:Default Value: ``'report/unit'``
:Possible Values: Any valid directory or sub-directory.
:Description: Directory or sub-directory where the unittest summary report in XML format will be saved. |br|
This path is configured in :file:`pyproject.toml` and can be extracted by :ref:`JOBTMPL/ExtractConfiguration`.
.. _JOBTMPL/UnitTesting/Input/unittest_report_xml_filename:
unittest_report_xml_filename
============================
:Type: string
:Required: no
:Default Value: ``'TestReportSummary.xml'``
:Possible Values: Any valid filename accepted by ``pytest ... --junitxml=${unittest_report_xml_filename}``.
:Description: Filename of the generated JUnit XML report. |br|
This filename is configured in :file:`pyproject.toml` and can be extracted by :ref:`JOBTMPL/ExtractConfiguration`.
.. _JOBTMPL/UnitTesting/Input/coverage_config:
coverage_config
===============
:Type: string
:Required: no
:Default Value: ``'pyproject.toml'``
:Possible Values: TBD
.. _JOBTMPL/UnitTesting/Input/coverage_report_xml_directory:
coverage_report_xml_directory
=============================
:Type: string
:Required: no
:Default Value: ``'report/coverage'``
:Possible Values: Any valid directory or sub-directory.
:Description: Directory or sub-directory where the code covergae report in XML format will be saved. |br|
This path is configured in :file:`pyproject.toml` and can be extracted by :ref:`JOBTMPL/ExtractConfiguration`.
.. _JOBTMPL/UnitTesting/Input/coverage_report_xml_filename:
coverage_report_xml_filename
============================
:Type: string
:Required: no
:Default Value: ``'coverage.xml'``
:Possible Values: Any valid XML filename.
:Description: Filename of the generated code coverage report in Cobertura format. |br|
This filename is configured in :file:`pyproject.toml` and can be extracted by :ref:`JOBTMPL/ExtractConfiguration`.
.. _JOBTMPL/UnitTesting/Input/coverage_report_json_directory:
coverage_report_json_directory
==============================
:Type: string
:Required: no
:Default Value: ``'report/coverage'``
:Possible Values: Any valid directory or sub-directory.
:Description: Directory or sub-directory where the code covergae report in JSON format will be saved. |br|
This path is configured in :file:`pyproject.toml` and can be extracted by :ref:`JOBTMPL/ExtractConfiguration`.
.. _JOBTMPL/UnitTesting/Input/coverage_report_json_filename:
coverage_report_json_filename
=============================
:Type: string
:Required: no
:Default Value: ``'coverage.json'``
:Possible Values: Any valid JSON filename.
:Description: Filename of the generated code coverage report in Coverage.py JSON format. |br|
This filename is configured in :file:`pyproject.toml` and can be extracted by :ref:`JOBTMPL/ExtractConfiguration`.
.. _JOBTMPL/UnitTesting/Input/coverage_report_html_directory:
coverage_report_html_directory
==============================
:Type: string
:Required: no
:Default Value: ``'report/coverage/html'``
:Possible Values: Any valid directory or sub-directory.
:Description: Directory or sub-directory where the code covergae report in HTML format will be saved. |br|
This path is configured in :file:`pyproject.toml` and can be extracted by :ref:`JOBTMPL/ExtractConfiguration`.
.. _JOBTMPL/UnitTesting/Input/unittest_xml_artifact:
unittest_xml_artifact
=====================
:Type: string
:Required: no
:Possible Values: Any valid artifact name.
:Description: Name of the artifact containing the unittest report summary in XML format.
.. _JOBTMPL/UnitTesting/Input/unittest_html_artifact:
unittest_html_artifact
======================
:Type: string
:Required: no
:Possible Values: Any valid artifact name.
:Description: Name of the artifact containing the unittest report in HTML format.
.. _JOBTMPL/UnitTesting/Input/coverage_sqlite_artifact:
coverage_sqlite_artifact
========================
:Type: string
:Required: no
:Possible Values: Any valid artifact name.
:Description: Name of the artifact containing the code coverage report as SQLite database.
.. _JOBTMPL/UnitTesting/Input/coverage_xml_artifact:
coverage_xml_artifact
=====================
:Type: string
:Required: no
:Possible Values: Any valid artifact name.
:Description: Name of the artifact containing the code coverage report in XML format.
.. _JOBTMPL/UnitTesting/Input/coverage_json_artifact:
coverage_json_artifact
======================
:Type: string
:Required: no
:Possible Values: Any valid artifact name.
:Description: Name of the artifact containing the code coverage report in JSON format.
.. _JOBTMPL/UnitTesting/Input/coverage_html_artifact:
coverage_html_artifact
======================
:Type: string
:Required: no
:Possible Values: Any valid artifact name.
:Description: Name of the artifact containing the code coverage report in HTML format.
.. _JOBTMPL/UnitTesting/Secrets:
Secrets
*******
This job template needs no secrets.
Results
.. _JOBTMPL/UnitTesting/Outputs:
Outputs
*******
This job template has no output parameters.
.. _JOBTMPL/UnitTesting/Optimizations:
Optimizations
*************
The following optimizations can be used to reduce the template's runtime.
Disable unit test XML generation
If parameter :ref:`JOBTMPL/UnitTesting/Input/unittest_xml_artifact` is empty, no unit test summary report will be
generated and no JUnit XML artifact will be uploaded.
Disabled code coverage collection
If parameter :ref:`JOBTMPL/UnitTesting/Input/coverage_config` is empty, no code coverage will be collected.
Disable code coverage SQLite database artifact upload
If parameter :ref:`JOBTMPL/UnitTesting/Input/coverage_sqlite_artifact` is empty, the collected code coverage database
(SQLlite format) wont be uploaded as an artifact.
Disable code coverage report conversion to the Cobertura XML format.
If parameter :ref:`JOBTMPL/UnitTesting/Input/coverage_xml_artifact` is empty, no Cobertura XML file will be generated
from code coverage report. As no Cobertura XML file exists, no code coverage XML artifact will be uploaded.
Disable code coverage report conversion to the *Coverage.py* JSON format.
If parameter :ref:`JOBTMPL/UnitTesting/Input/coverage_json_artifact` is empty, no *Coverage.py* JSON file will be
generated from code coverage report. As no JSON file exists, no code coverage JSON artifact will be uploaded.
Disable code coverage report conversion to an HTML website.
If parameter :ref:`JOBTMPL/UnitTesting/Input/coverage_html_artifact` is empty, no coverage report HTML report will be
generated from code coverage report. As no HTML report exists, no code coverage HTML artifact will be uploaded.

BIN
doc/_static/pyTooling-Actions-UnitTesting.png vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 153 KiB

1
doc/conf.py
View File

@@ -220,6 +220,7 @@ extlinks = {
"ghsrc": (f"https://GitHub.com/{githubNamespace}/{githubProject}/blob/main/%s", None),
"wiki": (f"https://en.wikipedia.org/wiki/%s", None),
"pypi": (f"https://pypi.org/project/%s/", "PyPI: %s"),
"ucrt64": (f"https://packages.msys2.org/packages/mingw-w64-ucrt-x86_64-%s", "UCRT64: %s"),
"dockerhub": (f"https://hub.docker.com/r/%s", "Docker Hub: %s"),
}

2
doc/index.rst
View File

@@ -188,9 +188,9 @@ License
JobTemplate/index
JobTemplate/AllInOne/index
JobTemplate/Setup/index
JobTemplate/Documentation/index
JobTemplate/Testing/index
JobTemplate/Quality/index
JobTemplate/Documentation/index
JobTemplate/Package/index
JobTemplate/Publish/index
JobTemplate/Release/index