diff --git a/doc/JobTemplate/AllInOne/CompletePipeline.rst b/doc/JobTemplate/AllInOne/CompletePipeline.rst index a27e1ba..29b537d 100644 --- a/doc/JobTemplate/AllInOne/CompletePipeline.rst +++ b/doc/JobTemplate/AllInOne/CompletePipeline.rst @@ -72,7 +72,7 @@ It can be used for simple Python packages as well as namespace packages. * Automatic tagging of merge commits on main branch to trigger a tagged pipeline (release pipeline). * Create a release page with text derived from pull request description. -.. topic:: Behavior: +.. topic:: Behavior .. todo:: CompletePipeline:Behavior needs documentation. @@ -193,7 +193,7 @@ It can be used for simple Python packages as well as namespace packages. * :ref:`pyTooling/Actions/.github/workflows/CheckDocumentation.yml ` * :ref:`pyTooling/Actions/.github/workflows/StaticTypeCheck.yml ` * :ref:`pyTooling/Actions/.github/workflows/Package.yml ` - + * :gh:`actions/checkout` * :gh:`actions/setup-python` * :gh:`pyTooling/upload-artifact` diff --git a/doc/JobTemplate/Documentation/LaTeXDocumentation.rst b/doc/JobTemplate/Documentation/LaTeXDocumentation.rst index 9f55521..4193ee2 100644 --- a/doc/JobTemplate/Documentation/LaTeXDocumentation.rst +++ b/doc/JobTemplate/Documentation/LaTeXDocumentation.rst @@ -11,7 +11,7 @@ The ``LatexDocumentation`` job template ................ * Translate a LaTeX document to PDF. -.. topic:: Behavior: +.. topic:: Behavior .. todo:: LatexDocumentation:Behavior needs documentation. diff --git a/doc/JobTemplate/Documentation/SphinxDocumentation.rst b/doc/JobTemplate/Documentation/SphinxDocumentation.rst index 466ecee..672f972 100644 --- a/doc/JobTemplate/Documentation/SphinxDocumentation.rst +++ b/doc/JobTemplate/Documentation/SphinxDocumentation.rst @@ -18,7 +18,7 @@ The ``SphinxDocumentation`` job template .......... * Optionally: download code coverage artifact (JSON format) given by :ref:`JOBTMPL/SphinxDocumentation/Input/coverage_json_artifact`. * Optionally: download unit test report artifact (XML format) given by :ref:`JOBTMPL/SphinxDocumentation/Input/unittest_xml_artifact`. -.. topic:: Behavior: +.. topic:: Behavior .. todo:: SphinxDocumentation:Behavior needs documentation. diff --git a/doc/JobTemplate/Package/Package.rst b/doc/JobTemplate/Package/Package.rst index c233490..e969655 100644 --- a/doc/JobTemplate/Package/Package.rst +++ b/doc/JobTemplate/Package/Package.rst @@ -11,7 +11,7 @@ as an artifact. * Package source code as wheel and source distribution. * Support packaging using :pypi:`build` (recommended) or :pypi:`setuptools`. -.. topic:: Behavior: +.. topic:: Behavior 1. Checkout repository. 2. Setup Python and install dependencies. diff --git a/doc/JobTemplate/Package/PublishOnPyPI.rst b/doc/JobTemplate/Package/PublishOnPyPI.rst index 3843afc..74a0f0a 100644 --- a/doc/JobTemplate/Package/PublishOnPyPI.rst +++ b/doc/JobTemplate/Package/PublishOnPyPI.rst @@ -9,14 +9,14 @@ Publish a wheel (``*.whl``) packages and/or source (``*.tar.gz``) package to `Py * Publish a Python package to `PyPI `__. -.. topic:: Behavior: +.. topic:: Behavior 1. Download package artifact 2. Publish source package(s) (``*.tar.gz``) 3. Publish wheel package(s) (``*.whl``) 4. Delete the artifact -.. topic:: Preconditions: +.. topic:: Preconditions 1. A PyPI account was created and the package name is either not occupied or the user has access rights for that package. diff --git a/doc/JobTemplate/Publish/PublishTestResults.rst b/doc/JobTemplate/Publish/PublishTestResults.rst index 3718d25..6dca46e 100644 --- a/doc/JobTemplate/Publish/PublishTestResults.rst +++ b/doc/JobTemplate/Publish/PublishTestResults.rst @@ -3,26 +3,71 @@ PublishTestResults ################## -This job downloads all artifacts and uploads jUnit XML reports as a Markdown page to GitHub Actions to visualize the -results a an item in the job list. For publishing, :gh:`dorny/test-reporter` is used. +This job downloads all matching JUnit report artifacts, merged the reports into one single report and then publishes the +JUnit test summary reports to multiple services for visualization and longterm statistical tracking. -**Behavior:** +Supported services are: -1. Checkout repository -2. Download (all) artifacts -3. Publish test results as a markdown report page to GitHub Actions. +* GitHub Actions - Job results using :gh:`dorny/test-reporter` +* `Codecov `__ -.. note:: +.. topic:: Features - The :gh:`actions/download-artifact` does not support wildcards to specify a subset of artifacts for downloading. - Thus, all artifacts need to be downloaded. + * Package source code as wheel and source distribution. + * Support packaging using :pypi:`build` (recommended) or :pypi:`setuptools`. -**Dependencies:** +.. topic:: Behavior -* :gh:`actions/checkout` -* :gh:`actions/download-artifact` -* :gh:`dorny/test-reporter` + 1. Checkout repository + 2. Download multiple artifacts containing test report summaries in JUnit XML format conforming to an artifact name + pattern (see :ref:`JOBTMPL/PublishTestResults/Input/unittest_artifacts_pattern`) for limiting the number of + downloaded artifacts and the herby generated traffic. + 3. Rename the found JUnit XML files. + 4. Merge all found JUnit XML files using :pypi:`pyEDAA.Reports` into a new JUnit XML file. |br| + Optionally, apply certain transformation and cleanup operations to the JUnit report structure. + 5. Publish test results as a markdown report page to GitHub Actions using :gh:`dorny/test-reporter`. + 6. Publish test results to `Codecov `__ using :gh:`codecov/test-results-action`. +.. topic:: Job Execution + + .. grid:: 2 + + .. grid-item:: + :columns: 4 + + .. rubric:: Job Steps + + .. image:: ../../_static/pyTooling-Actions-PublishTestResults.png + :width: 500px + + .. grid-item:: + :columns: 4 + + .. rubric:: Generated Dorny Report below Pipeline Graph + + .. image:: ../../_static/pyTooling-Actions-PublishTestResults-Dorny.png + :width: 500px + +.. topic:: Dependencies + + * :gh:`actions/checkout` + * :gh:`pyTooling/download-artifact` + + * :gh:`actions/download-artifact` + + * pip + + * :pypi:`pyEDAA.Reports` + + * :gh:`dorny/test-reporter` + * :gh:`codecov/test-results-action` + * :gh:`pyTooling/upload-artifact` + + * :gh:`actions/upload-artifact` + + + +.. _JOBTMPL/PublishTestResults/Instantiation: Instantiation ************* @@ -54,35 +99,263 @@ Complex Example - CodeCoverage - UnitTesting +.. seealso:: -Parameters -********** + :ref:`JOBTMPL/UnitTesting` + tbd + :ref:`JOBTMPL/PublishCoverageResults` + tbd -report_files + +.. _JOBTMPL/PublishTestResults/Parameters: + +Parameter Summary +***************** + +.. rubric:: Goto :ref:`input parameters ` + ++---------------------------------------------------------------------+----------+----------+---------------------------------------------------------------------+ +| Parameter Name | Required | Type | Default | ++=====================================================================+==========+==========+=====================================================================+ +| :ref:`JOBTMPL/PublishTestResults/Input/ubuntu_image_version` | no | string | ``'24.04'`` | ++---------------------------------------------------------------------+----------+----------+---------------------------------------------------------------------+ +| :ref:`JOBTMPL/PublishTestResults/Input/unittest_artifacts_pattern` | no | string | ``'*-UnitTestReportSummary-XML-*'`` | ++---------------------------------------------------------------------+----------+----------+---------------------------------------------------------------------+ +| :ref:`JOBTMPL/PublishTestResults/Input/merged_junit_filename` | no | string | ``'Unittesting.xml'`` | ++---------------------------------------------------------------------+----------+----------+---------------------------------------------------------------------+ +| :ref:`JOBTMPL/PublishTestResults/Input/merged_junit_artifact` | no | string | ``''`` | ++---------------------------------------------------------------------+----------+----------+---------------------------------------------------------------------+ +| :ref:`JOBTMPL/PublishTestResults/Input/merge-input-dialect` | no | string | ``'pyTest-JUnit'`` | ++---------------------------------------------------------------------+----------+----------+---------------------------------------------------------------------+ +| :ref:`JOBTMPL/PublishTestResults/Input/merge-output-dialect` | no | string | ``'pyTest-JUnit'`` | ++---------------------------------------------------------------------+----------+----------+---------------------------------------------------------------------+ +| :ref:`JOBTMPL/PublishTestResults/Input/additional_merge_args` | no | string | ``'"--pytest=rewrite-dunder-init;reduce-depth:pytest.tests.unit"'`` | ++---------------------------------------------------------------------+----------+----------+---------------------------------------------------------------------+ +| :ref:`JOBTMPL/PublishTestResults/Input/testsuite-summary-name` | no | string | ``''`` | ++---------------------------------------------------------------------+----------+----------+---------------------------------------------------------------------+ +| :ref:`JOBTMPL/PublishTestResults/Input/publish` | no | string | ``'true'`` | ++---------------------------------------------------------------------+----------+----------+---------------------------------------------------------------------+ +| :ref:`JOBTMPL/PublishTestResults/Input/report_title` | no | string | ``'Unit Test Results'`` | ++---------------------------------------------------------------------+----------+----------+---------------------------------------------------------------------+ +| :ref:`JOBTMPL/PublishTestResults/Input/dorny` | no | string | ``'true'`` | ++---------------------------------------------------------------------+----------+----------+---------------------------------------------------------------------+ +| :ref:`JOBTMPL/PublishTestResults/Input/codecov` | no | string | ``'false'`` | ++---------------------------------------------------------------------+----------+----------+---------------------------------------------------------------------+ +| :ref:`JOBTMPL/PublishTestResults/Input/codecov_flags` | no | string | ``'unittest'`` | ++---------------------------------------------------------------------+----------+----------+---------------------------------------------------------------------+ + +.. rubric:: Goto :ref:`secrets ` + ++-----------------------------------------------------------+----------+----------+--------------+ +| Token Name | Required | Type | Default | ++===========================================================+==========+==========+==============+ +| :ref:`JOBTMPL/PublishTestResults/Secret/CODECOV_TOKEN` | no | string | — — — — | ++-----------------------------------------------------------+----------+----------+--------------+ + +.. rubric:: Goto :ref:`output parameters ` + +This job template has no output parameters. + + +.. _JOBTMPL/PublishTestResults/Inputs: + +Input Parameters +**************** + +.. _JOBTMPL/PublishTestResults/Input/ubuntu_image_version: + +ubuntu_image_version +==================== + +:Type: string +:Required: no +:Default Value: ``'24.04'`` +:Possible Values: See `actions/runner-images - Available Images `__ + for available Ubuntu image versions. +:Description: Version of the Ubuntu image used to run this job. + + .. note:: + + Unfortunately, GitHub Actions has only a `limited set of functions `__, + thus, the usual Ubuntu image name like ``'ubuntu-24.04'`` can't be split into image name and image + version. + + +.. _JOBTMPL/PublishTestResults/Input/unittest_artifacts_pattern: + +unittest_artifacts_pattern +========================== + +:Type: string +:Required: no +:Default Value: ``'*-UnitTestReportSummary-XML-*'`` +:Possible Values: Any valid artifact matching pattern using fixed text and ``*`` characters. +:Description: tbd + + +.. _JOBTMPL/PublishTestResults/Input/merged_junit_filename: + +merged_junit_filename +===================== + +:Type: string +:Required: no +:Default Value: ``'Unittesting.xml'`` +:Possible Values: Any valid filename suitable for a JUnit XML report. +:Description: The filename for the merged JUnit report in XML format. |br| + See :ref:`JOBTMPL/PublishTestResults/Input/merge-output-dialect` for the used JUnit dialect in the + merged report file. + + +.. _JOBTMPL/PublishTestResults/Input/merged_junit_artifact: + +merged_junit_artifact +===================== + +:Type: string +:Required: no +:Default Value: ``''`` +:Possible Values: Any valid artifact name. +:Description: + + +.. _JOBTMPL/PublishTestResults/Input/merge-input-dialect: + +merge-input-dialect +=================== + +:Type: string +:Required: no +:Default Value: ``'pyTest-JUnit'`` +:Possible Values: tbd +:Description: tbd + + +.. _JOBTMPL/PublishTestResults/Input/merge-output-dialect: + +merge-output-dialect +==================== + +:Type: string +:Required: no +:Default Value: ``'pyTest-JUnit'`` +:Possible Values: tbd +:Description: tbd + + +.. _JOBTMPL/PublishTestResults/Input/additional_merge_args: + +additional_merge_args +===================== + +:Type: string +:Required: no +:Default Value: ``'"--pytest=rewrite-dunder-init;reduce-depth:pytest.tests.unit"'`` +:Possible Values: tbd +:Description: tbd + + +.. _JOBTMPL/PublishTestResults/Input/testsuite-summary-name: + +testsuite-summary-name +====================== + +:Type: string +:Required: no +:Default Value: ``''`` +:Possible Values: tbd +:Description: tbd + + +.. _JOBTMPL/PublishTestResults/Input/publish: + +publish +======= + +:Type: string +:Required: no +:Default Value: ``'true'`` +:Possible Values: tbd +:Description: tbd + + +.. _JOBTMPL/PublishTestResults/Input/report_title: + +report_title ============ -+----------------+----------+----------+---------------------------------+ -| Parameter Name | Required | Type | Default | -+================+==========+==========+=================================+ -| report_files | optional | string | ``artifacts/**/*.xml`` | -+----------------+----------+----------+---------------------------------+ +:Type: string +:Required: no +:Default Value: ``'Unit Test Results'`` +:Possible Values: tbd +:Description: tbd -Pattern of jUnit report files to publish as Markdown. -The parameter can be a comma separated list. Wildcards are supported. +.. _JOBTMPL/PublishTestResults/Input/dorny: -.. hint:: +dorny +===== - All artifacts are downloaded into directory ``artifacts``, thus the pattern should include this directory as a - prefix. +:Type: string +:Required: no +:Default Value: ``'true'`` +:Possible Values: tbd +:Description: tbd +.. _JOBTMPL/PublishTestResults/Input/codecov: + +codecov +======= + +:Type: string +:Required: no +:Default Value: ``'false'`` +:Possible Values: tbd +:Description: tbd + + +.. _JOBTMPL/PublishTestResults/Input/codecov_flags: + +codecov_flags +============= + +:Type: string +:Required: no +:Default Value: ``'unittest'`` +:Possible Values: tbd +:Description: tbd + + +.. _JOBTMPL/PublishTestResults/Secrets: + Secrets ******* -This job template needs no secrets. +.. _JOBTMPL/PublishTestResults/Secret/CODECOV_TOKEN: -Results +CODECOV_TOKEN +============= + +:Type: string +:Required: no +:Default Value: — — — — +:Description: The token to publish unit test results on `Codecov `__. + + +.. _JOBTMPL/PublishTestResults/Outputs: + +Outputs ******* This job template has no output parameters. + + +.. _JOBTMPL/PublishTestResults/Optimizations: + +Optimizations +************* + +This template offers no optimizations (reduced job runtime). + +merged_junit_artifact diff --git a/doc/JobTemplate/Setup/ExtractConfiguration.rst b/doc/JobTemplate/Setup/ExtractConfiguration.rst index 2771040..b8ab9f3 100644 --- a/doc/JobTemplate/Setup/ExtractConfiguration.rst +++ b/doc/JobTemplate/Setup/ExtractConfiguration.rst @@ -19,7 +19,7 @@ The ``ExtractConfiguration`` job template is a ..... * Extract code coverage XML report file as directory name, filename and full path from :file:`pyproject.toml`. * Extract code coverage JSON report file as directory name, filename and full path from :file:`pyproject.toml`. -.. topic:: Behavior: +.. topic:: Behavior .. todo:: ExtractConfiguration:Behavior needs documentation. diff --git a/doc/JobTemplate/Testing/UnitTesting.rst b/doc/JobTemplate/Testing/UnitTesting.rst index 17865b1..1d46f43 100644 --- a/doc/JobTemplate/Testing/UnitTesting.rst +++ b/doc/JobTemplate/Testing/UnitTesting.rst @@ -22,7 +22,7 @@ file. * Provide code coverage results as pytest JSON file. * Provide code coverage results as HTML report. -.. topic:: Behavior: +.. topic:: Behavior .. todo:: UnitTesting:Behavior needs documentation. diff --git a/doc/_static/pyTooling-Actions-PublishTestResults-Dorny.png b/doc/_static/pyTooling-Actions-PublishTestResults-Dorny.png new file mode 100644 index 0000000..8b00722 Binary files /dev/null and b/doc/_static/pyTooling-Actions-PublishTestResults-Dorny.png differ diff --git a/doc/_static/pyTooling-Actions-PublishTestResults.png b/doc/_static/pyTooling-Actions-PublishTestResults.png new file mode 100644 index 0000000..72c7aa2 Binary files /dev/null and b/doc/_static/pyTooling-Actions-PublishTestResults.png differ