Compare commits

..

61 Commits

Author SHA1 Message Date
umarcor
14ac6c6386 v0.4.1 2022-01-17 01:30:38 +01:00
Patrick Lehmann
18357ec213 UnitTesting: add options 'pacboy' and 'mingw_requirements' 2022-01-17 01:29:16 +01:00
umarcor
4220a50041 UnitTesting: add options 'pacboy' and 'mingw_requirements' 2022-01-17 00:07:56 +01:00
umarcor
d8264eab8a v0.4.0 2022-01-09 20:58:08 +01:00
Patrick Lehmann
b9d3839abb Extend Unit Tests to multiple systems (ubuntu, windows, msys2 and macos) 2022-01-09 20:56:58 +01:00
umarcor
997d548e60 Parameters: update py36 warning, add py311 notice 2022-01-09 20:39:36 +01:00
umarcor
83cd572694 UnitTesting: update description of input 'jobs' 2022-01-09 20:39:12 +01:00
Unai Martinez-Corral
68a446b9b6 UnitTesting: workarounds for MSYS2-MinGW64 (pacman) (#35) 2022-01-09 20:19:56 +01:00
Patrick Lehmann
43f0b79e88 UnitTesting/msys2: install system packages through 'pacboy'; refactor 2022-01-09 20:05:51 +01:00
umarcor
b3d8a9c5ec UnitTesting: refactor 2022-01-09 20:02:09 +01:00
umarcor
edb6ca364e UnitTesting/msys2: install system packages through 'pacboy' 2022-01-09 20:02:09 +01:00
Patrick Lehmann
e00f5cf53d Added MSYS2-MinGW64 specific code. 2022-01-09 20:00:36 +01:00
Unai Martinez-Corral
0da8c5a5c5 UnitTesting: install wheel; support py3.11; fix scripting for Windows (PowerShell commands) (#34) 2022-01-09 19:34:23 +01:00
Patrick Lehmann
c9bee6fe65 Require wheel to be installed before installing requirements. 2022-01-09 18:44:06 +01:00
Patrick Lehmann
94bb01d586 Renamed Python 3.11 version. 2022-01-09 18:25:02 +01:00
Patrick Lehmann
0fdef33cb4 Improved PoSh code. 2022-01-09 18:10:42 +01:00
Patrick Lehmann
e1f7599d79 Added Python 3.11 (currently RC). 2022-01-09 18:07:29 +01:00
Patrick Lehmann
dad5e71bfe Added PowerShell code. 2022-01-09 17:39:16 +01:00
umarcor
60d77c2292 Parameters: support system 'msys2' (MINGW64); update UnitTesting accordingly 2022-01-07 01:40:42 +01:00
umarcor
3f489f0bed Parameters: add option 'system_list'; UnitTesting now requires field 'system' in the matrix 2022-01-07 00:39:14 +01:00
umarcor
26afa43fa4 Parameters: remove 3.6 from default python_version_list 2022-01-07 00:37:22 +01:00
Patrick Lehmann
c8003f1a0e Added PyCharm project files. 2021-12-26 11:11:30 +01:00
Unai Martinez-Corral
6413469cdf v0.3.0 2021-12-26 01:40:09 +01:00
Unai Martinez-Corral
8dbacda32c Pytest using pyproject.toml (#29)
# New Features
* Allow configuration of the unit test directory (default: `tests/unit`).
* Allow configuration of a `pyproject.toml` or `.coveragerc` file.
* Extract values from `pyproject.toml` or `.coveragerc`.

# Changes
* Jobs deriving from template job `CoverageCollection` need to specify their `.coveragerc` content and `pytest.ini` content in a `pyproject.toml` file or provide the job parameter `coverage_config` with the path to the `.coveragerc` file.

# Bug Fixes
*None*

-------------
* Closes #2.
2021-12-25 23:40:50 +01:00
Patrick Lehmann
78b225195f Updated README according to latest changes. 2021-12-24 21:05:04 +01:00
umarcor
1fbeef36d6 CoverageCollection: skip config file if empty 2021-12-24 16:37:45 +01:00
umarcor
9846c9e60c CoverageCollection: use 'pyproject.toml' by default 2021-12-24 16:31:47 +01:00
umarcor
b8564eb389 CoverageCollection: xmlFile defaults to './coverage.xml' 2021-12-24 16:24:55 +01:00
umarcor
62cd2d1d0f CoverageCollection: htmlDirectory defaults to 'htmlcov' 2021-12-24 16:22:14 +01:00
umarcor
9bd8004dfb CoverageCollection: pass output directory to coverage html 2021-12-24 15:59:23 +01:00
umarcor
925b44a8a8 CoverageCollection: fix variable name 2021-12-24 15:54:09 +01:00
Patrick Lehmann
9d8c1ecc05 Merge branch 'dev' into toml 2021-12-24 14:15:28 +01:00
umarcor
66c7b4b619 ExamplePipeline/ArtifactCleanup: needs PublishTestResults instead of UnitTesting 2021-12-24 14:11:37 +01:00
Patrick Lehmann
b399aa8f93 Parameters: mark Python 3.6 black, update others, warn about unsupported versions 2021-12-24 14:00:34 +01:00
umarcor
dcd0a4b617 Parameters: mark Python 3.6 black, update others, warn about unsupported versions 2021-12-24 13:52:30 +01:00
Patrick Lehmann
d7c765ba79 Fixed how to access complex nested key-value pairs. 2021-12-24 13:04:31 +01:00
Patrick Lehmann
fa10ed076c Install dependency tomli before script execution. 2021-12-24 12:56:03 +01:00
Patrick Lehmann
9dfafd588e Changed scripting from bash to Python. Also use .coveragerc as fallback. 2021-12-24 12:55:59 +01:00
Patrick Lehmann
09f7504de4 Fixed path to project root. 2021-12-24 12:55:56 +01:00
Patrick Lehmann
6ad23eabf5 Extract information from TOML file. 2021-12-24 12:55:50 +01:00
Patrick Lehmann
bb855d572d Pytest using pyproject.toml. 2021-12-24 12:54:14 +01:00
umarcor
250cceb80d releaser: update README.md and DEVELOPMENT.md 2021-12-21 02:05:21 +01:00
umarcor
615aafc0b4 releaser: add DEVELOPMENT.md 2021-12-21 02:05:21 +01:00
umarcor
1f3d12ef95 README: add 'Container Step' to Context; add References 2021-12-21 01:39:02 +01:00
Unai Martinez-Corral
608670c6b9 v0.2.3
# Bug Fixes
*  releaser/composite: quote path to releaser.py (#27)
2021-12-21 01:02:13 +01:00
Unai Martinez-Corral
9cee3342e3 releaser/composite: quote path to releaser.py (#27) 2021-12-21 00:45:30 +01:00
jeremyd2019
aec2613cd1 releaser/composite: quote path to releaser.py
Fixes #26
2021-12-20 14:45:07 -08:00
Patrick Lehmann
7c406d96e7 v0.2.2 2021-12-20 08:53:56 +01:00
umarcor
459faf880a releaser: use GitHub CLI by default; remove option 'use-gh-cli' 2021-12-20 07:28:13 +01:00
umarcor
e832625624 releaser: refactor 2021-12-20 07:28:13 +01:00
umarcor
ee02a39a5e releaser: build and use image ghcr.io/pytooling/releaser 2021-12-20 07:28:13 +01:00
umarcor
52491e6bcc releaser: add option 'use-gh-cli' 2021-12-20 06:41:40 +01:00
umarcor
4177a535f1 releaser: refactor; add func GetOrCreateRelease 2021-12-20 06:41:40 +01:00
umarcor
548437b824 releaser: refactor; reorder CheckRefSemVer and GetRepositoryHandle 2021-12-20 06:41:39 +01:00
umarcor
596d0d774f releaser: refactor; add func GetRepositoryHandler 2021-12-20 06:41:39 +01:00
umarcor
877928ba4a releaser: refactor; add func CheckRefSemVer 2021-12-20 06:41:39 +01:00
umarcor
a06d90f456 releaser: refactor; add func UploadArtifacts 2021-12-20 06:41:33 +01:00
umarcor
dcaff1e5a1 releaser: refactor; add func GetReleaseHandler 2021-12-20 06:41:24 +01:00
umarcor
4d666520a0 releaser: refactor; add func GetGitHubAPIHandler 2021-12-20 06:41:19 +01:00
umarcor
cea369d703 releaser: refactor; add func GetListOfArtifacts 2021-12-20 06:41:13 +01:00
umarcor
08a19429d4 releaser: refactor; add func UpdateReference 2021-12-20 06:41:08 +01:00
15 changed files with 493 additions and 312 deletions

View File

@@ -35,6 +35,16 @@ on:
required: false required: false
default: '-r tests/requirements.txt' default: '-r tests/requirements.txt'
type: string type: string
unittest_directory:
description: 'Path to the directory containing unit tests.'
required: false
default: 'tests/unit'
type: string
coverage_config:
description: 'Path to the .coveragerc file. Use pyproject.toml by default.'
required: false
default: 'pyproject.toml'
type: string
artifact: artifact:
description: 'Name of the coverage artifact.' description: 'Name of the coverage artifact.'
required: true required: true
@@ -62,27 +72,68 @@ jobs:
- name: 🗂 Install dependencies - name: 🗂 Install dependencies
run: | run: |
python -m pip install -U pip python -m pip install -U pip
python -m pip install tomli
python -m pip install ${{ inputs.requirements }} python -m pip install ${{ inputs.requirements }}
- name: 🔁 Extract configurations from pyproject.toml
id: getVariables
shell: python
run: |
from pathlib import Path
from tomli import load as tomli_load
htmlDirectory = 'htmlcov'
xmlFile = './coverage.xml'
coverageRC = "${{ inputs.coverage_config }}".strip()
# Read output paths from 'pyproject.toml' file
if coverageRC == "pyproject.toml":
pyProjectFile = Path("pyproject.toml")
if pyProjectFile.exists():
with pyProjectFile.open("rb") as file:
pyProjectSettings = tomli_load(file)
htmlDirectory = pyProjectSettings["tool"]["coverage"]["html"]["directory"]
xmlFile = pyProjectSettings["tool"]["coverage"]["xml"]["output"]
else:
print(f"File '{pyProjectFile}' not found and no ' .coveragerc' file specified.")
# Read output paths from '.coveragerc' file
elif len(coverageRC) > 0:
coverageRCFile = Path(coverageRC)
if coverageRCFile.exists():
with coverageRCFile.open("rb") as file:
coverageRCSettings = tomli_load(file)
htmlDirectory = coverageRCSettings["html"]["directory"]
xmlFile = coverageRCSettings["xml"]["output"]
else:
print(f"File '{coverageRCFile}' not found.")
print(f"::set-output name=coverage_report_html_directory::{htmlDirectory}")
print(f"::set-output name=coverage_report_xml::{xmlFile}")
print(f"DEBUG:\n html={htmlDirectory}\n xml={xmlFile}")
- name: Collect coverage - name: Collect coverage
continue-on-error: true continue-on-error: true
run: | run: |
python -m pytest -rA --cov=.. --cov-config=tests/.coveragerc tests/unit --color=yes [ 'x${{ inputs.coverage_config }}' != 'x' ] && PYCOV_ARGS='--cov-config=${{ inputs.coverage_config }}' || unset PYCOV_ARGS
python -m pytest -rA --cov=. $PYCOV_ARGS ${{ inputs.unittest_directory }} --color=yes
- name: Convert to cobertura format - name: Convert to cobertura format
run: coverage xml run: coverage xml
- name: Convert to HTML format - name: Convert to HTML format
run: | run: |
coverage html coverage html -d ${{ steps.getVariables.outputs.coverage_report_html_directory }}
rm htmlcov/.gitignore rm ${{ steps.getVariables.outputs.coverage_report_html_directory }}/.gitignore
- name: 📤 Upload 'Coverage Report' artifact - name: 📤 Upload 'Coverage Report' artifact
continue-on-error: true continue-on-error: true
uses: actions/upload-artifact@v2 uses: actions/upload-artifact@v2
with: with:
name: ${{ inputs.artifact }} name: ${{ inputs.artifact }}
path: htmlcov path: ${{ steps.getVariables.outputs.coverage_report_html_directory }}
if-no-files-found: error if-no-files-found: error
retention-days: 1 retention-days: 1
@@ -90,7 +141,7 @@ jobs:
continue-on-error: true continue-on-error: true
uses: codecov/codecov-action@v1 uses: codecov/codecov-action@v1
with: with:
file: ./coverage.xml file: ${{ steps.getVariables.outputs.coverage_report_xml }}
flags: unittests flags: unittests
env_vars: PYTHON env_vars: PYTHON
@@ -99,4 +150,4 @@ jobs:
uses: codacy/codacy-coverage-reporter-action@master uses: codacy/codacy-coverage-reporter-action@master
with: with:
project-token: ${{ secrets.codacy_token }} project-token: ${{ secrets.codacy_token }}
coverage-reports: ./coverage.xml coverage-reports: ${{ steps.getVariables.outputs.coverage_report_xml }}

View File

@@ -33,7 +33,12 @@ on:
python_version_list: python_version_list:
description: 'Space separated list of Python versions to run tests with.' description: 'Space separated list of Python versions to run tests with.'
required: false required: false
default: '3.6 3.7 3.8 3.9 3.10' default: '3.7 3.8 3.9 3.10'
type: string
system_list:
description: 'Space separated list of systems to run tests on.'
required: false
default: 'ubuntu windows msys2 macos'
type: string type: string
name: name:
description: 'Name of the tool.' description: 'Name of the tool.'
@@ -75,16 +80,39 @@ jobs:
print("Parameters:") print("Parameters:")
print(params) print(params)
systems = '${{ inputs.system_list }}'.split(' ')
versions = '${{ inputs.python_version_list }}'.split(' ')
if '3.6' in versions:
print("::warning title=Deprecated::Support for Python 3.6 ended in 2021.12.23.")
if '3.11' in versions:
print(f"::notice title=Experimental::Python 3.11 (3.11.0-alpha3) is a pre-release.")
data = { data = {
'3.6': { 'icon': '🔴', 'until': '23.12.2021' }, 'python': {
'3.7': { 'icon': '🟠', 'until': '27.06.2023' }, '3.6': { 'icon': '', 'until': '2021.12.23' },
'3.8': { 'icon': '🟡', 'until': 'Oct. 2024' }, '3.7': { 'icon': '🔴', 'until': '2023.06.27' },
'3.9': { 'icon': '🟢', 'until': 'Oct. 2025' }, '3.8': { 'icon': '🟠', 'until': '2024.10' },
'3.10': { 'icon': '🟢', 'until': 'Oct. 2026' }, '3.9': { 'icon': '🟡', 'until': '2025.10' },
'3.10': { 'icon': '🟢', 'until': '2026.10' },
'3.11': { 'icon': '🟣', 'until': '2027.10' },
},
'sys': {
'ubuntu': { 'icon': '🐧', 'runs-on': 'ubuntu-latest', 'shell': 'bash' },
'windows': { 'icon': '🧊', 'runs-on': 'windows-latest', 'shell': 'pwsh' },
'msys2': { 'icon': '🟦', 'runs-on': 'windows-latest', 'shell': 'msys2 {0}' },
'macos': { 'icon': '🍎', 'runs-on': 'macos-latest', 'shell': 'bash' }
}
} }
jobs = [ jobs = [
{'python': version, 'icon': data[version]['icon']} {
for version in '${{ inputs.python_version_list }}'.split(' ') 'sysicon': data['sys'][system]['icon'],
'system': system,
'runs-on': data['sys'][system]['runs-on'],
'shell': data['sys'][system]['shell'],
'pyicon': data['python'][version]['icon'],
'python': '3.11.0-alpha.3' if version == '3.11' else version
}
for system in systems
for version in (versions if system != 'msys2' else ['3.9'])
] ]
print(f'::set-output name=python_jobs::{jobs!s}') print(f'::set-output name=python_jobs::{jobs!s}')
print("Python jobs:") print("Python jobs:")

View File

@@ -39,65 +39,27 @@ env:
jobs: jobs:
test:
Image:
runs-on: ubuntu-latest runs-on: ubuntu-latest
env:
DOCKER_BUILDKIT: 1
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v2
- run: echo "Build some tool and generate some (versioned) artifacts" > artifact-$(date -u +"%Y-%m-%dT%H-%M-%SZ").txt - name: Build container image
run: docker build -t ghcr.io/pytooling/releaser -f releaser/Dockerfile releaser
- name: Single - name: Push container image
uses: ./releaser uses: ./with-post-step
with: with:
rm: true main: |
token: ${{ secrets.GITHUB_TOKEN }} echo '${{ github.token }}' | docker login ghcr.io -u GitHub-Actions --password-stdin
files: artifact-*.txt docker push ghcr.io/pytooling/releaser
post: docker logout ghcr.io
- name: List
uses: ./releaser
with:
token: ${{ secrets.GITHUB_TOKEN }}
files: |
artifact-*.txt
README.md
- name: Add artifacts/*.txt
run: |
mkdir artifacts
echo "Build some tool and generate some artifacts" > artifacts/artifact.txt
touch artifacts/empty_file.txt
- name: Single in subdir
uses: ./releaser
with:
token: ${{ secrets.GITHUB_TOKEN }}
files: artifacts/artifact.txt
- name: Add artifacts/*.md
run: |
echo "releaser hello" > artifacts/hello.md
echo "releaser world" > artifacts/world.md
- name: Directory wildcard
uses: ./releaser
with:
token: ${{ secrets.GITHUB_TOKEN }}
files: artifacts/*
- name: Add artifacts/subdir
run: |
mkdir artifacts/subdir
echo "Test recursive glob" > artifacts/subdir/deep_file.txt
- name: Directory wildcard (recursive)
uses: ./releaser
with:
token: ${{ secrets.GITHUB_TOKEN }}
files: artifacts/**
test-composite: Composite:
needs: test
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v2
@@ -152,3 +114,63 @@ jobs:
with: with:
token: ${{ secrets.GITHUB_TOKEN }} token: ${{ secrets.GITHUB_TOKEN }}
files: artifacts/** files: artifacts/**
Test:
needs:
- Image
- Composite
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- run: echo "Build some tool and generate some (versioned) artifacts" > artifact-$(date -u +"%Y-%m-%dT%H-%M-%SZ").txt
- name: Single
uses: ./releaser
with:
rm: true
token: ${{ secrets.GITHUB_TOKEN }}
files: artifact-*.txt
- name: List
uses: ./releaser
with:
token: ${{ secrets.GITHUB_TOKEN }}
files: |
artifact-*.txt
README.md
- name: Add artifacts/*.txt
run: |
mkdir artifacts
echo "Build some tool and generate some artifacts" > artifacts/artifact.txt
touch artifacts/empty_file.txt
- name: Single in subdir
uses: ./releaser
with:
token: ${{ secrets.GITHUB_TOKEN }}
files: artifacts/artifact.txt
- name: Add artifacts/*.md
run: |
echo "releaser hello" > artifacts/hello.md
echo "releaser world" > artifacts/world.md
- name: Directory wildcard
uses: ./releaser
with:
token: ${{ secrets.GITHUB_TOKEN }}
files: artifacts/*
- name: Add artifacts/subdir
run: |
mkdir artifacts/subdir
echo "Test recursive glob" > artifacts/subdir/deep_file.txt
- name: Directory wildcard (recursive)
uses: ./releaser
with:
token: ${{ secrets.GITHUB_TOKEN }}
files: artifacts/**

View File

@@ -26,7 +26,7 @@ on:
workflow_call: workflow_call:
inputs: inputs:
jobs: jobs:
description: 'JSON list with field <python>, telling the versions to run tests with.' description: 'JSON list with environment fields, telling the system and Python versions to run tests with.'
required: true required: true
type: string type: string
requirements: requirements:
@@ -34,6 +34,25 @@ on:
required: false required: false
default: '-r tests/requirements.txt' default: '-r tests/requirements.txt'
type: string type: string
pacboy:
description: 'MSYS2 dependencies to be installed through pacboy (pacman).'
required: false
default: >-
python-pip:p
python-wheel:p
python-coverage:p
python-lxml:p
type: string
mingw_requirements:
description: 'Override Python dependencies to be installed through pip on MSYS2 (MINGW64) only.'
required: false
default: ''
type: string
unittest_directory:
description: 'Path to the directory containing unit tests.'
required: false
default: 'tests/unit'
type: string
artifact: artifact:
description: "Generate unit test report with junitxml and upload results as an artifact." description: "Generate unit test report with junitxml and upload results as an artifact."
required: false required: false
@@ -43,38 +62,71 @@ on:
jobs: jobs:
UnitTesting: UnitTesting:
name: ${{ matrix.icon }} Unit Tests using Python ${{ matrix.python }} name: ${{ matrix.sysicon }} ${{ matrix.pyicon }} Unit Tests using Python ${{ matrix.python }}
runs-on: ubuntu-latest runs-on: ${{ matrix.runs-on }}
strategy: strategy:
fail-fast: false fail-fast: false
matrix: matrix:
include: ${{ fromJson(inputs.jobs) }} include: ${{ fromJson(inputs.jobs) }}
defaults:
run:
shell: ${{ matrix.shell }}
steps: steps:
- name: ⏬ Checkout repository - name: ⏬ Checkout repository
uses: actions/checkout@v2 uses: actions/checkout@v2
- name: '🟦 Setup MSYS2'
if: matrix.system == 'msys2'
uses: msys2/setup-msys2@v2
with:
msystem: MINGW64
update: true
pacboy: ${{ inputs.pacboy }}
- name: 🐍 Setup Python ${{ matrix.python }} - name: 🐍 Setup Python ${{ matrix.python }}
if: matrix.system != 'msys2'
uses: actions/setup-python@v2 uses: actions/setup-python@v2
with: with:
python-version: ${{ matrix.python }} python-version: ${{ matrix.python }}
- name: 🔧 Install dependencies - name: ⚙️ Update pip
run: python -m pip install -U pip
- name: 🔧 Install wheel and pip dependencies
if: matrix.system != 'msys2'
run: | run: |
python -m pip install -U pip python -m pip install -U wheel
python -m pip install ${{ inputs.requirements }} python -m pip install ${{ inputs.requirements }}
- name: 🔧 Install pip dependencies
if: matrix.system == 'msys2'
run: |
if [ 'x${{ inputs.mingw_requirements }}' != 'x' ]; then
python -m pip install ${{ inputs.mingw_requirements }}
else
python -m pip install ${{ inputs.requirements }}
fi
- name: ☑ Run unit tests - name: ☑ Run unit tests
if: matrix.system == 'windows'
run: |
$PYTEST_ARGS = if ("${{ inputs.artifact }}".length -gt 0) { "--junitxml=TestReport.xml" } else { "" }
python -m pytest -rA ${{ inputs.unittest_directory }} $PYTEST_ARGS --color=yes
- name: ☑ Run unit tests
if: matrix.system != 'windows'
run: | run: |
[ 'x${{ inputs.artifact }}' != 'x' ] && PYTEST_ARGS='--junitxml=TestReport.xml' || unset PYTEST_ARGS [ 'x${{ inputs.artifact }}' != 'x' ] && PYTEST_ARGS='--junitxml=TestReport.xml' || unset PYTEST_ARGS
python -m pytest -rA tests/unit $PYTEST_ARGS --color=yes python -m pytest -rA ${{ inputs.unittest_directory }} $PYTEST_ARGS --color=yes
- name: 📤 Upload 'TestReport.xml' artifact - name: 📤 Upload 'TestReport.xml' artifact
if: inputs.artifact != '' if: inputs.artifact != ''
uses: actions/upload-artifact@v2 uses: actions/upload-artifact@v2
with: with:
name: ${{ inputs.artifact }}-${{ matrix.python }} name: ${{ inputs.artifact }}-${{ matrix.system }}-${{ matrix.python }}
path: TestReport.xml path: TestReport.xml
if-no-files-found: error if-no-files-found: error
retention-days: 1 retention-days: 1

8
.idea/Actions.iml generated Normal file
View File

@@ -0,0 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<module type="PYTHON_MODULE" version="4">
<component name="NewModuleRootManager">
<content url="file://$MODULE_DIR$" />
<orderEntry type="inheritedJdk" />
<orderEntry type="sourceFolder" forTests="false" />
</component>
</module>

View File

@@ -0,0 +1,6 @@
<component name="InspectionProjectProfileManager">
<settings>
<option name="USE_PROJECT_PROFILE" value="false" />
<version value="1.0" />
</settings>
</component>

8
.idea/modules.xml generated Normal file
View File

@@ -0,0 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="ProjectModuleManager">
<modules>
<module fileurl="file://$PROJECT_DIR$/.idea/Actions.iml" filepath="$PROJECT_DIR$/.idea/Actions.iml" />
</modules>
</component>
</project>

View File

@@ -34,6 +34,7 @@ jobs:
with: with:
name: ToolName name: ToolName
# Optional # Optional
system_list: 'ubuntu windows msys2 macos'
python_version: '3.10' python_version: '3.10'
python_version_list: '3.8 3.9 3.10' python_version_list: '3.8 3.9 3.10'
@@ -45,6 +46,13 @@ jobs:
jobs: ${{ needs.Params.outputs.python_jobs }} jobs: ${{ needs.Params.outputs.python_jobs }}
# Optional # Optional
requirements: '-r tests/requirements.txt' requirements: '-r tests/requirements.txt'
pacboy: >-
python-pip:p
python-wheel:p
python-coverage:p
python-lxml:p
mingw_requirements: '-r tests/requirements.mingw.txt'
unittest_directory: 'tests/unit'
artifact: ${{ fromJson(needs.Params.outputs.params).artifacts.unittesting }} artifact: ${{ fromJson(needs.Params.outputs.params).artifacts.unittesting }}
Coverage: Coverage:
@@ -148,7 +156,7 @@ jobs:
uses: pyTooling/Actions/.github/workflows/ArtifactCleanUp.yml@main uses: pyTooling/Actions/.github/workflows/ArtifactCleanUp.yml@main
needs: needs:
- Params - Params
- UnitTesting - PublishTestResults
- Coverage - Coverage
- StaticTypeCheck - StaticTypeCheck
- BuildTheDocs - BuildTheDocs

View File

@@ -7,24 +7,30 @@ language for writing reusable CI code.
However, Python being equally popular and capable, usage of JS/TS might be bypassed, with some caveats. However, Python being equally popular and capable, usage of JS/TS might be bypassed, with some caveats.
This repository gathers reusable CI tooling for testing, packaging and distributing Python projects and documentation. This repository gathers reusable CI tooling for testing, packaging and distributing Python projects and documentation.
## Context ## Context
GitHub Actions supports four types of reusable code: GitHub Actions supports five procedures to reuse code:
- JavaScript Action. - JavaScript Action:
- [docs.github.com: actions/creating-actions/creating-a-javascript-action](https://docs.github.com/en/actions/creating-actions/creating-a-javascript-action) - [docs.github.com: actions/creating-actions/creating-a-javascript-action](https://docs.github.com/en/actions/creating-actions/creating-a-javascript-action)
- Container Action. - Container Action:
- [docs.github.com: actions/creating-actions/creating-a-docker-container-action](https://docs.github.com/en/actions/creating-actions/creating-a-docker-container-action) - [docs.github.com: actions/creating-actions/creating-a-docker-container-action](https://docs.github.com/en/actions/creating-actions/creating-a-docker-container-action)
- Composite Action. - Container Step:
- [docs.github.com: actions/learn-github-actions/workflow-syntax-for-github-actions#example-using-a-docker-public-registry-action](https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#example-using-a-docker-public-registry-action)
- [docs.github.com: actions/learn-github-actions/workflow-syntax-for-github-actions#jobsjob_idstepswithargs](https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#jobsjob_idstepswithargs)
- Composite Action:
- [docs.github.com: actions/creating-actions/creating-a-composite-action](https://docs.github.com/en/actions/creating-actions/creating-a-composite-action) - [docs.github.com: actions/creating-actions/creating-a-composite-action](https://docs.github.com/en/actions/creating-actions/creating-a-composite-action)
- [github.blog/changelog: 2020-08-07-github-actions-composite-run-steps](https://github.blog/changelog/2020-08-07-github-actions-composite-run-steps/) - [github.blog/changelog: 2020-08-07-github-actions-composite-run-steps](https://github.blog/changelog/2020-08-07-github-actions-composite-run-steps/)
- [github.blog/changelog: 2021-08-25-github-actions-reduce-duplication-with-action-compositio](https://github.blog/changelog/2021-08-25-github-actions-reduce-duplication-with-action-composition/) - [github.blog/changelog: 2021-08-25-github-actions-reduce-duplication-with-action-compositio](https://github.blog/changelog/2021-08-25-github-actions-reduce-duplication-with-action-composition/)
- Reusable Workflows. - Reusable Workflow:
- [docs.github.com: actions/learn-github-actions/reusing-workflows](https://docs.github.com/en/actions/learn-github-actions/reusing-workflows) - [docs.github.com: actions/learn-github-actions/reusing-workflows](https://docs.github.com/en/actions/learn-github-actions/reusing-workflows)
- [github.blog/changelog: 2021-10-05-github-actions-dry-your-github-actions-configuration-by-reusing-workflows](https://github.blog/changelog/2021-10-05-github-actions-dry-your-github-actions-configuration-by-reusing-workflows/) - [github.blog/changelog: 2021-10-05-github-actions-dry-your-github-actions-configuration-by-reusing-workflows](https://github.blog/changelog/2021-10-05-github-actions-dry-your-github-actions-configuration-by-reusing-workflows/)
Leaving JavaScript and Container Actions aside, the main differences between Composite Actions and Reusable Workflows Container Actions and Container Steps are almost equivalent: Actions use a configuration file (`action.yml`), while
are the following: Steps do not.
Leaving JavaScript and Container Actions and Steps aside, the main differences between Composite Actions and Reusable
Workflows are the following:
- Composite Actions can be executed from a remote/external path or from the checked out branch, and from any location. - Composite Actions can be executed from a remote/external path or from the checked out branch, and from any location.
However, Reusable Workflows can only be used through a remote/external path (`{owner}/{repo}/{path}/{filename}@{ref}`), However, Reusable Workflows can only be used through a remote/external path (`{owner}/{repo}/{path}/{filename}@{ref}`),
@@ -74,6 +80,7 @@ It allows using the `post` feature with scripts written in bash, python or any o
the environment. the environment.
See: [actions/runner#1478](https://github.com/actions/runner/issues/1478). See: [actions/runner#1478](https://github.com/actions/runner/issues/1478).
## Reusable workflows ## Reusable workflows
This repository provides 10+ Reusable Workflows based on the CI pipelines of the repos in this organisation, This repository provides 10+ Reusable Workflows based on the CI pipelines of the repos in this organisation,
@@ -88,13 +95,16 @@ As shown in the screenshot above, the expected order is:
- Global: - Global:
- [Parameters](.github/workflows/Parameters.yml): a workaround for the limitations to handle global variables in - [Parameters](.github/workflows/Parameters.yml): a workaround for the limitations to handle global variables in
GitHub Actions workflows (see [actions/runner#480](https://github.com/actions/runner/issues/480)). GitHub Actions workflows (see [actions/runner#480](https://github.com/actions/runner/issues/480)).
It generates outputs with artifact names and job matrices to be used in other jobs. It generates outputs with artifact names and job matrices to be used in later running jobs.
- Code testing/analysis: - Code testing/analysis:
- [UnitTesting](.github/workflows/UnitTesting.yml): run unit test with `pytest` using multiple versions of Python, and - [UnitTesting](.github/workflows/UnitTesting.yml): run unit test with `pytest` using multiple versions of Python, and
optionally upload results as XML reports. optionally upload results as XML reports. Configuration options to `pytest` should be given via section
- [CoverageCollection](.github/workflows/CoverageCollection.yml): collect coverage data with `pytest` using a single `[tool.pytest.ini_options]` in a `pyproject.toml` file.
version of Python, generate HTML and Cobertura (XML) reports, upload the HTML report as an artifact, and upload the - [CoverageCollection](.github/workflows/CoverageCollection.yml): collect code coverage data (incl. branch coverage)
results to Codecov and Codacy. with `pytest`/`pytest-cov`/`coverage.py` using a single version of Python (latest). It generates HTML and Cobertura
(XML)reports, upload the HTML report as an artifact, and upload the test results to Codecov and Codacy. Configuration
options to `pytest` and `coverage.py` should be given via section `[tool.pytest.ini_options]` and `[tool.coverage.*]`
in a `pyproject.toml` file.
- [StaticTypeCheck](.github/workflows/StaticTypeCheck.yml): collect static type check result with `mypy`, and - [StaticTypeCheck](.github/workflows/StaticTypeCheck.yml): collect static type check result with `mypy`, and
optionally upload results as an HTML report. optionally upload results as an HTML report.
Example `commands`: Example `commands`:
@@ -121,7 +131,7 @@ As shown in the screenshot above, the expected order is:
mypy --html-report ../htmlmypy -p ToolName mypy --html-report ../htmlmypy -p ToolName
``` ```
- [VerifyDocs](.github/workflows/VerifyDocs.yml): extract code examples from the README and test. - [VerifyDocs](.github/workflows/VerifyDocs.yml): extract code examples from the README and test these code snippets.
- Packaging and releasing: - Packaging and releasing:
- [Release](.github/workflows/Release.yml): publish GitHub Release. - [Release](.github/workflows/Release.yml): publish GitHub Release.
- [Package](.github/workflows/Package.yml): generate source and wheel packages, and upload them as an artifact. - [Package](.github/workflows/Package.yml): generate source and wheel packages, and upload them as an artifact.
@@ -150,19 +160,23 @@ Find further usage cases in the following list of projects:
- [VHDL/pyVHDLModel](https://github.com/VHDL/pyVHDLModel/tree/main/.github/workflows) - [VHDL/pyVHDLModel](https://github.com/VHDL/pyVHDLModel/tree/main/.github/workflows)
## References
- [hdl/containers#48](https://github.com/hdl/containers/issues/48)
## Contributors ## Contributors
* [Patrick Lehmann](https://GitHub.com/Paebbels) * [Patrick Lehmann](https://GitHub.com/Paebbels)
* [Unai Martinez-Corral](https://GitHub.com/umarcor) (Maintainer) * [Unai Martinez-Corral](https://GitHub.com/umarcor) (Maintainer)
* [and more...](https://GitHub.com/pyTooling/Actions/graphs/contributors) * [and more...](https://GitHub.com/pyTooling/Actions/graphs/contributors)
## License ## License
This Python package (source code) licensed under [Apache License 2.0](LICENSE.md). This Python package (source code) licensed under [Apache License 2.0](LICENSE.md).
The accompanying documentation is licensed under [Creative Commons - Attribution 4.0 (CC-BY 4.0)](doc/Doc-License.rst). The accompanying documentation is licensed under [Creative Commons - Attribution 4.0 (CC-BY 4.0)](doc/Doc-License.rst).
---
------------------------- SPDX-License-Identifier: Apache-2.0
SPDX-License-Identifier: Apache-2.0

8
releaser/DEVELOPMENT.md Normal file
View File

@@ -0,0 +1,8 @@
# Releaser Development
- [pyTooling/pyAttributes](https://github.com/pyTooling/pyAttributes) or
[willmcgugan/rich](https://github.com/willmcgugan/rich) might be used to enhance the UX.
- It might be desirable to have pyTooling.Version.SemVersion handle the regular expression from
[semver.org](https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string), and use
proper Python classes in **Releaser**.

View File

@@ -1,4 +1,12 @@
FROM python:3.9-slim-bullseye FROM python:3.9-slim-bullseye
COPY releaser.py /releaser.py COPY releaser.py /releaser.py
RUN pip install PyGithub --progress-bar off RUN pip install PyGithub --progress-bar off \
&& apt update -qq \
&& apt install -y curl \
&& curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | \
dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | \
tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
&& apt update -qq \
&& apt install -y gh
CMD ["/releaser.py"] CMD ["/releaser.py"]

View File

@@ -6,8 +6,8 @@
Combined with a workflow that is executed periodically, **Releaser** allows to provide a fixed release name for users willing Combined with a workflow that is executed periodically, **Releaser** allows to provide a fixed release name for users willing
to use daily/nightly artifacts of a project. to use daily/nightly artifacts of a project.
Furthermore, when any [semver](https://semver.org) compilant tagged commit is pushed, **Releaser** can create a release and Furthermore, when any [semver](https://semver.org) compilant tagged commit is pushed, **Releaser** can create a release
upload assets. and upload assets.
## Context ## Context
@@ -17,16 +17,19 @@ GitHub provides official clients for the GitHub API through [github.com/octokit]
- [octokit.rb](https://github.com/octokit/octokit.rb) ([octokit.github.io/octokit.rb](http://octokit.github.io/octokit.rb)) - [octokit.rb](https://github.com/octokit/octokit.rb) ([octokit.github.io/octokit.rb](http://octokit.github.io/octokit.rb))
- [octokit.net](https://github.com/octokit/octokit.net) ([octokitnet.rtfd.io](https://octokitnet.rtfd.io)) - [octokit.net](https://github.com/octokit/octokit.net) ([octokitnet.rtfd.io](https://octokitnet.rtfd.io))
When GitHub Actions was released in 2019, two Actions were made available through [github.com/actions](https://github.com/actions) for dealing with GitHub Releases: When GitHub Actions was released in 2019, two Actions were made available through
[github.com/actions](https://github.com/actions) for dealing with GitHub Releases:
- [actions/create-release](https://github.com/actions/create-release) - [actions/create-release](https://github.com/actions/create-release)
- [actions/upload-release-asset](https://github.com/actions/upload-release-asset) - [actions/upload-release-asset](https://github.com/actions/upload-release-asset)
However, those Actions were contributed by an employee in spare time, not officially supported by GitHub. However, those Actions were contributed by an employee in spare time, not officially supported by GitHub.
Therefore, they were unmaintained before GitHub Actions was out of the private beta (see [actions/upload-release-asset#58](https://github.com/actions/upload-release-asset/issues/58)) and, a year later, archived. Therefore, they were unmaintained before GitHub Actions was out of the private beta
(see [actions/upload-release-asset#58](https://github.com/actions/upload-release-asset/issues/58))
and, a year later, archived.
Those Actions are based on [actions/toolkit](https://github.com/actions/toolkit)'s hydrated version of octokit.js. Those Actions are based on [actions/toolkit](https://github.com/actions/toolkit)'s hydrated version of octokit.js.
From a practical point of view, [actions/github-script](https://github.com/actions/github-script) is the natural replacement to those Actions, since it allows to use a pre-authenticated octokit.js client along with the workflow run context. From a practical point of view, [actions/github-script](https://github.com/actions/github-script) is the natural replacement to those Actions, since it allows to use a pre-authenticated *octokit.js* client along with the workflow run context.
Still, it requires writing plain JavaScript. Still, it requires writing plain JavaScript.
Alternatively, there are non-official GitHub API libraries available in other languages (see [docs.github.com: rest/overview/libraries](https://docs.github.com/en/rest/overview/libraries)). Alternatively, there are non-official GitHub API libraries available in other languages (see [docs.github.com: rest/overview/libraries](https://docs.github.com/en/rest/overview/libraries)).
@@ -45,6 +48,20 @@ as assets.
In this context, one of the main use cases of **Releaser** is pushing artifacts as release assets. In this context, one of the main use cases of **Releaser** is pushing artifacts as release assets.
Thus, the name of the Action. Thus, the name of the Action.
GitHub provides an official CLI tool, written in golang: [cli/cli](https://github.com/cli/cli).
When the Python version of **Releaser** was written, `cli` was evaluated as an alternative to *PyGitHub*.
`gh release` was (and still is) not flexible enough to update the reference of a release, without deleting and
recreating it (see [cli.github.com: manual/gh_release_create](https://cli.github.com/manual/gh_release_create)).
Deletion and recreation is unfortunate, because it notifies all the watchers of a repository
(see [eine/tip#111](https://github.com/eine/tip/issues/111)).
However, [cli.github.com: manual/gh_release_upload](https://cli.github.com/manual/gh_release_upload) handles uploading
artifacts as assets faster and with better stability for larger files than *PyGitHub*
(see [msys2/msys2-installer#36](https://github.com/msys2/msys2-installer/pull/36)).
Furthermore, the GitHub CLI is installed on GitHub Actions' default virtual environments.
Although `gh` does not support login through SSH (see [cli/cli#3715](https://github.com/cli/cli/issues/3715)), on GitHub
Actions a token is available `${{ github.token }}`.
Therefore, **Releaser** uses `gh release upload` internally.
## Usage ## Usage
The following block shows a minimal YAML workflow file: The following block shows a minimal YAML workflow file:
@@ -72,7 +89,7 @@ jobs:
# Update tag and pre-release # Update tag and pre-release
# - Update (force-push) tag to the commit that is used in the workflow. # - Update (force-push) tag to the commit that is used in the workflow.
# - Upload artifacts defined by the user. # - Upload artifacts defined by the user.
- uses: pyTooling/Actions/releaser@main - uses: pyTooling/Actions/releaser@r0
with: with:
token: ${{ secrets.GITHUB_TOKEN }} token: ${{ secrets.GITHUB_TOKEN }}
files: | files: |
@@ -80,52 +97,19 @@ jobs:
README.md README.md
``` ```
### Troubleshooting
GitHub's internal connections seem not to be very stable; as a result, uploading artifacts as assets does produce
failures rather frequently, particularly if large tarballs are to be published.
When failures are produced, some assets are left in a broken state within the release.
**Releaser** tries to handle those cases by first uploading assets with a `tmp.*` name and then renaming them; if an existing
`tmp.*` is found, it is removed and the upload is retried.
Therefore, restarting the **Releaser** job should suffice for "fixing" a failing run.
Note:
Currently, GitHub Actions does not allow restarting a single job.
That is unfortunate, because **Releaser** is typically used as the last dependent job in the workflows.
Hence, running **Releaser** again requires restarting the whole workflow.
Fortunately, restarting individual jobs is expected to be supported on GitHub Actions in the future.
See [github/roadmap#271](https://github.com/github/roadmap/issues/271) and [actions/runner#432](https://github.com/actions/runner/issues/432).
If the tip/nightly release generated with **Releaser** is broken, and restarting the run cannot fix it, the recommended
procedure is the following:
1. Go to `https://github.com/<name>/<repo>/releases/edit/<tag>`.
2. Edit the assets to:
- Remove the ones with a warning symbol and/or named starting with `tmp.*`.
- Or, remove all of them.
3. Save the changes (click the `Update release` button) and restart the **Releaser** job in CI.
5. If that does still not work, remove the release and restart the **Releaser** job in CI.
See also [eine/tip#160](https://github.com/eine/tip/issues/160).
Note:
If all the assets are removed, or if the release itself is removed, tip/nightly assets won't be available for
users until the workflow is successfully run.
For instance, Action [setup-ghdl-ci](https://github.com/ghdl/setup-ghdl-ci) uses assets from [ghdl/ghdl: releases/tag/nightly](https://github.com/ghdl/ghdl/releases/tag/nightly).
Hence, it is recommended to try removing the conflictive assets only, in order to maximise the availability.
### Composite Action ### Composite Action
The default implementation of **Releaser** is a Container Action. The default implementation of **Releaser** is a Container Action.
Therefore, in each run, the container image is built before starting the job. Therefore, a pre-built container image is pulled before starting the job.
Alternatively, a Composite Action version is available: `uses: pyTooling/Actions/releaser/composite@main`. Alternatively, a Composite Action version is available: `uses: pyTooling/Actions/releaser/composite@main`.
The Composite version installs the dependencies on the host (the runner environment), instead of using a container. The Composite version installs the dependencies on the host (the runner environment), instead of using a container.
Both implementations are functionally equivalent from **Releaser**'s point of view; however, the Composite Action allows users Both implementations are functionally equivalent from **Releaser**'s point of view; however, the Composite Action allows
to tweak the version of Python by using [actions/setup-python](https://github.com/actions/setup-python) before. users to tweak the version of Python by using [actions/setup-python](https://github.com/actions/setup-python) before.
## Options ## Options
All options can be optionally provided as environment variables: `INPUT_TOKEN`, `INPUT_FILES`, `INPUT_TAG`, `INPUT_RM` and/or `INPUT_SNAPSHOTS`. All options can be optionally provided as environment variables: `INPUT_TOKEN`, `INPUT_FILES`, `INPUT_TAG`, `INPUT_RM`
and/or `INPUT_SNAPSHOTS`.
### token (required) ### token (required)
@@ -133,7 +117,8 @@ Token to make authenticated API calls; can be passed in using `{{ secrets.GITHUB
### files (required) ### files (required)
Either a single filename/pattern or a multi-line list can be provided. All the artifacts are uploaded regardless of the hierarchy. Either a single filename/pattern or a multi-line list can be provided. All the artifacts are uploaded regardless of the
hierarchy.
For creating/updating a release without uploading assets, set `files: none`. For creating/updating a release without uploading assets, set `files: none`.
@@ -143,18 +128,28 @@ The default tag name for the tip/nightly pre-release is `tip`, but it can be opt
### rm ### rm
Set option `rm` to `true` for systematically removing previous artifacts (e.g. old versions). Otherwise (by default), all previours artifacts are preserved or overwritten. Set option `rm` to `true` for systematically removing previous artifacts (e.g. old versions).
Otherwise (by default), all previours artifacts are preserved or overwritten.
Note:
If all the assets are removed, or if the release itself is removed, tip/nightly assets won't be available for
users until the workflow is successfully run.
For instance, Action [setup-ghdl-ci](https://github.com/ghdl/setup-ghdl-ci) uses assets from [ghdl/ghdl: releases/tag/nightly](https://github.com/ghdl/ghdl/releases/tag/nightly).
Hence, it is recommended to try removing the conflictive assets only, in order to maximise the availability.
### snapshots ### snapshots
Whether to create releases from any tag or to treat some as snapshots. By default, all the tags with non-empty `prerelease` field (see [semver.org: Is there a suggested regular expression (RegEx) to check a SemVer string?](https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string)) are considered snapshots; neither a release is created nor assets are uploaded. Whether to create releases from any tag or to treat some as snapshots.
By default, all the tags with non-empty `prerelease` field (see [semver.org: Is there a suggested regular expression (RegEx) to check a SemVer string?](https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string))
are considered snapshots; neither a release is created nor assets are uploaded.
## Advanced/complex use cases ## Advanced/complex use cases
**Releaser** is essentially a very fine wrapper to use the GitHub Actions context data along with the classes **Releaser** is essentially a very thin wrapper to use the GitHub Actions context data along with the classes
and methods of PyGithub. and methods of PyGithub.
Similarly to [actions/github-script](https://github.com/actions/github-script), users with advanced/complex requirements might find it desirable to write their own Python script, instead of using **Releaser**. Similarly to [actions/github-script](https://github.com/actions/github-script), users with advanced/complex requirements
might find it desirable to write their own Python script, instead of using **Releaser**.
In fact, since `shell: python` is supported in GitHub Actions, using Python does *not* require any Action. In fact, since `shell: python` is supported in GitHub Actions, using Python does *not* require any Action.
For prototyping purposes, the following job might be useful: For prototyping purposes, the following job might be useful:

View File

@@ -42,4 +42,4 @@ inputs:
default: true default: true
runs: runs:
using: 'docker' using: 'docker'
image: 'Dockerfile' image: 'docker://ghcr.io/pytooling/releaser'

View File

@@ -48,7 +48,7 @@ runs:
run: pip install PyGithub --progress-bar off run: pip install PyGithub --progress-bar off
- shell: bash - shell: bash
run: ${{ github.action_path }}/../releaser.py run: '''${{ github.action_path }}/../releaser.py'''
env: env:
INPUT_TOKEN: ${{ inputs.token }} INPUT_TOKEN: ${{ inputs.token }}
INPUT_FILES: ${{ inputs.files }} INPUT_FILES: ${{ inputs.files }}

View File

@@ -22,196 +22,169 @@
# SPDX-License-Identifier: Apache-2.0 # # SPDX-License-Identifier: Apache-2.0 #
# ==================================================================================================================== # # ==================================================================================================================== #
import re import re
from sys import argv, stdout, exit as sys_exit from sys import argv as sys_argv, stdout, exit as sys_exit
from os import environ, getenv from os import environ, getenv
from glob import glob from glob import glob
from pathlib import Path from pathlib import Path
from github import Github, GithubException from github import Github, GithubException
from subprocess import check_call
print("· Get list of artifacts to be uploaded")
args = [] paramTag = getenv("INPUT_TAG", "tip")
files = [] paramFiles = getenv("INPUT_FILES", None).split()
paramRM = getenv("INPUT_RM", "false") == "true"
paramSnapshots = getenv("INPUT_SNAPSHOTS", "true").lower() == "true"
paramToken = (
environ["GITHUB_TOKEN"]
if "GITHUB_TOKEN" in environ
else environ["INPUT_TOKEN"]
if "INPUT_TOKEN" in environ
else None
)
paramRepo = getenv("GITHUB_REPOSITORY", None)
paramRef = getenv("GITHUB_REF", None)
paramSHA = getenv("GITHUB_SHA", None)
if "INPUT_FILES" in environ:
args = environ["INPUT_FILES"].split()
if len(argv) > 1: def GetListOfArtifacts(argv, files):
args = args + argv[1:] print("· Get list of artifacts to be uploaded")
args = files if files is not None else []
if len(args) == 1 and args[0] == "none": if len(argv) > 1:
files = [] args += argv[1:]
print("! Skipping 'files' because it's set to 'none") if len(args) == 1 and args[0].lower() == "none":
elif len(args) == 0: print("! Skipping 'files' because it's set to 'none")
stdout.flush() return []
raise (Exception("Glob patterns need to be provided as positional arguments or through envvar 'INPUT_FILES'!")) elif len(args) == 0:
else:
for item in args:
print(f" glob({item!s}):")
for fname in [fname for fname in glob(item, recursive=True) if not Path(fname).is_dir()]:
if Path(fname).stat().st_size == 0:
print(f" - ! Skipping empty file {fname!s}")
continue
print(f" - {fname!s}")
files.append(fname)
if len(files) < 1:
stdout.flush() stdout.flush()
raise (Exception("Empty list of files to upload/update!")) raise (Exception("Glob patterns need to be provided as positional arguments or through envvar 'INPUT_FILES'!"))
else:
flist = []
for item in args:
print(f" glob({item!s}):")
for fname in [fname for fname in glob(item, recursive=True) if not Path(fname).is_dir()]:
if Path(fname).stat().st_size == 0:
print(f" - ! Skipping empty file {fname!s}")
continue
print(f" - {fname!s}")
flist.append(fname)
if len(flist) < 1:
stdout.flush()
raise (Exception("Empty list of files to upload/update!"))
return flist
print("· Get GitHub API handler (authenticate)")
if "GITHUB_TOKEN" in environ: def GetGitHubAPIHandler(token):
gh = Github(environ["GITHUB_TOKEN"]) print("· Get GitHub API handler (authenticate)")
elif "INPUT_TOKEN" in environ: if token is not None:
gh = Github(environ["INPUT_TOKEN"]) return Github(token)
else: raise (Exception("Need credentials to authenticate! Please, provide 'GITHUB_TOKEN' or 'INPUT_TOKEN'"))
if "GITHUB_USER" not in environ or "GITHUB_PASS" not in environ:
def CheckRefSemVer(gh_ref, tag, snapshots):
print("· Check SemVer compliance of the reference/tag")
env_tag = None
if gh_ref[0:10] == "refs/tags/":
env_tag = gh_ref[10:]
if env_tag != tag:
rexp = r"^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$"
semver = re.search(rexp, env_tag)
if semver == None and env_tag[0] == "v":
semver = re.search(rexp, env_tag[1:])
tag = env_tag
if semver == None:
print(f"! Could not get semver from {gh_ref!s}")
print(f"! Treat tag '{tag!s}' as a release")
return (tag, env_tag, False)
else:
if semver.group("prerelease") is None:
# is a regular semver compilant tag
return (tag, env_tag, False)
elif snapshots:
# is semver compilant prerelease tag, thus a snapshot (we skip it)
print("! Skipping snapshot prerelease")
sys_exit()
return (tag, env_tag, True)
def GetRepositoryHandler(gh, repo):
print("· Get Repository handler")
if repo is None:
stdout.flush() stdout.flush()
raise ( raise (Exception("Repository name not defined! Please set 'GITHUB_REPOSITORY"))
Exception( return gh.get_repo(repo)
"Need credentials to authenticate! Please, provide 'GITHUB_TOKEN', 'INPUT_TOKEN', or 'GITHUB_USER' and 'GITHUB_PASS'"
def GetOrCreateRelease(gh_repo, tag, sha, is_prerelease):
print("· Get Release handler")
gh_tag = None
try:
gh_tag = gh_repo.get_git_ref(f"tags/{tag!s}")
except Exception:
stdout.flush()
if gh_tag:
try:
return (gh_repo.get_release(tag), False)
except Exception:
return (gh_repo.create_git_release(tag, tag, "", draft=True, prerelease=is_prerelease), True)
else:
err_msg = f"Tag/release '{tag!s}' does not exist and could not create it!"
if sha is None:
raise (Exception(err_msg))
try:
return (
gh_repo.create_git_tag_and_release(
tag, "", tag, "", sha, "commit", draft=True, prerelease=is_prerelease
),
True,
) )
except Exception:
raise (Exception(err_msg))
def UpdateReference(gh_release, tag, sha, is_prerelease, is_draft):
print("· Update Release reference (force-push tag)")
if is_draft:
# Unfortunately, it seems not possible to update fields 'created_at' or 'published_at'.
print(" > Update (pre-)release")
gh_release.update_release(
gh_release.title,
"" if gh_release.body is None else gh_release.body,
draft=False,
prerelease=is_prerelease,
tag_name=gh_release.tag_name,
target_commitish=gh_release.target_commitish,
) )
gh = Github(environ["GITHUB_USER"], environ["GITHUB_PASS"])
print("· Get Repository handler") if sha is not None:
print(f" > Force-push '{tag!s}' to {sha!s}")
if "GITHUB_REPOSITORY" not in environ: gh_repo.get_git_ref(f"tags/{tag!s}").edit(sha)
stdout.flush()
raise (Exception("Repository name not defined! Please set 'GITHUB_REPOSITORY"))
gh_repo = gh.get_repo(environ["GITHUB_REPOSITORY"])
print("· Get Release handler")
tag = getenv("INPUT_TAG", "tip")
env_tag = None
gh_ref = environ["GITHUB_REF"]
is_prerelease = True
is_draft = False
if gh_ref[0:10] == "refs/tags/":
env_tag = gh_ref[10:]
if env_tag != tag:
rexp = r"^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$"
semver = re.search(rexp, env_tag)
if semver == None and env_tag[0] == "v":
semver = re.search(rexp, env_tag[1:])
tag = env_tag
if semver == None:
print(f"! Could not get semver from {gh_ref!s}")
print(f"! Treat tag '{tag!s}' as a release")
is_prerelease = False
else:
if semver.group("prerelease") is None:
# is a regular semver compilant tag
is_prerelease = False
elif getenv("INPUT_SNAPSHOTS", "true") == "true":
# is semver compilant prerelease tag, thus a snapshot (we skip it)
print("! Skipping snapshot prerelease")
sys_exit()
gh_tag = None
try:
gh_tag = gh_repo.get_git_ref(f"tags/{tag!s}")
except Exception:
stdout.flush()
if gh_tag:
try:
gh_release = gh_repo.get_release(tag)
except Exception:
gh_release = gh_repo.create_git_release(tag, tag, "", draft=True, prerelease=is_prerelease)
is_draft = True
else:
err_msg = f"Tag/release '{tag!s}' does not exist and could not create it!"
if "GITHUB_SHA" not in environ:
raise (Exception(err_msg))
try:
gh_release = gh_repo.create_git_tag_and_release(
tag, "", tag, "", environ["GITHUB_SHA"], "commit", draft=True, prerelease=is_prerelease
)
is_draft = True
except Exception:
raise (Exception(err_msg))
print("· Cleanup and/or upload artifacts")
artifacts = files
assets = gh_release.get_assets()
def delete_asset_by_name(name): files = GetListOfArtifacts(sys_argv, paramFiles)
for asset in assets: stdout.flush()
if asset.name == name: [tag, env_tag, is_prerelease] = CheckRefSemVer(paramRef, paramTag, paramSnapshots)
asset.delete_asset() stdout.flush()
return gh_repo = GetRepositoryHandler(GetGitHubAPIHandler(paramToken), paramRepo)
stdout.flush()
[gh_release, is_draft] = GetOrCreateRelease(gh_repo, tag, paramSHA, is_prerelease)
stdout.flush()
if paramRM:
def upload_asset(artifact, name):
try:
return gh_release.upload_asset(artifact, name=name)
except GithubException as ex:
if "already_exists" in [err["code"] for err in ex.data["errors"]]:
print(f" - {name} exists already! deleting...")
delete_asset_by_name(name)
else:
print(f" - uploading failed: {ex}")
except Exception as ex:
print(f" - uploading failed: {ex}")
print(f" - retry uploading {name}...")
return gh_release.upload_asset(artifact, name=name)
def replace_asset(artifacts, asset):
print(f" > {asset!s}\n {asset.name!s}:")
for artifact in artifacts:
aname = str(Path(artifact).name)
if asset.name == aname:
print(f" - uploading tmp.{aname!s}...")
new_asset = upload_asset(artifact, name=f"tmp.{aname!s}")
print(f" - removing...{aname!s}")
asset.delete_asset()
print(f" - renaming tmp.{aname!s} to {aname!s}...")
new_asset.update_asset(aname, label=aname)
artifacts.remove(artifact)
return
print(" - keep")
if getenv("INPUT_RM", "false") == "true":
print("· RM set. All previous assets are being cleared...") print("· RM set. All previous assets are being cleared...")
for asset in assets: for asset in gh_release.get_assets():
print(f" - {asset.name}") print(f" - {asset.name}")
asset.delete_asset() asset.delete_asset()
else:
for asset in assets:
replace_asset(artifacts, asset)
for artifact in artifacts:
print(f" > {artifact!s}:\n - uploading...")
gh_release.upload_asset(artifact)
stdout.flush() stdout.flush()
print("· Update Release reference (force-push tag)")
if is_draft: print("· Cleanup and/or upload artifacts")
# Unfortunately, it seems not possible to update fields 'created_at' or 'published_at'. env = environ.copy()
print(" > Update (pre-)release") env["GITHUB_TOKEN"] = paramToken
gh_release.update_release( cmd = ["gh", "release", "upload", "--repo", paramRepo, "--clobber", tag] + files
gh_release.title, print(f" > {' '.join(cmd)}")
"" if gh_release.body is None else gh_release.body, check_call(cmd, env=env)
draft=False, stdout.flush()
prerelease=is_prerelease,
tag_name=gh_release.tag_name,
target_commitish=gh_release.target_commitish,
)
if ("GITHUB_SHA" in environ) and (env_tag is None): UpdateReference(gh_release, tag, paramSHA if env_tag is None else None, is_prerelease, is_draft)
sha = environ["GITHUB_SHA"]
print(f" > Force-push '{tag!s}' to {sha!s}")
gh_repo.get_git_ref(f"tags/{tag!s}").edit(sha)