--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@0a3b1406

Confluence: get only those build jobs with a corresponding successful deploy job

We cannot handle build jobs and deploy jobs similarly, because all build
jobs are successful usually, while not all deploy jobs necessarily have
run. Get build.env only from those build job which have a corresponding
successful deploy job, otherwise MACHINE and SUPPORTED_HARDWARE fields
always contain the full list of built platforms, even if they might have
not been deployed.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@97228cdb

Move MACHINE variable from deploy.env to build.env

The Confluence stage is the only stage that needs the MACHINE variable.
Since the collect_release_information script already loads the build.env
from the build jobs, there is no need to store this variable in
deploy.env.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ba156b0f

Confluence: check if variables are defined

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@cae6e282

Confluence: improve variable passing to template

Use a general mechanism of storing all job variables for each build job
separately, so that we have all documentation files in the
FILES_documentation variable. This makes the dedicated HTML_FILES
variable obsolete.
The merged variables are still kept, though, so that we have all
machines in the MACHINE variable to loop over.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@41f8a49f

Confluence: add error handling if artifact not found

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@1a519057

Make documentation files configurable

This disables the documentation jobs for SDK and FNGSystem.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@303e24eb

Build documentation in dedicated job

Previously the documentation files were rendered as part of the package
jobs. Each package job converted the same set of static markdown files
to HTML files, which was redundant.

Add dedicated jobs to build, package and deploy the documentation files.

Modify convert_md2html.py to support file globs as arguments.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@838da585

Refactoring: move Alphaplan stage code to the end

This reflects the stage order in the code.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ff31d083

Refactoring: use multiple MACHINE loops for better structure

This is a preparation for adding dedicated documentation jobs. In order
to have these grouped by stage along with their corresponding machine
jobs, we have to get rid of the giant outer loop and add loops for each
stage instead.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@69973c95

Deploy: save deployed files in dotenv variable instead of dedicated file

Previously the deployed files were stored in a dedicated text file
called files.txt. Move this information to a dotenv variable called
FILES, so that we don't need special handling for it in the Confluence
stage.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@45d04033

Remove disabling sourcery low-code-quality in package_release

This check is not failing anymore since the latest refactorings.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@cf0336d6

Define manifest file to build from

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@a7503809

Confluence: add supported hardware information

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@0ef32ce7

Update simulated build to kirkstone-14.0

Remove temporary workarounds that were necessary due to changes after
the previously used release kirkstone-7.0.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@64dc51d3

Use regex operator to check for multiple variables

The "&&" operator does not work as expected when checking if multiple
variables are set.

See for reference:
https://gitlab.com/gitlab-org/gitlab/-/issues/321371

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@89fbce24

Hide Confluence stage if confluence variables not set

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@db50e93b

Hide deploy stage if DEPLOY_RELEASE_TARGET not set

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f4aa0048

Hide FTP stage if DEPLOY_FTP_TARGET not set

This makes the variable HIDE_FTP_UPLOAD_STAGE obsolete, because we can
hide the FTP stage now by setting DEPLOY_FTP_TARGET to an empty string.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@409c4f27

Do not run CI test sub-pipeline on master

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@a731b156

Remove DISTRO_IMAGES

This variable has been removed in Yocto. Its contents have been merged
into DISTRO_RELEASE_ARTEFACTS.

See for reference:
seco-ne/yocto/layers/meta-seconorth-distro@a316e95e

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@45f2aca7

Refactoring: remove parsing of testdata.json in package stage

Extract and save these variables in the build stage instead and
explicitly pass them as arguments to the package script.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8ca2487d

Move .simulate_build below .build_yocto

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@2808f067

Fix build.env contents for simulated build

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@73d00715

Pretty print release metadata JSON file

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f2a29f53

Refactoring: remove RELEASE_NAME from deploy targets

Append it to the configured target at the deploy class level instead.
This removes the need for eval'ing the variables before usage.
RELEASE_NAME can be used directly at the class level because it is known
from build-version.env.

This makes the deploy behavior identical to the Azure stage.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@e136e5e2

Remove top-level "release" folder from package

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@1d5e479f

Define MACHINE variable on job level

Instead of passing MACHINE from stage to stage or loading it from
testdata.json, use the original value from the Jinja2 loop and set it
directly for each job where it is used.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@cdd47023

Introduce separate variables to set RELEASE_NAME and RELEASE_VERSION

The expressions to calculate RELEASE_NAME and RELEASE_VERSION cannot be
stored within these variables themseselves. If set on the trigger level,
they would override the calculated values in build-version.env then,
as trigger variables always have the highest precedence.

Use separate variables RELEASE_VERSION_EXPRESSION and
RELEASE_NAME_EXPRESSION to define how RELEASE_VERSION and RELEASE_NAME
are calculated instead.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@372d192b

Use RELEASE_VERSION and RELEASE_NAME from build-version job

Instead of passing these variables from stage to stage or regenerating
their values in later stages, use the ones set in the build-version job
at all places.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@cad30c20

Remove RELEASE_SUFFIX variable

The RELEASE_NAME variable can be set directly now, so there's no need to
have a dedicated RELEASE_SUFFIX anymore. A previous configuration like

  RELEASE_SUFFIX: "-custom"

can now be achieved using:

  RELEASE_NAME: "Yocto-${RELEASE_VERSION}-custom"

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@922f49f8

Add yocto version job

This adds a machine-independent job "build-version" which populates the
RELEASE_VERSION and RELEASE_NAME variables, so that following jobs can
use these without depending on the various build jobs.
The variables can be set from the trigger job in a project's
`.gitlab-ci.yml` file. They are eval'ed before saving them to
version.env, so we can use deferred variable expansion or even command
execution to construct their values. This mechanism is already used for
the Flash-N-Go System variables.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@073ae81e

LAVA: rename MACHINE to LAVA_MACHINE

Rename variable in order to avoid confusion with the original MACHINE
variable used everywhere else. Also rename the local Jinja2 variable to
include an underscore to make naming consistent.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@a91baa55

Clean build.env before writing

This had been accidentally removed in
5e36715ef6cf98df4c1b98fedddc0c3c50ed4040.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@1860ca8e

Remove LOG_PREFIX variable

This was used in times when the same code was executed from different
places. We don't do that anymore, so the variable is obsolete.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@3d2dbe4d

Documentation: update job generation chapter

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@17b87f84

Refactoring: remove CI_PARAM_ prefix from variables

There is no need to add a prefix to the variables. Basically every
variable can be set or overridden at the trigger level, so we just use
plain variable names for everything.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@825f3686

Refactoring: do not rename *_PATH variables in build.env

We were using different variable names for the same variable in build
and package stages. Unify these names in order to reduce potential
confusion.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@2346bb98

Refactoring: make build variable naming consistent

Use underscore to separate words within variable name.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@581541b9

Refactoring: improve FNG_INSTALL_URL definition

FNG_INSTALL_URL was defined in build.env only, and in order to use it
during the build we sourced build.env after writing it.
Remove this workaround and define the variable cleanly in the first
place. This removes the intermediate variable FNG_INSTALL_PATH.
Also move the "/artifact" path from JOB_URL to FNG_INSTALL_URL in order
to make the JOB_URL variable name match better.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@35784f29

Refactoring: improve IMAGEPATH et al. definition

On every usage of IMAGEPATH, SDKPATH and LICENSESPATH we prepended
BUILDPATH to it. Prepend it directly on definition to prevent
repetition.
Also remove redundant variable IMAGEBASEPATH.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@5e36715e

Remove needless variables from build.env

There were three variables set in build.env of which the values are
already known in the environment for all jobs, because they are passed
from the parent pipeline. Use the original variables instead of defining
new ones.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@84aa6afb

Fix LAVA job name

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8160c7d5

Sourcery: enable "low-code-quality" check globally

Disable it locally instead, so that
 1. new code additions are always checked, and
 2. we can refactor one function at a time to make the check pass.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ef60c378

Sourcery: keep "for-append-to-extend" check disabled

This is another check that suggests changes an average developer without
profound Python experience would find hard to understand. Disable it.

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/for-append-to-extend/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@cf371aca

alphaplan_fwr: Remove obsolete pkg.py artifact

The pkg.py file is no longer deployed by the build process.

See commit
seco-ne/yocto/layers/meta-seconorth-distro@e79e8579

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@b2b43f71

Sourcery: keep "use-named-expression" check disabled

This suggestion aims to reduce code size, but IMO for the expense of
readability for non-python-experts, so we disable it.

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/use-named-expression/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ae8c5423

Sourcery: keep "use-next" check disabled

I don't see any benefit in accepting the refactoring suggestions based
on this rule. IMO it makes the code harder to read, especially for non-
experienced Python developers.

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/use-next/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@9462f826

Refactoring: raise specific errors

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/raise-specific-error/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@61eb3e11

Switch all strings to formatted string literals (f-strings)

The old printf-style string formatting syntax using '%' is not
recommended anymore. See for reference:
https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting
https://docs.sourcery.ai/Reference/Python/Default-Rules/replace-interpolation-with-fstring/

Python 3.6 introduced f-strings as a better alternative:
https://docs.python.org/3/reference/lexical_analysis.html#f-strings

Consequently switch to this new syntax for all strings.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@b2fb3e93

Combine build-yocto.yml and and build-common.yml to build-pipeline.yml

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@e6808701

Add "Deploy Azure" stage

This stage contains jobs to deploy packaged artifacts to our Azure blob
storage. The required variables `AZURE_STORAGE_ACCOUNT` and
`AZURE_STORAGE_SAS_TOKEN` are stored in the GitLab CI/CD variables.
The storage container `AZURE_CONTAINER_NAME` and artifact path
`AZURE_TARGET_FOLDER` are passed from the parent pipeline.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f8c1d732

Add confluence stage

This adds two jobs to the Yocto pipeline:

 - generate-confluence-page
 - publish-confluence-page

The first one generates a confluence page from a template
(`confluence-page.xml.jinja2`) using information downloaded from all
successful "Deploy FTP" jobs of the same pipeline.

The second one publishes this page in Confluence and displays a link to
the new page. If the page already exists it is overwritten.

The place in Confluence where the new page is published is configurable
via the `CONFLUENCE_SPACE` and `CONFLUENCE_PARENT_ID` variables.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@fee9493b

generate-build-pipeline: re-add YAML format parsing

This was accidentally forgotten in the previous commit.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@9b777f29

Rename "generate_job_from_template" script to "render_jinja2_template"

This makes the script more general and open for other future purposes.
The parsing of the YAML content is conditional now, depending on the new
"format" argument.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@b20f6b95

Use multiple CI test pipelines

Instead of mixing dedicated CI test jobs and Yocto build simulation jobs
within one pipeline, use the new multi-build-pipelines architecture to
split them up into separate child pipelines.
This also makes most of the Yocto pipeline code reusable, so that we
don't have to declare all the jobs again in ci-test.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f9dc2517

Rename "build jobs" to "build pipeline"

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@633c9fb4

get_pipeline_jobs: remove redundant job ID output

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@347745c5

download_job_artifacts: add error message if artifact not found

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@50f7396d

ci-test: re-enable build simulation caching

In 438f1e2aae23472d692ce6314e8c683f950657d2 we accidentally disabled
caching for the build simulation by moving the cache definition to a job
class. The order of inheritance led to the cache configuration being
overwritten by the one of the "buildbase" class. Change inheritance
order to fix this.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@30f7d966

update_submodule: fix integration branch name in case of multiple MRs

When there are multiple open merge requests for a commit, we currently
take the newest one to determine the integration branch name. This is
not necessarily the correct one, we can have an open merge request with
the commit being part of the commit history while the top commit is a
different one.

Add a check if the commit is the top commit of the merge request in
order to get the correct integration branch name in these cases.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@438f1e2a

ci-test: use more recent release for build simulation

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@1c900bf9

Update docs/add-new-project-to-pipeline.md
--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8854fd8f

download_job_artifacts: fix temp file creation

mkstemp() returns a tuple containing an OS-level handle to an open file
(as would be returned by os.open()) and the absolute pathname of that
file, in that order.

Use higher-level function NamedTemporaryFile() instead.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@151f1cd6

Refactoring: remove redundant if

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/remove-redundant-if/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@872ec6d9

Refactoring: swap if and else branches

Move the main part of the action to the if branch.

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/swap-if-else-branches/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@c058236c

Sourcery: skip a certain suggestion for better readability

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@b9196ae5

Fix typo in comment

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@eba0649d

gitignore: add .vscode and convert line-endings to LF

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@876da56d

Refactoring: merge dictionary assignments

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/merge-dict-assign/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@e8e3ff37

Refactoring: use single assignment for "if, else"

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/assign-if-exp/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@14eaf45a

Refactoring: inline immediately returned variables

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/inline-immediately-returned-variable/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f9096a52

Refactoring: use "{}" for creating an empty dictionary

This is the most concise and Pythonic way to create a dictionary.

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/dict-literal/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@a3baa39e

Refactoring: use list comprehensions

A list comprehension can create the list on one line, cutting out the
clutter of declaring an empty list and then appending values.

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/list-comprehension/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@52de8d7c

Refactoring: use dictionary comprehensions

A dictionary comprehension can create the dictionary on one line,
cutting out the clutter of declaring an empty dict and then adding
items.

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/dict-comprehension/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@5d09b9de

Refactoring: simplify length comparisons

Something we often do is check whether a list or sequence has elements
before we try and do something with it.

A Pythonic way of doing this is just to use the fact that Python lists
and sequences evaluate to True if they have elements, and False
otherwise.

Doing it this way is a convention, set out in Python's PEP8 style guide.

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/simplify-len-comparison/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@9406ad75

Yocto build: separate images in multiple pipelines

Instead of building the Yocto image, the Yocto SDK and the FNGSystem
image in one single pipeline, separate them into three independent
pipelines that are triggered in parallel.

This makes the concept more modular: we have a single general pipeline
now which is configurable from outside via variables. Hence we can have
a custom number of pipelines along with custom build targets in each
project without having to make code changes in the gitlab-ci project.

The default Yocto manifest pipeline configures three build pipelines:

- yocto-build-jobs
- sdk-build-jobs
- fngsystem-build-jobs

In a project including the Yocto manifest pipeline, we can disable
certain build pipelines using job rules, e.g. disabling the SDK build:

sdk-build-jobs:
  rules:
    - when: never

Furthermore we can add more pipelines by simply adding jobs extending
the '.build-jobs' class in the project's .gitlab-ci.yml:

yocto-custom-build-jobs:
  extends:
    - .build-jobs
  variables:
    BITBAKE_TASK: build
    CI_PARAM_IMAGE: custom-image
    CI_PARAM_DISTRO: custom-distro
    PACKAGE_TYPE: image

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@447804d2

.gitlab-ci: increase analyze timeout

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@5762a54c

Yocto build: unify image and SDK package jobs

Image and SDK package jobs call the same package script just with
different arguments. Instead of having two job classes `package_release`
and `package_sdk` for these two tasks, merge them into the base class
`package` and make the differences configurable via a variable
`PACKAGE_TYPE`.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8e72eac2

Yocto build: add variable for manual builds

Instead of hard-coding the rules for manual builds in each actual job,
conditionally add this to the `buildbase` class and add a variable
`MANUAL_BUILD` to the according jobs.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@e6d71996

Yocto build: unify image and SDK build jobs

Image and SDK builds share a lot of similar code. Instead of having two
job classes `build_yocto_image` and `build_yocto_sdk` for these two
tasks, merge them into the base class `build_yocto` and make the
differences configurable via a variable.

The `dump_install_command` part of the script, which was not executed
for SDK builds, is always present now, but it's only executed if the
`INSTALLSCRIPT` variable is set, which is not the case for SDK builds.

The `collect_srcrevs` part of the script is executed in all cases. It
was not part of the SDK build before, but it's not less relevant there.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f892500f

Yocto build: make main artifacts path configurable

Instead of specifying all possible artifacts paths and abusing the fact
that GitLab ignores non-existing paths during artifact upload, implement
a cleaner solution with a configurable path.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@c227e053

Yocto build: use common artifact pathspec for image and SDK builds

This is the first step on the way to a common pipeline for all build
configurations.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@fa2766e6

README: review documentation for adding new project

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@c027f42b

README: use consistent heading style

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f8bf107b

Sourcery: keep min-max-identity check disabled

IMO code is easier to read without the proposed change:
https://docs.sourcery.ai/Reference/Python/Default-Rules/min-max-identity/

Move the disabled check to the top of the list and add a comment that
all the checks below need to be verified.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@01484573

Refactoring: replace if-expression with or

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/or-if-exp-identity/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ccdac3d5

Sourcery: enable more passing checks

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@d5908ee5

Refactoring: remove needless str() from print()

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/remove-str-from-print/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@479861fb

Sourcery: enable passing checks

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@7c87d4d1

Sourcery: skip use-assigned-variable

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@26591c0b

Refactoring: use datetime.now() instead of datetime.today()

now() is officially preferred over today() according to the docs.
See for reference:

https://docs.sourcery.ai/Reference/Python/Default-Rules/use-datetime-now-not-today/
https://docs.python.org/3/library/datetime.html#datetime.datetime.now

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@414ebe6d

Add python code analysis via sourcery

Disable all checks which do not pass currently. These should be enabled
in the future along with a code refactoring to make them pass.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@df817ecf

README: Add howto to create a new project.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@31e5a427

CI: package: Increase timeout to 90

We had timeouts during the upload of the artifacts
so I increase it to 90 minutes.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@23680f48

Fix check for branch protection state

Since the changes of 939ade2b9f98b3e1569332270a7ab16ce7d70b4a we observe
that pipelines are executed on non-protected branches, which is not what
we want.

GitLab's documentation was a bit unclear on the according CI variables
CI_COMMIT_REF_PROTECTED and CI_MERGE_REQUEST_TARGET_BRANCH_PROTECTED, as
stated in this stack overflow answer:
https://stackoverflow.com/a/59023344/3018229

We need to explicitly check for the variable values to be "true" instead
of just checking whether they are set.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@939ade2b

Depend intgration pipeline execution on branch protection state

Previously we had the master branch hard-coded in a variable in order to
determine whether to run the integration pipeline (on non-master branch)
or the merge pipeline (on master branch). This variable had to be
adapted everytime we copy a branch, which can potentially be forgotten
and lead to problems.

Remove the hard-coded branch and use the branch protection state for
this decision instead in order to make this more flexible.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@dbc49ad0

Simplify yamllint job rules

The rules for the yamllint jobs have been defined in the .yamllint class
before. They were overridden in the manifest pipeline's yamllint job,
though, which has its own rules, so we don't need them to be inherited
in this case. And in the manifest integration pipeline the rules can be
much easier, so define them there instead of at the class-level.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@cd8de4ff

.gitlab-ci: add analyze stage for limiting YAML script blocks

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@e943a55e

Use project path instead of name in integration branch name

Usually project name and path are identical after project creation. It
is possible, though, to change the name of a project without changing
its path.
When triggering a pipeline on the integration branch, we are using the
CI_PROJECT_NAME variable to reference the project. Unexpectedly, this
variable contains the *path*, not the *name* of the project (see
https://gitlab.com/gitlab-org/gitlab/-/issues/20030).
Hence we have to use the project path in all other places where the
integration branch name is constructed in our Python code.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@bd7025b8

Manifest: run automatic pipeline on tags

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@068749c7

Manifest build: add support for SSH_CONFIG variable

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@d67898a2

Manifest: skip build if "skip build" label is set

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@bffc6097

Manifest: switch from branch pipelines to merge request pipelines

This will make the manifest project support the "skip build" label.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@5ba1ac68

Revert "Skip build if "skip build" label is set"

This reverts commit 4db257d36052c74eabed7ef43a05d98d3998ffec.

It does not work on the manifest, because we do not run merge request
pipelines there, so CI_MERGE_REQUEST_LABELS is not set at all.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@4db257d3

Skip build if "skip build" label is set

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@9d430fe8

Manifest integration: do not cancel previous pipelines on master

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8b08a083

Fix error handling on merge

GitLab API has changed:
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/94164/diffs#diff-content-c201dae00b72426693ddcb40edf6005e7702e935

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@a59a93a0

Package: Add needs to changelog, to package changelog also

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8edbf2a6

Changelog generator: fix typo

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@e6ae06dd

Changelog generator: pass projects as arguments

Instead of hard-coding the project list in the changelog generator
script, make them passable via arguments. This way we can have different
changelogs in different envrionments. Hence we're adding the changelog
job in the ci-test environment as well.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ed36b465

mirror_mr_pipeline: include pipeline-not-found message in output

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@5ee69ab9

mirror_mr_pipeline: fix error when no pipeline found

This case should trigger a new pipeline instead of reporting an error.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@e69d303e

Changelog generator: remove debug output

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@0264fe09

Changelog generator: get all merge requests

This fixes older releases not appearing in the changelog.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@c657f31f

mirror_mr_pipeline: improve debug output

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@473f12b1

mirror_mr_pipeline: add main call

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@88708b64

Convert CRLF to LF

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@169b2fa7

Improve mirroring MR pipeline

Refactor three scripts into one in order to support triggering a new
pipeline in the case of a skipped MR pipeline. This was not possible to
implement with the previous three-script design. The single scripts
were not used in other places anyway.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@897ba059

Manifest integration: remove dynamic job generation

The dynamic job was only necessary because GitLab did not expand
variables in the "trigger:project" keyword:
https://gitlab.com/gitlab-org/gitlab/-/issues/347469

This has been fixed in GitLab 15.3, so we don't need it anymore:
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/92346

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ea1f3bad

cancel_pipelines: include downstream pipelines

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@c8f97623

Add job to cancel outdated pipelines in merge requests

Everytime we push a new commit to a branch, a new pipeline is created.
Often the pipeline for the previous commit has not finished yet, and
GitLab does not automatically cancel it. It consumes valuable build
time, even if we don't need the results anymore.
Adding a MR pipeline job to search for previous pipelines on the same
branch and cancel them explicitly.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@bab6f811

Package: move SDK files to subfolder "sdk"

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8bcaa426

scripts: Add delete_job_artifacts script to delete all artifacts from untagged jobs

For a given project all artifacts may be deleted. This was needed due to
space limitations on the gitlab server.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@28906544

Reduce artifact expiration of build jobs to 1 week

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@d404f257

Alphaplan: allow failure on import

Otherwise the parent pipeline is stuck in "running" state until the
manual action has been performed. See for reference:
https://gitlab.com/gitlab-org/gitlab/-/issues/361574

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@dd2a6d7d

Fix error message

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@04ac29ad

CI-Test: fix deploy-files job

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@22c519f1

Move FTP jobs to FTP runner

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@84d0616b

Add new deploy pipeline to Yocto

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f0436868

Move reusable parts from CI-Test to common

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@3d3b4d09

Yocto: fix SDK package

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@37360288

Yocto: fix SDK build

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@2b6fc739

Package: add comment

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@95d6e368

alphaplan_fwr_import: add error handling

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@277d4af7

Package & deploy: do not allow missing dotenv file

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@3c68ccab

Alphaplan: add script for importing alphaplan data

The script has mostly been copied from alphaplan_fwr.py, but with
different input data.

Adding a job to the CI-test environment for testing this.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@3b3238a1

Alphaplan: add script for generating alphaplan data

This script has mostly been copied from alphaplan_fwr.py, but with
different input data (artifacts from the SoftwareStore deploy job) and
without actually importing the data to the Alphaplan database.

Adding a job to the CI-test environment for testing this.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@387b8184

Deploy: save variables in deploy.env

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@902c9276

Deploy: save rsync file list for later stages

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@15622149

CI-test: use caching between the pipeline stages

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@a96a923e

CI-test: add deploy stages for SoftwareStore and FTP

Rename all stages to uppercase on this occasion, since we're adding
stages with names containing multiple words.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@056f1af1

Add package class and stage

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@a1972724

Add script to query job logs filtered by status, tag, and log pattern

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@d501fed4

Update isort config

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@9982711f

Move deploy jobs to deploy runner

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@e48b4c0b

package_release: Do not fail if a file does not exist

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@a5e27f6e

FTP upload: Add flag to hide FTP upload stage

This is a workaround to prevent customer releases being uploaded to our
public FTP area. It should be removed as soon as we support uploading to
different FTP target folders.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@543a2170

manifest_package: add release suffix

Customer specific manifest projects can now set the RELEASE_SUFFIX
variable, which then gets appended to the deployment folder names.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@c763c43f

alphaplan_fwr: change variable "release_name_local" to "release_name"

Otherwise it can be assumed that it might be correlated with
outputdir_local, which is not the case.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@2d8d26a5

package_release: more refactoring

- Make output-dir an argument that can be passed multiple times instead
  of hard-coding two dirs with certain meanings.
- Remove LOCALDIR variable from package.env, because it is not used
  anywhere (only occurence was job class ".uploadsdkftp" which wasn't
  used anywhere either).

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@dc3698c8

package_release: refactoring

Improve readability of package_release.py and its submodule
generate_release_metadata.py.

- Add describing comments throughout the file.
- Remove duplicate code from functions. Simplify them, so that they do
  what their names say (and nothing more).
- Rename variables and function arguments to make their purpose clearer.
- Harmonize variable names with their corresponding command line arguments.
- Add pydoc for functions.
- Add type declarations to function arguments.
- Generate MD5 and metadata files in the source directories so that they
  can be copied along with the rest of the files.