GitBot authored 1 year ago and Tobias Kahlki committed 1 year ago

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@64dc51d3

Use regex operator to check for multiple variables

The "&&" operator does not work as expected when checking if multiple
variables are set.

See for reference:
https://gitlab.com/gitlab-org/gitlab/-/issues/321371

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@89fbce24

Hide Confluence stage if confluence variables not set

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@db50e93b

Hide deploy stage if DEPLOY_RELEASE_TARGET not set

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f4aa0048

Hide FTP stage if DEPLOY_FTP_TARGET not set

This makes the variable HIDE_FTP_UPLOAD_STAGE obsolete, because we can
hide the FTP stage now by setting DEPLOY_FTP_TARGET to an empty string.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@409c4f27

Do not run CI test sub-pipeline on master

GitBot authored 1 year ago and Tobias Kahlki committed 1 year ago

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@a731b156

Remove DISTRO_IMAGES

This variable has been removed in Yocto. Its contents have been merged
into DISTRO_RELEASE_ARTEFACTS.

See for reference:
meta-seconorth-distro@a316e95e

GitBot authored 1 year ago and Tobias Kahlki committed 1 year ago

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@45f2aca7

Refactoring: remove parsing of testdata.json in package stage

Extract and save these variables in the build stage instead and
explicitly pass them as arguments to the package script.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8ca2487d

Move .simulate_build below .build_yocto

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@2808f067

Fix build.env contents for simulated build

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@73d00715

Pretty print release metadata JSON file

GitBot authored 1 year ago and Tobias Kahlki committed 1 year ago

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f2a29f53

Refactoring: remove RELEASE_NAME from deploy targets

Append it to the configured target at the deploy class level instead.
This removes the need for eval'ing the variables before usage.
RELEASE_NAME can be used directly at the class level because it is known
from build-version.env.

This makes the deploy behavior identical to the Azure stage.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@e136e5e2

Remove top-level "release" folder from package

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@1d5e479f

Define MACHINE variable on job level

Instead of passing MACHINE from stage to stage or loading it from
testdata.json, use the original value from the Jinja2 loop and set it
directly for each job where it is used.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@cdd47023

Introduce separate variables to set RELEASE_NAME and RELEASE_VERSION

The expressions to calculate RELEASE_NAME and RELEASE_VERSION cannot be
stored within these variables themseselves. If set on the trigger level,
they would override the calculated values in build-version.env then,
as trigger variables always have the highest precedence.

Use separate variables RELEASE_VERSION_EXPRESSION and
RELEASE_NAME_EXPRESSION to define how RELEASE_VERSION and RELEASE_NAME
are calculated instead.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@372d192b

Use RELEASE_VERSION and RELEASE_NAME from build-version job

Instead of passing these variables from stage to stage or regenerating
their values in later stages, use the ones set in the build-version job
at all places.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@cad30c20

Remove RELEASE_SUFFIX variable

The RELEASE_NAME variable can be set directly now, so there's no need to
have a dedicated RELEASE_SUFFIX anymore. A previous configuration like

  RELEASE_SUFFIX: "-custom"

can now be achieved using:

  RELEASE_NAME: "Yocto-${RELEASE_VERSION}-custom"

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@922f49f8

Add yocto version job

This adds a machine-independent job "build-version" which populates the
RELEASE_VERSION and RELEASE_NAME variables, so that following jobs can
use these without depending on the various build jobs.
The variables can be set from the trigger job in a project's
`.gitlab-ci.yml` file. They are eval'ed before saving them to
version.env, so we can use deferred variable expansion or even command
execution to construct their values. This mechanism is already used for
the Flash-N-Go System variables.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@073ae81e

LAVA: rename MACHINE to LAVA_MACHINE

Rename variable in order to avoid confusion with the original MACHINE
variable used everywhere else. Also rename the local Jinja2 variable to
include an underscore to make naming consistent.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@a91baa55

Clean build.env before writing

This had been accidentally removed in
5e36715ef6cf98df4c1b98fedddc0c3c50ed4040.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@1860ca8e

Remove LOG_PREFIX variable

This was used in times when the same code was executed from different
places. We don't do that anymore, so the variable is obsolete.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@3d2dbe4d

Documentation: update job generation chapter

Felix Gerking authored 1 year ago and Tobias Kahlki committed 1 year ago

The VPU firmware binary for the imx6dl SoCs was not present in the
image. Adding the mx6dl override adds the firmware-imx-vpu-imx6d package
to the image.

Felix Gerking authored 1 year ago and Tobias Kahlki committed 1 year ago

Perform the do_install task only for the seco-mx6 machines, otherwise the
firmware binaries conflict with the files from the firmware-imx recipe
of the Freescale layer.

The Flash-N-Go System bootscript should also check for an additional
custom bootscript and run it if one was found.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@17b87f84

Refactoring: remove CI_PARAM_ prefix from variables

There is no need to add a prefix to the variables. Basically every
variable can be set or overridden at the trigger level, so we just use
plain variable names for everything.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@825f3686

Refactoring: do not rename *_PATH variables in build.env

We were using different variable names for the same variable in build
and package stages. Unify these names in order to reduce potential
confusion.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@2346bb98

Refactoring: make build variable naming consistent

Use underscore to separate words within variable name.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@581541b9

Refactoring: improve FNG_INSTALL_URL definition

FNG_INSTALL_URL was defined in build.env only, and in order to use it
during the build we sourced build.env after writing it.
Remove this workaround and define the variable cleanly in the first
place. This removes the intermediate variable FNG_INSTALL_PATH.
Also move the "/artifact" path from JOB_URL to FNG_INSTALL_URL in order
to make the JOB_URL variable name match better.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@35784f29

Refactoring: improve IMAGEPATH et al. definition

On every usage of IMAGEPATH, SDKPATH and LICENSESPATH we prepended
BUILDPATH to it. Prepend it directly on definition to prevent
repetition.
Also remove redundant variable IMAGEBASEPATH.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@5e36715e

Remove needless variables from build.env

There were three variables set in build.env of which the values are
already known in the environment for all jobs, because they are passed
from the parent pipeline. Use the original variables instead of defining
new ones.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@84aa6afb

Fix LAVA job name

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8160c7d5

Sourcery: enable "low-code-quality" check globally

Disable it locally instead, so that
 1. new code additions are always checked, and
 2. we can refactor one function at a time to make the check pass.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ef60c378

Sourcery: keep "for-append-to-extend" check disabled

This is another check that suggests changes an average developer without
profound Python experience would find hard to understand. Disable it.

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/for-append-to-extend/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@cf371aca

alphaplan_fwr: Remove obsolete pkg.py artifact

The pkg.py file is no longer deployed by the build process.

See commit
meta-seconorth-distro@e79e8579

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@b2b43f71

Sourcery: keep "use-named-expression" check disabled

This suggestion aims to reduce code size, but IMO for the expense of
readability for non-python-experts, so we disable it.

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/use-named-expression/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ae8c5423

Sourcery: keep "use-next" check disabled

I don't see any benefit in accepting the refactoring suggestions based
on this rule. IMO it makes the code harder to read, especially for non-
experienced Python developers.

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/use-next/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@9462f826

Refactoring: raise specific errors

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/raise-specific-error/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@61eb3e11

Switch all strings to formatted string literals (f-strings)

The old printf-style string formatting syntax using '%' is not
recommended anymore. See for reference:
https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting
https://docs.sourcery.ai/Reference/Python/Default-Rules/replace-interpolation-with-fstring/

Python 3.6 introduced f-strings as a better alternative:
https://docs.python.org/3/reference/lexical_analysis.html#f-strings

Consequently switch to this new syntax for all strings.

As the u-boot for seco-mx6-fsl does not work yet, the install script
should only be generated for seco-mx6 instead for all mx6q machines.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@b2fb3e93

Combine build-yocto.yml and and build-common.yml to build-pipeline.yml

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@e6808701

Add "Deploy Azure" stage

This stage contains jobs to deploy packaged artifacts to our Azure blob
storage. The required variables `AZURE_STORAGE_ACCOUNT` and
`AZURE_STORAGE_SAS_TOKEN` are stored in the GitLab CI/CD variables.
The storage container `AZURE_CONTAINER_NAME` and artifact path
`AZURE_TARGET_FOLDER` are passed from the parent pipeline.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

Currently yocto-install-services package only holds
seco-hmi-postinstallation.service.
Which is to be used in Edgehogs builds.
Service will check if gfxml-overlay.dtbo is present on correspondig
bootfs_X partition and will start fnginstall-postinstallation if overlay
is absent.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

Added initial SRCREV.conf with gfxml2dto commit set.
SECO-NE stores this file in manifest repo.
Edgehog stores this file in machine layers and copies it into build/conf
during initial setup.
This is needed as gfxml2dto recipe is set to use dunfell branch, which
is outdated.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

Mikhail Vanyulin authored 1 year ago and Jonas Höppner committed 1 year ago

Edgehog imx6 machines use zImage, while SECO-NE uses uImage.
So we change bootz to bootm in U-Boot boot script for Edgehog builds.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

Mikhail Vanyulin authored 1 year ago and Jonas Höppner committed 1 year ago

Added kernel to IMAGE_BOOT_FILES, as Edgehog builds use U-Boot boot script,
which loads kernel from bootfs_X partitions.
bootfs_X partition is filled with IMAGE_BOOT_FILES.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

Mikhail Vanyulin authored 1 year ago and Jonas Höppner committed 1 year ago

Edgehog boot script uses uImage as kernel name.
Enable KERNEL_IMAGETYPE_SYMLINK for Edgehog builds, as it is taken from deploy
directory during WIC creation.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f8c1d732

Add confluence stage

This adds two jobs to the Yocto pipeline:

 - generate-confluence-page
 - publish-confluence-page

The first one generates a confluence page from a template
(`confluence-page.xml.jinja2`) using information downloaded from all
successful "Deploy FTP" jobs of the same pipeline.

The second one publishes this page in Confluence and displays a link to
the new page. If the page already exists it is overwritten.

The place in Confluence where the new page is published is configurable
via the `CONFLUENCE_SPACE` and `CONFLUENCE_PARENT_ID` variables.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@fee9493b

generate-build-pipeline: re-add YAML format parsing

This was accidentally forgotten in the previous commit.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@9b777f29

Rename "generate_job_from_template" script to "render_jinja2_template"

This makes the script more general and open for other future purposes.
The parsing of the YAML content is conditional now, depending on the new
"format" argument.

The 'main' branch was renamed to main some time ago, but the recipe was
not adapted.

Mikhail Vanyulin authored 1 year ago and Tobias Kahlki committed 1 year ago

For Edgehog builds WIC requires combine_kernel file to present in deploy
directory.
Adding dependency to bootscript recipe, which provides combine_kernel.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

Mikhail Vanyulin authored 1 year ago and Tobias Kahlki committed 1 year ago

For Edgehog builds WIC creation needs combine_kernel file, so we place it
to deploy directory.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

Mikhail Vanyulin authored 1 year ago and Tobias Kahlki committed 1 year ago

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

Mikhail Vanyulin authored 1 year ago and Tobias Kahlki committed 1 year ago

SysData is an Edgehog internal standardized way to store data from bootloader
to the user space level.
Added u-boot-seco-imx-fw-sysdata recipe, which will provide fw_sysdata
utility to access sysdata from the linux.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

Mikhail Vanyulin authored 1 year ago and Tobias Kahlki committed 1 year ago

Add Edgehog specific MACHINEOVERRIDES.
Aside from providing corresponding overrides in recipes, this will also fix
dependency loop for seco-image-edgehog.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

Mikhail Vanyulin authored 1 year ago and Tobias Kahlki committed 1 year ago

Edghog build uses its own build setup process.
Added layer specific files for that.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@b20f6b95

Use multiple CI test pipelines

Instead of mixing dedicated CI test jobs and Yocto build simulation jobs
within one pipeline, use the new multi-build-pipelines architecture to
split them up into separate child pipelines.
This also makes most of the Yocto pipeline code reusable, so that we
don't have to declare all the jobs again in ci-test.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f9dc2517

Rename "build jobs" to "build pipeline"

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@633c9fb4

get_pipeline_jobs: remove redundant job ID output

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@347745c5

download_job_artifacts: add error message if artifact not found

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@50f7396d

ci-test: re-enable build simulation caching

In 438f1e2aae23472d692ce6314e8c683f950657d2 we accidentally disabled
caching for the build simulation by moving the cache definition to a job
class. The order of inheritance led to the cache configuration being
overwritten by the one of the "buildbase" class. Change inheritance
order to fix this.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@30f7d966

update_submodule: fix integration branch name in case of multiple MRs

When there are multiple open merge requests for a commit, we currently
take the newest one to determine the integration branch name. This is
not necessarily the correct one, we can have an open merge request with
the commit being part of the commit history while the top commit is a
different one.

Add a check if the commit is the top commit of the merge request in
order to get the correct integration branch name in these cases.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@438f1e2a

ci-test: use more recent release for build simulation

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@1c900bf9

Update docs/add-new-project-to-pipeline.md
--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8854fd8f

download_job_artifacts: fix temp file creation

mkstemp() returns a tuple containing an OS-level handle to an open file
(as would be returned by os.open()) and the absolute pathname of that
file, in that order.

Use higher-level function NamedTemporaryFile() instead.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@151f1cd6

Refactoring: remove redundant if

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/remove-redundant-if/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@872ec6d9

Refactoring: swap if and else branches

Move the main part of the action to the if branch.

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/swap-if-else-branches/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@c058236c

Sourcery: skip a certain suggestion for better readability

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@b9196ae5

Fix typo in comment

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@eba0649d

gitignore: add .vscode and convert line-endings to LF

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@876da56d

Refactoring: merge dictionary assignments

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/merge-dict-assign/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@e8e3ff37

Refactoring: use single assignment for "if, else"

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/assign-if-exp/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@14eaf45a

Refactoring: inline immediately returned variables

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/inline-immediately-returned-variable/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f9096a52

Refactoring: use "{}" for creating an empty dictionary

This is the most concise and Pythonic way to create a dictionary.

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/dict-literal/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@a3baa39e

Refactoring: use list comprehensions

A list comprehension can create the list on one line, cutting out the
clutter of declaring an empty list and then appending values.

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/list-comprehension/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@52de8d7c

Refactoring: use dictionary comprehensions

A dictionary comprehension can create the dictionary on one line,
cutting out the clutter of declaring an empty dict and then adding
items.

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/dict-comprehension/

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@5d09b9de

Refactoring: simplify length comparisons

Something we often do is check whether a list or sequence has elements
before we try and do something with it.

A Pythonic way of doing this is just to use the fact that Python lists
and sequences evaluate to True if they have elements, and False
otherwise.

Doing it this way is a convention, set out in Python's PEP8 style guide.

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/simplify-len-comparison/

Add recipe to build the install scripts and the post installation steps
from the new repository.
Set MACHINE_INSTALL_SCRIPTS variable to include the install scripts
independently from the distro.

Mikhail Vanyulin authored 1 year ago and Jonas Höppner committed 1 year ago

As we made our own recipe name, get rid of the bbappend file.
Also there is no longer need to set PV inside the recipe, so
set version in recipe names.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

Mikhail Vanyulin authored 1 year ago and Jonas Höppner committed 1 year ago

Renamed u-boot-imx recipes to make it reusable for Edgehog builds.
Also updated recipes that were dependent on it.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

In the external sharedconf repo a autotools base makefile build was
introduced to replace operations done in the recipe.
This adapts the recipe accordingly

The previous release had an bug, that prevented Flash-N-Go Boot from
correctly starting U-Boot.

The scripts are now used integrated as MACHINE RDPENDS, so the images
itself do not need to be modified.

BCS 746-001463

The alternate name is the same that is generated by imx-boot. It is used
to generate WiC images.