--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@19ff9d08

package_release: Fix missing variable for os.walk

Even though the variable isn't used in the package_release script, it
seems to be required by the os.walk function. Underscore satisfies the
linter preventing it from causeing unused variable errors.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@6e6f124b

package_release: Add a way to copy the whole directory

Item should end with "/" to be treated as a directory

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ab19eb61

CI: collect_release_information: jump to next pipeline if no deploy jobs found

See for error:
https://git.seco.com/seco-ne/yocto/infrastructure/ci-test/minimal-manifest/-/jobs/828261

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ae764fbc

CI: Confluence page: Fix not created mx6 section

The function that merges the variables of the different jobs, i.e:

`MACHINE="mx6"`

`MACHINE="mx6ull"`

used python `in` to check if a value already exists.
But `"mx6" in "mx6ull"` is true, so mx6 was dropped.
This rewrites it, storing the variables in a set (which is unique
automatically) instead of space seperated strings.
Space seperation is added before printing the variables to be used on
on the shell.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@094983ae

git_config: Add "*" as safedir to omnit 'dubious ownership' error

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@44000de1

Check bitbake recipes with linter

The job checks bitbake recipes added/updated in MR using the
oelint-adv tool (https://github.com/priv-kweihmann/oelint-adv).

.oelint.json is a rulefile, created with the command
"oelint-adv --print-rulefile". It can be used during development.
It contains default rules for now, but will probably be customized later.

To get the list of changed files in MR, we adjusted the GIT_DEPTH variable.
The new value is 50, which should be fine for getting MR changes,
and still small enough to keep checkout fast for large projects.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@9d9574d1

convert_md2html: fix text duplication

Fix the problem introduced by commit 4f06516 "convert_md2html: fix id
fields for headings".

The text converted from markdown was written twice to the HTML document.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@4f06516a



convert_md2html: fix id fields for headings

Converter adds "user-content-" to "id" fields for headings.
As a result links from TOC do not work.
Fix this for now by removing "user-content-" from the resulting html.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@97898e9e



convert_md2html: use gitlab api

Update convert_md2html.py to use gitlab api with gitlab flavor.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@4595835d

build:machines: Add Genio510 to the build job template

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@d8b5ab7b

scripts:firmware: Fix check for U-Boot file types

On i.MX6 devices, the U-Boot file types are not set at all. Extent check for
the existance of the uboot_file_types var.

Jonas Höppner authored 1 year ago and Tobias Kahlki committed 1 year ago

The BB_PRESSURE_MAX_CPU can be set to monitor /proc/pressure/cpu
when scheduling new tasks. I start with a relativly high value, I'm
currently not quite sure how to find out the correct one.

For the general feature
see:

https://www.linux-magazin.de/ausgaben/2020/08/psi
https://www.kernel.org/doc/html/latest/accounting/psi.html
https://docs.yoctoproject.org/bitbake/2.6/bitbake-user-manual/bitbake-user-manual-ref-variables.html#term-BB_PRESSURE_MAX_CPU

The general algorithm is to take the 'total=' number's increment in the
last second and compare it with the threshold.
The bitbake user manual suggests 500, but I've hardly seen such low
values, that's why I'm starting with a much higher one.

The actual usage of the limit is implemeted in
https://git.seco.com/seco-ne/3rd-party/yocto/poky/-/blob/seconorth/kirkstone/bitbake/lib/bb/runqueue.py?ref_type=heads#L193
Some additional debug messages might be added here to monitor, when the
limit is hit.

The old path provokes problems with the auto generated mount services
for our buildservers.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@3146ed1e

CI: increase timeout of package-documentation to 5m

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@4d5bca8f

CI: remove 'setup_ssh' as ssh access is disabled

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8353d920

CI: lava testing: replace ssh with https url and CI_JOB_TOKEN

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@e207c443

gitlab-ci: Disable sourcery

The sourcery plugin fails again due to a missing login. Disable it for
now.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@a3d1e77c

manifest:integration: Fix issue in trigger/include

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@b6e66192

pipelines:build: Add check to JSON generation

Without the check, we have a race condition when building from sources
outside of the Yocto group.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@42bfe4f0

build:prepare: Add setup script for Git Credentials

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@24653ff8

build:machines: Add i.MX93 to the build job template

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@7aae72ea

Re-enable sourcery

With the free open source plan, sourcery works only when called from the
repository's root directory. Re-enable it by calling it from there.

Also apply sourcery's suggestions to file generate_firmware_package.py,
which did not exist before disabling sourcery.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@51e0cf67

build:stages: Add Firmware Package JSON stage

The old Alphaplan stage is deprecated and was removed in a prior commit.
Add new stage to generate the Firmware Package JSON files that do replace
the old Alphaplan articles.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@1830cb02

And another improvement for the build-up-to-date job

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f7f33de5

Improve the output message in the build-up-to-date job a bit more

Use explicit line breaks to make the message look better.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@68aaab94

Improve the output message in the build-up-to-date job

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@26b9eb1d

Rename "check" job to "build-up-to-date"

This is a more meaningful name and might give a hint what the job does.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@07f0c898

Make build runner tag configurable

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@0d70d719

Disable sourcery check

Sourcery changed its license model to only be available for open source
projects with the free plan. According to the documentation, sourcery
should detect automatically whether it is running on an open source
repo. This does not work in our case, though. Disable it until we find a
way to correctly make it work.

Set the KERNEL_DEVICETREE_OVERLAYS_AUTOLOAD variable in the machine and
not the local.conf.

The extra rootfs space bloats up the fngsystem image. Since we don't use
it for any other devices, remove it here too.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@9be2c52b

pipeline:build: Fix indentation of here-doc

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@245e6537

pipeline:build: Make generation of install script optional

The Genio 700 platform doesn't have an fng_install.sh. Currently, the
pipeline fails because of the exit code in the script check.
Make the generation of the install command optional and only show a
warning if the script is missing.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8fedce8a

pipeline:build: Add MediaTek Genio 700 SoC

We use rpm packages per default in our Yocto setup.
Use it for MediaTek as well (different package classes provoke potential
issues in various pre-/post-steps and also reduce reusability between
platforms with the same arch).

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@6f773de5

Remove Alphaplan stage

We do not use Alphaplan anymore. Remove everything Alphaplan related
stuff from the pipeline.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@56bd1900

Confluence: add helper script to get page content

This can be helpful if something is configurable in the GUI editor and
we can't figure out to do the same thing via the API.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@c3b794c6

Confluence: enforce fixed page width on creation

On page creation via the API, the page always seems to be created with
full width, even if "full_width=False" is set. Update the page right
after creating it in order to correctly set it to fixed width.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@82053a2f

Confluence: set width for download table columns

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@96f5745c

Confluence: group download files by type

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@3697b4ea

cancel_pipelines: reverse order of cancellation

Cancel parent pipeline first before cancelling its children in order not
to have failed parent pipelines (trigger jobs are failed if their
downstream pipeline is cancelled).

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@5f4ca2fb

cancel_pipelines: list actually cancelled (child-)pipelines

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@c80cba84

Cancel all pipelines on newer commit

We already canceled running pipelines before. Gitlab, however, might
have already canceled previous pipelines automatically, but without
canceling their child pipelines.
Remove the "running" status filter when cancelling pipelines, so that
also already cancelled pipelines will be included.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@84eb6e22

Revert "Cancel all pipelines on newer commit"

This reverts commit f7582b74e7c565d55bb8a0315fe75c4ab4385fa2.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f7582b74

Cancel all pipelines on newer commit

We already canceled running pipelines before. Gitlab, however, might
have already canceled previous pipelines automatically, but without
canceling their child pipelines.
Extend the mechanism now to cancel all pipelines up to and including
"running" as well as "canceled" states.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@0ec55c07

Fix list() call in update_submodule

Here we don't just iterate over the results, so we have to use
"all=True" instead of "as_list=False". See previous commit for
explaination.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@4b3a517e

Make sure to get all items on Gitlab list() calls

The list() call on Gitlab classes uses pagination per default. If the
number of items exceeds a certain limit, we only get a part of the
results. We can pass "all=True" to the calls in order to disable
pagination and get all results.
If we're looping over the items, though, the recommended way of getting
them is to use a generator object, which works calling list() with
"as_list=False". See documentation for reference:

https://python-gitlab.readthedocs.io/en/v2.10.1/api-usage.html#pagination

Note: the "as_list" argument is deprecated in newer versions of the
python-gitlab library, in favor of the newer "iterator=True":
https://python-gitlab.readthedocs.io/en/v3.6.0/changelog.html#feature
We are using v2.10.1 of the library, though, so this commit also updates
calls using the new way, which is not working with our version.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@c292d144

Confluence: combine data from parallel child pipelines

We deploy Yocto image and SDK in different pipelines, which both have
their Confluence jobs with the same Confluence target. Previously these
pipelines have been independent, so that running the Confluence job in
one of them would overwrite the Confluence page if it has been run in
the other pipeline before. Hence we could either have the image files or
the SDK files in the Confluence page.
Combine these now by checking for other parallel child pipelines with
the same Confluence configuration, so that running the Confluence jobs
in the Yocto pipeline also includes the information from the SDK build
and vice versa.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@53106170

Fix ci-test SDK version

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@d6d4ab61

Fix ci-test SDK download

Since the implementation of multiple child pipelines we cannot use the
previous artifacts download links anymore specifying tag and job name,
because the jobs for image build and SDK build are equal. Use direct
download links instead via job number.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@bc85e805

Build documentation in Yocto

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@0a3b1406

Confluence: get only those build jobs with a corresponding successful deploy job

We cannot handle build jobs and deploy jobs similarly, because all build
jobs are successful usually, while not all deploy jobs necessarily have
run. Get build.env only from those build job which have a corresponding
successful deploy job, otherwise MACHINE and SUPPORTED_HARDWARE fields
always contain the full list of built platforms, even if they might have
not been deployed.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@97228cdb

Move MACHINE variable from deploy.env to build.env

The Confluence stage is the only stage that needs the MACHINE variable.
Since the collect_release_information script already loads the build.env
from the build jobs, there is no need to store this variable in
deploy.env.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ba156b0f

Confluence: check if variables are defined

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@cae6e282

Confluence: improve variable passing to template

Use a general mechanism of storing all job variables for each build job
separately, so that we have all documentation files in the
FILES_documentation variable. This makes the dedicated HTML_FILES
variable obsolete.
The merged variables are still kept, though, so that we have all
machines in the MACHINE variable to loop over.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@41f8a49f

Confluence: add error handling if artifact not found

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@1a519057

Make documentation files configurable

This disables the documentation jobs for SDK and FNGSystem.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@303e24eb

Build documentation in dedicated job

Previously the documentation files were rendered as part of the package
jobs. Each package job converted the same set of static markdown files
to HTML files, which was redundant.

Add dedicated jobs to build, package and deploy the documentation files.

Modify convert_md2html.py to support file globs as arguments.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@838da585

Refactoring: move Alphaplan stage code to the end

This reflects the stage order in the code.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ff31d083

Refactoring: use multiple MACHINE loops for better structure

This is a preparation for adding dedicated documentation jobs. In order
to have these grouped by stage along with their corresponding machine
jobs, we have to get rid of the giant outer loop and add loops for each
stage instead.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@69973c95

Deploy: save deployed files in dotenv variable instead of dedicated file

Previously the deployed files were stored in a dedicated text file
called files.txt. Move this information to a dotenv variable called
FILES, so that we don't need special handling for it in the Confluence
stage.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@45d04033

Remove disabling sourcery low-code-quality in package_release

This check is not failing anymore since the latest refactorings.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@cf0336d6

Define manifest file to build from

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@a7503809

Confluence: add supported hardware information

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@0ef32ce7

Update simulated build to kirkstone-14.0

Remove temporary workarounds that were necessary due to changes after
the previously used release kirkstone-7.0.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@64dc51d3

Use regex operator to check for multiple variables

The "&&" operator does not work as expected when checking if multiple
variables are set.

See for reference:
https://gitlab.com/gitlab-org/gitlab/-/issues/321371

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@89fbce24

Hide Confluence stage if confluence variables not set

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@db50e93b

Hide deploy stage if DEPLOY_RELEASE_TARGET not set

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f4aa0048

Hide FTP stage if DEPLOY_FTP_TARGET not set

This makes the variable HIDE_FTP_UPLOAD_STAGE obsolete, because we can
hide the FTP stage now by setting DEPLOY_FTP_TARGET to an empty string.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@409c4f27

Do not run CI test sub-pipeline on master

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@a731b156

Remove DISTRO_IMAGES

This variable has been removed in Yocto. Its contents have been merged
into DISTRO_RELEASE_ARTEFACTS.

See for reference:
seco-ne/yocto/layers/meta-seconorth-distro@a316e95e

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@45f2aca7

Refactoring: remove parsing of testdata.json in package stage

Extract and save these variables in the build stage instead and
explicitly pass them as arguments to the package script.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8ca2487d

Move .simulate_build below .build_yocto

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@2808f067

Fix build.env contents for simulated build

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@73d00715

Pretty print release metadata JSON file

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f2a29f53

Refactoring: remove RELEASE_NAME from deploy targets

Append it to the configured target at the deploy class level instead.
This removes the need for eval'ing the variables before usage.
RELEASE_NAME can be used directly at the class level because it is known
from build-version.env.

This makes the deploy behavior identical to the Azure stage.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@e136e5e2

Remove top-level "release" folder from package

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@1d5e479f

Define MACHINE variable on job level

Instead of passing MACHINE from stage to stage or loading it from
testdata.json, use the original value from the Jinja2 loop and set it
directly for each job where it is used.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@cdd47023

Introduce separate variables to set RELEASE_NAME and RELEASE_VERSION

The expressions to calculate RELEASE_NAME and RELEASE_VERSION cannot be
stored within these variables themseselves. If set on the trigger level,
they would override the calculated values in build-version.env then,
as trigger variables always have the highest precedence.

Use separate variables RELEASE_VERSION_EXPRESSION and
RELEASE_NAME_EXPRESSION to define how RELEASE_VERSION and RELEASE_NAME
are calculated instead.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@372d192b

Use RELEASE_VERSION and RELEASE_NAME from build-version job

Instead of passing these variables from stage to stage or regenerating
their values in later stages, use the ones set in the build-version job
at all places.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@cad30c20

Remove RELEASE_SUFFIX variable

The RELEASE_NAME variable can be set directly now, so there's no need to
have a dedicated RELEASE_SUFFIX anymore. A previous configuration like

  RELEASE_SUFFIX: "-custom"

can now be achieved using:

  RELEASE_NAME: "Yocto-${RELEASE_VERSION}-custom"

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@922f49f8

Add yocto version job

This adds a machine-independent job "build-version" which populates the
RELEASE_VERSION and RELEASE_NAME variables, so that following jobs can
use these without depending on the various build jobs.
The variables can be set from the trigger job in a project's
`.gitlab-ci.yml` file. They are eval'ed before saving them to
version.env, so we can use deferred variable expansion or even command
execution to construct their values. This mechanism is already used for
the Flash-N-Go System variables.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@073ae81e

LAVA: rename MACHINE to LAVA_MACHINE

Rename variable in order to avoid confusion with the original MACHINE
variable used everywhere else. Also rename the local Jinja2 variable to
include an underscore to make naming consistent.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@a91baa55

Clean build.env before writing

This had been accidentally removed in
5e36715ef6cf98df4c1b98fedddc0c3c50ed4040.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@1860ca8e

Remove LOG_PREFIX variable

This was used in times when the same code was executed from different
places. We don't do that anymore, so the variable is obsolete.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@3d2dbe4d

Documentation: update job generation chapter

Some custom projects already include the NoGPLv3 or GPLv2 layer. If these
layers are already present, the Bitbake parsing fails. As a precaution,
check if the layers are already present and only add them if not.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@17b87f84

Refactoring: remove CI_PARAM_ prefix from variables

There is no need to add a prefix to the variables. Basically every
variable can be set or overridden at the trigger level, so we just use
plain variable names for everything.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@825f3686

Refactoring: do not rename *_PATH variables in build.env

We were using different variable names for the same variable in build
and package stages. Unify these names in order to reduce potential
confusion.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@2346bb98

Refactoring: make build variable naming consistent

Use underscore to separate words within variable name.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@581541b9

Refactoring: improve FNG_INSTALL_URL definition

FNG_INSTALL_URL was defined in build.env only, and in order to use it
during the build we sourced build.env after writing it.
Remove this workaround and define the variable cleanly in the first
place. This removes the intermediate variable FNG_INSTALL_PATH.
Also move the "/artifact" path from JOB_URL to FNG_INSTALL_URL in order
to make the JOB_URL variable name match better.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@35784f29

Refactoring: improve IMAGEPATH et al. definition

On every usage of IMAGEPATH, SDKPATH and LICENSESPATH we prepended
BUILDPATH to it. Prepend it directly on definition to prevent
repetition.
Also remove redundant variable IMAGEBASEPATH.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@5e36715e

Remove needless variables from build.env

There were three variables set in build.env of which the values are
already known in the environment for all jobs, because they are passed
from the parent pipeline. Use the original variables instead of defining
new ones.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@84aa6afb

Fix LAVA job name

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8160c7d5

Sourcery: enable "low-code-quality" check globally

Disable it locally instead, so that
 1. new code additions are always checked, and
 2. we can refactor one function at a time to make the check pass.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ef60c378

Sourcery: keep "for-append-to-extend" check disabled

This is another check that suggests changes an average developer without
profound Python experience would find hard to understand. Disable it.

See for reference:
https://docs.sourcery.ai/Reference/Python/Default-Rules/for-append-to-extend/

Actually fixes the lates commit.

Append "meta-seconorth-nogplv3" and "meta-gplv2" layers to bblayers.conf
only for "fngsystem-image" to avoid possible side effects for normal
builds.