--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@3eeebfdc

Changelog: Include all related projects into the changelog

In build-pipeline.yml, enable the new parameter by default.

YT-256

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@03af7519

Changelog-generator: Add parameter to include all integrated projects

Before this, the changelog was generated on a given list of projects,
all using the same branch.
The new parameter now goes the the specified projects from the
parameters, and detects all integrated projects into this.
So normally you would specify one or two manifest projects, everything
else is automatically added to the list.

This also handles different branch names. As for example the kernel
branches are different, the integration branch is used instead of the
branch from the command line.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@37b396e8

YT-247: Automate Initiation of Build Stages in Scheduled Pipelines

  Enable automatic initiation of the build stage for the SDK
  and the test stage for pipelines within scheduled pipelines.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@e3a76466

scripts: Add simple python script to convert license.manifest to md table

  if some unnecessary changes occur on upstream upstream repo,
  it fetches and recreates the tarball on the mirror, what may affect the build.
  Reconfigure the mirror sync so that if changes on the upstream lead to
  creating a new tarball, the old copy is stored in the archive directory.

  Add archive MIRROR into the configuration.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f6b301dc

YT-227 Fix submit LAVA task for smoke tests

  Smoke tests use a single type of test (boot.jinja2).
  However, some parameters need to be dynamic, such as device type.
  To address this, the `test-plan-args-var` introduced
  what allows environment variables with extra parameters to be used
  in the LAVA job generation script.
  The pipeline has been adjusted accordingly.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@be2a04ab

Enable verbose mode for submit_test.py

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@cf0071f2

YT-234: Fix ci-test pipelines (was broken with YT-65 MR)

 The YT-65 MR refactored the repo_checkout job,
 which is also used in CI-test pipelines.
 Added the necessary changes to the CI-test code.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@667964a6

Fix typos

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@bd7e9366

YT-65 We perform a repository checkout in the manifest project for each build job.
  Each pipeline builds code for multiple MACHINES.
  This results in multiple downloads from each repository in the manifest.
  Occasionally, certain repositories may fail to fetch data.
  To avoid fetching the same data repeatedly for each build job in the pipeline,
  the `repo_checkout` stage is added.
  In this stage, the repo tool prepares the repository structure once
  and stores it in the cache.
  Subsequent build jobs use the pre-fetched data from the cache.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@de29fa3e

    YT-230: Fix Manifest Pipeline after "Improve Release" Changes

    Fix typo introduced in previous commit

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@417b79ef

YT-230: Fix Manifest Pipeline after "Improve Release" Changes

The recent changes introduced new rules in manifest-pipeline-yocto.yml,
which have overwritten the general rules defined in build-pipeline.yml.
Use !reference to combine both sets of rules effectively.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@a7e07f19

YT-180 Generation of the firmware package failed due to a typo
  in the code that checks of existance of the variable.
  What caused  TypeError: unhashable type: 'list'

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@7bd78810

PBB-32 Fix suffix addition in FNGo pipeline for custom projects
 due to an issue in GitLab where escaped variables are not correctly
 passed to child pipelines, a workaround is required.
 As a result, we use raw variables.
 The DEFERRED_RELEASE_VERSION variable was introduced for this purpose.
 However, for custom builds, we have an additional place where we modify
 the DEFERRED_RELEASE_VERSION variable to add a suffix.
 Therefore, extra interim variables need to be added to handle the workaround
 and ensure we get the raw variable in the end.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@83f4f610

Take group path from group instead of options

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@93eabeb3

YT-157 Adjust GitLab Backup Script to avoid shared projects
 Ensure backups are only processed for direct member projects of a group.
 If a project is shared with a group, backing it up without the appropriate
 permissions can cause the backup to fail.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@77a91f92

YT-93 Improve release workflow

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@edbfded2

YT-164 Sort artifacts on confluence release page for FNG system

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@6ab45a4c

YT-190 Enable credential.useHttpPath for git credential manages in CI pipeline

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@516090fa

YT-167 Implement parsing full URL and multiple tokens

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8abfbca7

YT-176 Skip projects with no permissions
  If a project have no permissions,
  it probably won't be processed in seco-ne CI pipelines.

  Exclude venv directory from version control

For using a local folder, the path has to start with the file://
directive.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@01d6973b

Scripts: Add script to delete all integration/ branches

It happens that the integration branches are left over from provious
runs of the pipeline, when MRs are closed without merging, for example.

Also a bug in the gitlab-ci makes the pipeline fail, when an integration
branch should be reused. The new scripts just deletes all integration
branch in the given project tree without further checks.

Needs to be run manually.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@c05acbe1

Add scripts to setup a virtual python environment

Use:
```
cd script
source setup_venv.sh
```
to change to the venv.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@58554035

scripts:release_info: Deep copy dict in merge_variables

Without the deep copy, the merge operation manipulates the dict it is
storing the new values in.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@bd4caeae

report_image_diff: rework variables extraction

The current solution makes too much assumptions about the pipeline and needs
to reimplement some of the gitlab yaml interpretion.

The new version of the script extracts MACHINE and DISTRO from the jobs themselves:
- MACHINE is part of the build job name ("build-$MACHINE");
- DISTRO is printed by bitbake and can be found in build log.

Typos

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@cbf168eb

build:credentials: Fix error when storing credentials in cache

The credentials helper wasn't storing the credentials in the cache.
Seemingly, the echo added whitespaces at the beginning of the lines
which can't be interpreted by the git credential approve command.
Also the USERNAME was renamed to TOKEN_USER because the variable is
already filled with the system users name.

Mikhail Vanyulin authored 10 months ago and Jonas Höppner committed 9 months ago

If MULTICONFIG=true parameter is passed to setup-environment script, it
will enable multiconfig in local.conf if multiconfigs configurations were
found in layers.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

Add option to use the global download directory as local mirror instead
of as actual download directory.
This option is a "cleaner" approach of sharing the download state
between multiple users on the same system.
Sharing the same folder directly often led to permission issues in the
past.

This reverts commit f4485d8d.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@d09e3a96

CI: generate_firmware: Add releasename as name to json file

To use the generated json file in the json-fwr tooling, it helps to fill
the name.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@c2cd3b43

pipeline:confluence: Add parent for Flash-N-Go release page

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@887db0df

pipelines: Add wic.img files to supported artefacts

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@54f17a69

Update CI_IMAGES_REVISION

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@4fd7f8a8

report_image_diff: get rid of hardcode

Parse machines, pipeline names and distros from gitlab yaml file instead of
harcoding these values in the script.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@05d6a8db

fullbuildpipeline.py: fix timeout

The problem was found in pipeline:
https://git.seco.com/seco-ne/yocto/manifest/-/jobs/954246

Fix missing update of upstream pipeline status inside while loop.

The timeout in the script was not in sync with the default timeout for build jobs.
Set timeout in __get_build_pipelines() function to 8 hours.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@3077d252

handle_artifacts: add "try" block for keep_/delete_artifacts

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@b0be529b

handle_artifacts: fix case when pipeline was canceled

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@1a4a5ab1

Refactoring: move common code to separate files

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@bf6de920

convert_md2html: new font style for main text

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@dc43f865

convert_md2html: move content to left

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f0088d25

convert_md2html: fix long commands

Several blocks of code are too long and don't fit on the page.
Words are not wrapped to the next line.
To fix this, adjust the span slacc in css.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@caa6efa2

convert_md2html: restore several css rules

- fix h1 color
- fix missing gorizontal line above h1
- increased h2 font size
- fix missing borders for table
- restore link style

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@28a1bc5e

convert_md2html: Update the style for the generated html file

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@6656c465

CI: allow failure in new jobs 'handle_artifacts' and 'report-image-diff'

It seems that these jobs are not yet completly errorsafe and we could
ignore the failure without changing the build artifacts.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@de9851be

ci: add a job to handle artifacts after merge

In the latest commit in the manifest it often happens, that the pipeline
does not contain the artifacts itself, but just a link to the pipeline of the MR.

Added a python script with corresponding GitLab CI job to launch this script.
The script searches for the latest pipelines for default branch in manifest repo,
and then decides either to keep or delete artifacts for particular build.

The build artifacts are kept for:
- the latest commit
- the last successful build
- the tagged commits (e.g. "fngsystem/47.0", "kirkstone/20.0")

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@1299a9d6

CI: add "report-image-diff" job

Add a python script and a GitLab CI job to run it. The job is triggered
when the build is complete.

The script retrieves build artifacts for the default/main branch
(currently "kirkstone") in the manifest repo and build artifacts
for the integration branch of merge request.

After that, it compares those builds and creates a summary of the overall
changes. In particular:
- size of compressed artifacts (artifacts.zip)
- size of built image
- difference between manifest files

The summary is reported to stdout and as a comment to the merge request.
Each run of the pipeline means removing the old summary comment and
creating a new one when the build is finished.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@dcc8d92b

build:hardware: Add Trizeps VIII Plus, fix MX93 and sort entries

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@add2bb17

Revert "CI: remove 'setup_ssh' as ssh access is disabled"

The SSH setup is required for some private projects that pull in
external repos via SSH.

This reverts commit 4d5bca8fb8aaebf8794ffa88a6d0aef779c9eedf.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@19ff9d08

package_release: Fix missing variable for os.walk

Even though the variable isn't used in the package_release script, it
seems to be required by the os.walk function. Underscore satisfies the
linter preventing it from causeing unused variable errors.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@6e6f124b

package_release: Add a way to copy the whole directory

Item should end with "/" to be treated as a directory

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ab19eb61

CI: collect_release_information: jump to next pipeline if no deploy jobs found

See for error:
https://git.seco.com/seco-ne/yocto/infrastructure/ci-test/minimal-manifest/-/jobs/828261

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ae764fbc

CI: Confluence page: Fix not created mx6 section

The function that merges the variables of the different jobs, i.e:

`MACHINE="mx6"`

`MACHINE="mx6ull"`

used python `in` to check if a value already exists.
But `"mx6" in "mx6ull"` is true, so mx6 was dropped.
This rewrites it, storing the variables in a set (which is unique
automatically) instead of space seperated strings.
Space seperation is added before printing the variables to be used on
on the shell.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@094983ae

git_config: Add "*" as safedir to omnit 'dubious ownership' error

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@44000de1

Check bitbake recipes with linter

The job checks bitbake recipes added/updated in MR using the
oelint-adv tool (https://github.com/priv-kweihmann/oelint-adv).

.oelint.json is a rulefile, created with the command
"oelint-adv --print-rulefile". It can be used during development.
It contains default rules for now, but will probably be customized later.

To get the list of changed files in MR, we adjusted the GIT_DEPTH variable.
The new value is 50, which should be fine for getting MR changes,
and still small enough to keep checkout fast for large projects.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@9d9574d1

convert_md2html: fix text duplication

Fix the problem introduced by commit 4f06516 "convert_md2html: fix id
fields for headings".

The text converted from markdown was written twice to the HTML document.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@4f06516a



convert_md2html: fix id fields for headings

Converter adds "user-content-" to "id" fields for headings.
As a result links from TOC do not work.
Fix this for now by removing "user-content-" from the resulting html.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@97898e9e



convert_md2html: use gitlab api

Update convert_md2html.py to use gitlab api with gitlab flavor.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@4595835d

build:machines: Add Genio510 to the build job template

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@d8b5ab7b

scripts:firmware: Fix check for U-Boot file types

On i.MX6 devices, the U-Boot file types are not set at all. Extent check for
the existance of the uboot_file_types var.

Jonas Höppner authored 1 year ago and Tobias Kahlki committed 1 year ago

The BB_PRESSURE_MAX_CPU can be set to monitor /proc/pressure/cpu
when scheduling new tasks. I start with a relativly high value, I'm
currently not quite sure how to find out the correct one.

For the general feature
see:

https://www.linux-magazin.de/ausgaben/2020/08/psi
https://www.kernel.org/doc/html/latest/accounting/psi.html
https://docs.yoctoproject.org/bitbake/2.6/bitbake-user-manual/bitbake-user-manual-ref-variables.html#term-BB_PRESSURE_MAX_CPU

The general algorithm is to take the 'total=' number's increment in the
last second and compare it with the threshold.
The bitbake user manual suggests 500, but I've hardly seen such low
values, that's why I'm starting with a much higher one.

The actual usage of the limit is implemeted in
https://git.seco.com/seco-ne/3rd-party/yocto/poky/-/blob/seconorth/kirkstone/bitbake/lib/bb/runqueue.py?ref_type=heads#L193
Some additional debug messages might be added here to monitor, when the
limit is hit.

The old path provokes problems with the auto generated mount services
for our buildservers.