if some unnecessary changes occur on upstream upstream repo,
  it fetches and recreates the tarball on the mirror, what may affect the build.
  Reconfigure the mirror sync so that if changes on the upstream lead to
  creating a new tarball, the old copy is storred in the archive directory.

  Add archive MIRROR into the configuration.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@83f4f610

Take group path from group instead of options

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@93eabeb3

YT-157 Adjust GitLab Backup Script to avoid shared projects
 Ensure backups are only processed for direct member projects of a group.
 If a project is shared with a group, backing it up without the appropriate
 permissions can cause the backup to fail.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@77a91f92

YT-93 Improve release workflow

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@edbfded2

YT-164 Sort artifacts on confluence release page for FNG system

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@6ab45a4c

YT-190 Enable credential.useHttpPath for git credential manages in CI pipeline

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@516090fa

YT-167 Implement parsing full URL and multiple tokens

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8abfbca7

YT-176 Skip projects with no permissions
  If a project have no permissions,
  it probably won't be processed in seco-ne CI pipelines.

  Exclude venv directory from version control

For using a local folder, the path has to start with the file://
directive.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@01d6973b

Scripts: Add script to delete all integration/ branches

It happens that the integration branches are left over from provious
runs of the pipeline, when MRs are closed without merging, for example.

Also a bug in the gitlab-ci makes the pipeline fail, when an integration
branch should be reused. The new scripts just deletes all integration
branch in the given project tree without further checks.

Needs to be run manually.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@c05acbe1

Add scripts to setup a virtual python environment

Use:
```
cd script
source setup_venv.sh
```
to change to the venv.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@58554035

scripts:release_info: Deep copy dict in merge_variables

Without the deep copy, the merge operation manipulates the dict it is
storing the new values in.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@bd4caeae

report_image_diff: rework variables extraction

The current solution makes too much assumptions about the pipeline and needs
to reimplement some of the gitlab yaml interpretion.

The new version of the script extracts MACHINE and DISTRO from the jobs themselves:
- MACHINE is part of the build job name ("build-$MACHINE");
- DISTRO is printed by bitbake and can be found in build log.

Typos

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@cbf168eb

build:credentials: Fix error when storing credentials in cache

The credentials helper wasn't storing the credentials in the cache.
Seemingly, the echo added whitespaces at the beginning of the lines
which can't be interpreted by the git credential approve command.
Also the USERNAME was renamed to TOKEN_USER because the variable is
already filled with the system users name.

Mikhail Vanyulin authored 10 months ago and Jonas Höppner committed 9 months ago

If MULTICONFIG=true parameter is passed to setup-environment script, it
will enable multiconfig in local.conf if multiconfigs configurations were
found in layers.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

Add option to use the global download directory as local mirror instead
of as actual download directory.
This option is a "cleaner" approach of sharing the download state
between multiple users on the same system.
Sharing the same folder directly often led to permission issues in the
past.

This reverts commit f4485d8d.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@d09e3a96

CI: generate_firmware: Add releasename as name to json file

To use the generated json file in the json-fwr tooling, it helps to fill
the name.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@c2cd3b43

pipeline:confluence: Add parent for Flash-N-Go release page

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@887db0df

pipelines: Add wic.img files to supported artefacts

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@54f17a69

Update CI_IMAGES_REVISION

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@4fd7f8a8

report_image_diff: get rid of hardcode

Parse machines, pipeline names and distros from gitlab yaml file instead of
harcoding these values in the script.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@05d6a8db

fullbuildpipeline.py: fix timeout

The problem was found in pipeline:
https://git.seco.com/seco-ne/yocto/manifest/-/jobs/954246

Fix missing update of upstream pipeline status inside while loop.

The timeout in the script was not in sync with the default timeout for build jobs.
Set timeout in __get_build_pipelines() function to 8 hours.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@3077d252

handle_artifacts: add "try" block for keep_/delete_artifacts

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@b0be529b

handle_artifacts: fix case when pipeline was canceled

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@1a4a5ab1

Refactoring: move common code to separate files

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@bf6de920

convert_md2html: new font style for main text

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@dc43f865

convert_md2html: move content to left

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f0088d25

convert_md2html: fix long commands

Several blocks of code are too long and don't fit on the page.
Words are not wrapped to the next line.
To fix this, adjust the span slacc in css.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@caa6efa2

convert_md2html: restore several css rules

- fix h1 color
- fix missing gorizontal line above h1
- increased h2 font size
- fix missing borders for table
- restore link style

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@28a1bc5e

convert_md2html: Update the style for the generated html file

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@6656c465

CI: allow failure in new jobs 'handle_artifacts' and 'report-image-diff'

It seems that these jobs are not yet completly errorsafe and we could
ignore the failure without changing the build artifacts.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@de9851be

ci: add a job to handle artifacts after merge

In the latest commit in the manifest it often happens, that the pipeline
does not contain the artifacts itself, but just a link to the pipeline of the MR.

Added a python script with corresponding GitLab CI job to launch this script.
The script searches for the latest pipelines for default branch in manifest repo,
and then decides either to keep or delete artifacts for particular build.

The build artifacts are kept for:
- the latest commit
- the last successful build
- the tagged commits (e.g. "fngsystem/47.0", "kirkstone/20.0")

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@1299a9d6

CI: add "report-image-diff" job

Add a python script and a GitLab CI job to run it. The job is triggered
when the build is complete.

The script retrieves build artifacts for the default/main branch
(currently "kirkstone") in the manifest repo and build artifacts
for the integration branch of merge request.

After that, it compares those builds and creates a summary of the overall
changes. In particular:
- size of compressed artifacts (artifacts.zip)
- size of built image
- difference between manifest files

The summary is reported to stdout and as a comment to the merge request.
Each run of the pipeline means removing the old summary comment and
creating a new one when the build is finished.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@dcc8d92b

build:hardware: Add Trizeps VIII Plus, fix MX93 and sort entries

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@add2bb17

Revert "CI: remove 'setup_ssh' as ssh access is disabled"

The SSH setup is required for some private projects that pull in
external repos via SSH.

This reverts commit 4d5bca8fb8aaebf8794ffa88a6d0aef779c9eedf.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@19ff9d08

package_release: Fix missing variable for os.walk

Even though the variable isn't used in the package_release script, it
seems to be required by the os.walk function. Underscore satisfies the
linter preventing it from causeing unused variable errors.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@6e6f124b

package_release: Add a way to copy the whole directory

Item should end with "/" to be treated as a directory

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ab19eb61

CI: collect_release_information: jump to next pipeline if no deploy jobs found

See for error:
https://git.seco.com/seco-ne/yocto/infrastructure/ci-test/minimal-manifest/-/jobs/828261

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@ae764fbc

CI: Confluence page: Fix not created mx6 section

The function that merges the variables of the different jobs, i.e:

`MACHINE="mx6"`

`MACHINE="mx6ull"`

used python `in` to check if a value already exists.
But `"mx6" in "mx6ull"` is true, so mx6 was dropped.
This rewrites it, storing the variables in a set (which is unique
automatically) instead of space seperated strings.
Space seperation is added before printing the variables to be used on
on the shell.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@094983ae

git_config: Add "*" as safedir to omnit 'dubious ownership' error

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@44000de1

Check bitbake recipes with linter

The job checks bitbake recipes added/updated in MR using the
oelint-adv tool (https://github.com/priv-kweihmann/oelint-adv).

.oelint.json is a rulefile, created with the command
"oelint-adv --print-rulefile". It can be used during development.
It contains default rules for now, but will probably be customized later.

To get the list of changed files in MR, we adjusted the GIT_DEPTH variable.
The new value is 50, which should be fine for getting MR changes,
and still small enough to keep checkout fast for large projects.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@9d9574d1

convert_md2html: fix text duplication

Fix the problem introduced by commit 4f06516 "convert_md2html: fix id
fields for headings".

The text converted from markdown was written twice to the HTML document.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@4f06516a



convert_md2html: fix id fields for headings

Converter adds "user-content-" to "id" fields for headings.
As a result links from TOC do not work.
Fix this for now by removing "user-content-" from the resulting html.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@97898e9e



convert_md2html: use gitlab api

Update convert_md2html.py to use gitlab api with gitlab flavor.

Signed-off-by: Mikhail Vanyulin <mikhail.vanyulin@rtsoft.de>

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@4595835d

build:machines: Add Genio510 to the build job template

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@d8b5ab7b

scripts:firmware: Fix check for U-Boot file types

On i.MX6 devices, the U-Boot file types are not set at all. Extent check for
the existance of the uboot_file_types var.

Jonas Höppner authored 1 year ago and Tobias Kahlki committed 1 year ago

The BB_PRESSURE_MAX_CPU can be set to monitor /proc/pressure/cpu
when scheduling new tasks. I start with a relativly high value, I'm
currently not quite sure how to find out the correct one.

For the general feature
see:

https://www.linux-magazin.de/ausgaben/2020/08/psi
https://www.kernel.org/doc/html/latest/accounting/psi.html
https://docs.yoctoproject.org/bitbake/2.6/bitbake-user-manual/bitbake-user-manual-ref-variables.html#term-BB_PRESSURE_MAX_CPU

The general algorithm is to take the 'total=' number's increment in the
last second and compare it with the threshold.
The bitbake user manual suggests 500, but I've hardly seen such low
values, that's why I'm starting with a much higher one.

The actual usage of the limit is implemeted in
https://git.seco.com/seco-ne/3rd-party/yocto/poky/-/blob/seconorth/kirkstone/bitbake/lib/bb/runqueue.py?ref_type=heads#L193
Some additional debug messages might be added here to monitor, when the
limit is hit.

The old path provokes problems with the auto generated mount services
for our buildservers.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@3146ed1e

CI: increase timeout of package-documentation to 5m

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@4d5bca8f

CI: remove 'setup_ssh' as ssh access is disabled

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8353d920

CI: lava testing: replace ssh with https url and CI_JOB_TOKEN

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@e207c443

gitlab-ci: Disable sourcery

The sourcery plugin fails again due to a missing login. Disable it for
now.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@a3d1e77c

manifest:integration: Fix issue in trigger/include

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@b6e66192

pipelines:build: Add check to JSON generation

Without the check, we have a race condition when building from sources
outside of the Yocto group.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@42bfe4f0

build:prepare: Add setup script for Git Credentials

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@24653ff8

build:machines: Add i.MX93 to the build job template

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@7aae72ea

Re-enable sourcery

With the free open source plan, sourcery works only when called from the
repository's root directory. Re-enable it by calling it from there.

Also apply sourcery's suggestions to file generate_firmware_package.py,
which did not exist before disabling sourcery.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@51e0cf67

build:stages: Add Firmware Package JSON stage

The old Alphaplan stage is deprecated and was removed in a prior commit.
Add new stage to generate the Firmware Package JSON files that do replace
the old Alphaplan articles.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@1830cb02

And another improvement for the build-up-to-date job

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@f7f33de5

Improve the output message in the build-up-to-date job a bit more

Use explicit line breaks to make the message look better.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@68aaab94

Improve the output message in the build-up-to-date job

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@26b9eb1d

Rename "check" job to "build-up-to-date"

This is a more meaningful name and might give a hint what the job does.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@07f0c898

Make build runner tag configurable

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@0d70d719

Disable sourcery check

Sourcery changed its license model to only be available for open source
projects with the free plan. According to the documentation, sourcery
should detect automatically whether it is running on an open source
repo. This does not work in our case, though. Disable it until we find a
way to correctly make it work.

Set the KERNEL_DEVICETREE_OVERLAYS_AUTOLOAD variable in the machine and
not the local.conf.

The extra rootfs space bloats up the fngsystem image. Since we don't use
it for any other devices, remove it here too.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@9be2c52b

pipeline:build: Fix indentation of here-doc

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@245e6537

pipeline:build: Make generation of install script optional

The Genio 700 platform doesn't have an fng_install.sh. Currently, the
pipeline fails because of the exit code in the script check.
Make the generation of the install command optional and only show a
warning if the script is missing.

--

Commit: seco-ne/yocto/infrastructure/gitlab-ci@8fedce8a

pipeline:build: Add MediaTek Genio 700 SoC