Improve CI documentation

BCS 746-000636

README.md

+<!----------------------------------------------------------------------------->
 # gitlab-ci
+<!----------------------------------------------------------------------------->
 
-GitLab CI scripts
-
-This repository contains scripts for Yocto infrastructure tasks, e.g. automatic
-integration of project changes into the manifest. It is meant to be included into all
-relevant Yocto repositories as a [git submodule][1].
-
-
-## Deploy changes into all projects using 'gitlab-ci' as submodule
-
-The gitlab-ci repo is integrated into the other related projects as submodule and as
-reference in their .gitlab-ci.yml file. Both references need to be updated when a
-change in the repo should be deployed to the projects using it.
-
-To achieve the the CI pipeline is setup to update all these projects in one run.
-The submodule *must not* be updated manually, but the CI needs some manual steps
-to do the deployment.
-As this repo contains all the CI code some extra care is needed not to break the
-whole thing. To allow testing the CI, before everything is merged into the yocto
-pipeline a special set of test projects is available in the ci-test group.
-
-### The workflow to deploy a change 
-
-* Commit and push change to the gitlab-ci repo, create a Merge Request
-* The pipeline checks syntax and so on in the Analyze stage
-* The Integrate stage allows testing the change in the complete setup
-  There are two possible integrate steps, one to deploy the change into Merge Request in 
-  the ci-test group, the other is for the productive yocto setup.
-* When the integrate stage has run, a build is started in the related manifest projects
-
-* In all projects there are now integration branches created, named
-  'integrate/gitlab-ci/<gitlab-ci branch name>'
-  These could be used to add additional changes if needed. But the should only happen
-  in very special cases.
-
-* If everything is OK until here, (build OK, CI works as expected ...)
-  the change can be merged into the main branch of gitlab-ci.
-  *This only adds the patch to the gitlab-ci repo's main branch, the reference
-    in the projects using this are not yet updated*
-
-* After the merge a new pipeline is created. In this pipeline there are two 
-  manual jobs: 'merge-ci-test' and 'merge-yocto'
-  These jobs update the reference in the project groups 'ci-test' and 'yocto'.
-  This is split up to allow more testing in cases where the CI changes can only
-  be tested when working on the master branch.
-
-* When the behaviour of the CI on the master branch may be effected of the change
-  only 'merge-ci-test' should be executed. 
-  After this some tests may be done on the ci-test projects.
-  Probably you want to create a commit in minimal-foo, create a merge-request,
-  check the running pipeline, and merge it.
-  After that, check if the merge is succesfull and the manifest is updated as 
-  needed.
-
-* When everything is working the 'merge-yocto' pipeline can be executed, to 
-  deploy the change to the productive projects.
-* When the tests on 'ci-test' have failed, just create a new patch fixing the
-  issue in the gitlab-ci repo and do the testing again.
-
-
-
-## Providing `GITBOT_TOKEN`
-
-The scripts need a [personal GitLab access token][2] of a user with access to all
-relevant repositories (manifest and all contained projects) in order to work correctly.
-We have created the @guf-gitbot user for this task. Norman and Tim know the login
-credentials for it. These should not be needed, though, as the user is member of the
-[Garz & Fricke][3] group and thus already has access to all group projects. However,
-everyone can add the user as a member to every project he has access to, if needed.
-
-The personal access token of the @guf-gitbot user has to be provided on each repository
-via the CI environment variable `GITBOT_TOKEN` (set under project's settings -> CI/CD
--> Variables). Its value can be read on projects which already have this variable
-defined.
-
-
-## Workflow to add a repository to SRCREV.conv for reproducible bitbake builds
-
-Some bitbake recipes are set to `SRCREV = "${AUTOREV}"`. To enable reproducible
-builds, the current revision is written to the SRCREV.conf file. This file is
-part of the manifest repo, which is used for release tags. This workflow adds
-the a gitlab pipeline to the project that automatically sets the current
-version in the SRCREV.conf file.
-1. Add the gitlab-ci repo as submodule to the project by using the correct relative path\
-`git submodule add ../../yocto/infrastructure/gitlab-ci .gitlab-ci`
-1. Add an approriate .gitlab-ci.yml file. For example, copy it from the repository
-[egalxi2c][4]. Modify the following variables in the file:
-	*  MASTER\_BRANCH\_PROJECT: Set the project master branch
-	*  BB\_RECIPE\_NAME: Set the name of the bitbake recipe
-1. Create a corresponding entry in the SRCREV.conf file of the manifest repo\
-`SRCREV_pn-<bitbake-recipe> = "<current-commit-hash>"`
-1. Add the project into the gitlab-ci project's .gitlab-ci.yml, so changes to the
-CI are deployed into the new project.
-
-**Repository settings**
-1. Verify that the merge strategy is set to *Fast-forward merge*
-(General -> Merge requests -> Merge method)
-1. Enable the CI/CD functionality for the repository
-(General -> Visibility, project features, permissions -> CI/CD)
-1. Check that the default branch is protected and that *Maintainers + Developers*
-are allowed to merge (Repository -> Protected branches)
-
+This repository contains **GitLab CI scripts** for Yocto infrastructure tasks,
+e.g. automatic integration of project changes into the manifest. It is meant to
+be included into all relevant Yocto repositories as a [git submodule][1].
 
 [1]: https://git-scm.com/book/en/v2/Git-Tools-Submodules
+
+
+<!----------------------------------------------------------------------------->
+## The GitBot user
+<!----------------------------------------------------------------------------->
+
+Most scripts inside this repository need a [personal GitLab access token][2] of
+a user with access to all relevant repositories (manifest and all contained
+projects) in order to work correctly.
+We have created the @gitbot user for this task. The login credentials for it are
+stored in our [KeePass][3] safe. These should not be needed, though, as the user
+is member of the [`seco-ne`][4] group and thus already has access to all group
+projects. However, everyone can add the user as a member to every project he has
+access to, if needed.
+
+The personal access token of the @gitbot is provided via the CI environment
+variable `GITBOT_TOKEN` for all projects in the `seco-ne` group (set in the
+[group's CI/CD settings][5] under "Variables").
+
 [2]: https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html
-[3]: https://gitlab.com/garz-fricke
-[4]: https://gitlab.com/seco-ne/kernel/modules/egalaxi2c
+[3]: https://keepass.info
+[4]: https://git.seco.com/seco-ne
+[5]: https://git.seco.com/groups/seco-ne/yocto/-/settings/ci_cd
+
+
+<!----------------------------------------------------------------------------->
+## Automatic manifest integration
+<!----------------------------------------------------------------------------->
+
+See this chapter for information on how the automatic integration process works:
+
+▶ [Automatic manifest integration][6]
+
+[6]: docs/automatic-manifest-integration.md
+
+
+<!----------------------------------------------------------------------------->
+## Deployment
+<!----------------------------------------------------------------------------->
+
+See this chapter for information on how to deploy changes in the `gitlab-ci`
+repository to all repositories that are using it.
+
+▶ [gitlab-ci Deployment][7]
+
+[7]: docs/gitlab-ci-deployment.md

docs/CI_diagram.drawio

-<mxfile host="Electron" modified="2021-05-26T13:35:43.217Z" agent="5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/14.6.13 Chrome/89.0.4389.128 Electron/12.0.7 Safari/537.36" etag="EMktwfk9Tznhq6M5jyJl" version="14.6.13" type="device"><diagram name="Page-1" id="6133507b-19e7-1e82-6fc7-422aa6c4b21f">7V1bc6M4Fv41rpp9cIq77cd2LjO7m6nq7Z7a7X1KYSxjpgF5AXfi+fWroxsCge3E16TVlUrQkZCEzneOviMkeuDeZi+/FuFq+Tueo3TgWPOXgXs3cBx74gXkD0g2XOLYXBIXyZzLasHX5C/EhRaXrpM5KhsFK4zTKlk1hRHOcxRVDVlYFPi5WWyB02arqzBGmuBrFKa69D/JvFoyqeO6QZ3xG0ripWg68D2WMwuj73GB1zlvcOC4C/qPZWehqIw/abkM5/hZEbn3A/e2wLhiV9nLLUphdMW4sfseenJlxwuUV/vccP998fBtMv93NP37P/81HM3y4T/w0OfV/AjTNR+RNNyggne52ohxog+KoCp74E6fl0mFvq7CCHKfCTSIbFllKc9e4Lx6CLMkBVT8htIfqEqiEDKSNL3FKS5opa7rPjy4LpGn4Qyln3GZVAnOSV5Enol0wp3+QAXcmj62ClQYmgzTJO4s/olnzHBV4Yz3iIPP9ki6JH1P8vgPqObOqgVTfsPd0PHEbXwQ4MEwlKrgqcZwEx870ih66dWKLXVNrAjhDFXFhhR5EUjj8OAGFPDkcw1GezTiwqUCRH/MhSG3gFjWXYOAXHAcvAYTloaJLMyTBSqr88AiCKbOw8P7hUWchmXJKzgCRAKrCRHb68CI5XRgxAuOgJHReBLYtj9ZWAGajNF4aI80fCRkoOMirJAGkPI5ydIwRzoS1AHcqTGm2GiZpPPHcIPX8IBlRTywSE2XuEj+ItWGog2SXQjlOkGjxFe4kyuoQCUp81low5aix7CseJkIp2m4KpOZ7HAWFnGSS1hQqPAnfWiCmc8JpFZpKrS0mA1Y/VkS8WuK+amcW0RNOaZjWFYF/i5nKrtlO7xQv50pcHd6YNgB1n7nNZrcTLwmOMcd4Jz4OjYliruwyVv8Qmb8MI/JqMsmXb/ZnOtozY07LMEJmo2FKcFZTgA7hUEuNXuQz/o2E3E8zUSiAlH7sHL0TPlMliUV1JFTsiD8q1WgFR44QQqYnpF5OIgr6UYUuyIaqpo2xaDRhoKODmFrKVpUvZbG/d4jLXOnuMYvfEy9HnePSX2LlCJ7mcznKKfIr8IqZMYDEF9h4i+oIvwp+SHqurVu/IFPnuuWpO06TX6geFHd4pw8X5hQqCJimc8wWmJcVBD3u6vdsN40wbILwp53Cu860aCjKT9N2g51L7+wQ/MZ0VeK2rPg0Nbg4OpwcDvU3J69C1a2pf6LaHi0p5Man0DBjj59Upt3P0E4lFRk2IZRorsAolBECwVhBpaWz0r4I6feJ3KFn4QnuVlt9CqKNQmiCtZSki8KMsrFOqrWhT5vG/9yQvSN90Nf4JyI4dv69DRbE2pl2Ns7Y28HxRW2G7QCi5+Ru3UaiKdPwsR3wtrUEp5ng6MKwxJQp9EY16m4zh0I3eKeroKedfcvMBTtLJo+I03r7t42qlbHbHsytRtJ7obUcdxsCDa2cjRarDQO5tywuzg/06efaIkI6TH87GfiZyPnvPws8JvtyVdX10fQHP0dBbcQK1lIniZiY5jciDctwjxagph0yVqvoBiwuHnXurXxsYf62Ml1kzhHf/NpSNwpNH1pEif0+lYSZ1HH8qT4kifmSXYusZnw8CJ4uzR7Ey9/FJ2jeYwEIyNjssQxzsP0vpa2+Etd5hFT7wAw+RNV1YYTjHBNpq4GiNBLUn1Trv8LVRFdsNTdC6+ZJjYikZPn/aYmlLsgWd9GU+K+XgrEHhyedjvnIYOD10WE9pjhCc+MUd/aq9ND5AuUEkv90exFLxn6VBThRinAUV7X/BkECjGbBG2iNFZRs/sG27K23uAcWL7VIXLBnvG4HCwwON+JcxW+W3B+KIR1yPl+c7XVGTk3lvKv5f2YPfJKTsDX9ZWUDJGR0RBkItoPHNHK6JFj0rPMCwduH+M++7B+/8IiWSzDWmW3SEZA17Fb03DMAzkm81dXHL2aXSLn0fSlo1dXX+h63W4Ri/oRsznkPbmXi4ewrvteqH0vd5Gcf2gR3uu1iL/n7KD+NPUZFQkZULAKZTP3UcJbtyM+6NkLePTYQIsW/dY0xTp1smDA1ZdexXbYFuj2OTxwBF6qvfeY6PZndxBFfwsrPcz+3t8Skm5m9o01Gb/FzM6xvCRCgd3x+fhAGzwMCPridXfcfBlL8fwOenReS9EXFqIlCQNRimNtkMziwgdeXLCt9jkps51RGIm+uhAjQvbZgRS2qNBvNYbtH8j23StfTHDNYsJ5NH3pxQRv22KCdABD7htwIRYJyMjnDTgE/1vDsXWYizBfMijiWfgLqR9MjPzuvPobXML4WuD8hws+LdBVCpzjktp3o0hJJwgoYK1e6nbrY3F+++MEPhkPkNKD9TIlxsenI0Qkd3ANHfNhRHwyrLvK2rKswMGbqnHqapgaZA7fLiBy2WjLXMldiISZIWTZNFkzmFrGLA/SnMeAsGl/kAk+GHJUNgNylc/QFmtGU7dRsxqQgR/zdWYDWRbNabCbuhaF4dRlVZZTF20wnbqwznbEsMlPMvg151HuY7xHbRW4T53u5D+QTd0bG4LaCyoKaXREFla4EMhrNiSVKMbSVmHCHNE+QJRZEoO1E/KBuMiSwbgGrbWp5a6tyJ/lY7HawVeJPO6v6kxPuZF4LpkRKz1odE2hVM2+uX5n1yjB6uma+jTtnlFcdnRMI166DdNkYxCbGdzCt1u822vx9EyJzxmYL8+V+HLrUI8voIysww+ovKwJvG481j6Czs/bPYQ6IYPcU6Vf5JAzscLXQEgZG8glZwMpZ23MNsVEXhuf2IbBntqf0l9Uz5SzkRQdjVsqt9ty+kub56EyOdND/pvMTOKw28wUzKrYe401Adfbx5g6wLoXJpXqW5jsBR1lgL2g29MD7oU4zgoVeP3BMAjMsAuKbjcU3V5oNXkiyAtxVyfsrgJLozd65vFpoaS03HZv9FMGPtA3X31r7VNaCSMGb598yuU4ZpAsT8/OgJSenoGL7vMzfZXx11myeeNRP4ZHHSlwfo0VBIpnO8gMqLAZXfByzTBEBE8kUmPxkwhZ9G97mChrH4yYKMsxUZaJsvZzYybKkjkmyvronMBEWW9AnImyTJRloizjUd93lHXfE2apxzf5e60n+V5rv5Ob5sMbZ38/eul9r57f8YK0IrNW3HHIwuwm+sC7iYLWHllzVEnaiP71K8VGLPEZjj/xzDjQoztQ5qCudyuRp2/HNFuJTqHpi28l0rcUvvJcknQaT1nxtEpWKQJgPIHfMOeTrtvNXJyniQMr7+h8xLE+PdD+LyvYQLz2TETvF+t3nodwL3oeQnzUwCj+qN9aGe+pfMb9Lqb8n/eDIydUvuDz1275I6P84yvf3dvyDz2K2qP8AV/lU0LUen3Pvf8/</diagram></mxfile>
\ No newline at end of file

docs/automatic-manifest-integration.md

+<!----------------------------------------------------------------------------->
+# Automatic manifest integration
+<!----------------------------------------------------------------------------->
+
+**Contents:**
+- [Defined source code state](#defined-source-code-state)
+- [Integrating project changes](#integrating-project-changes)
+- [Merging project changes](#merging-project-changes)
+- [Adding a new project to SRCREV.conf](#adding-a-new-project-to-srcrevconf)
+
+
+<!----------------------------------------------------------------------------->
+## Defined source code state
+<!----------------------------------------------------------------------------->
+
+### Manifest
+
+Our [Yocto manifest][1] contains a list of projects which are used to build our
+Yocto distribution. A simplified version of the manifest file [`default.xml`][2]
+(here containing only two projects) would look something like this:
+
+```xml
+<manifest>
+    <remote
+        name="seco-ne"
+        alias="origin"
+        fetch="ssh://git@git.seco.com:4444/seco-ne/yocto"
+    />
+
+    <project
+        name="layers/meta-guf-machine"
+        revision="7aede82170ff7f92de0dcfeca89ed79849869214"
+        remote="seco-ne"
+        path="sources/meta-guf-machine"
+    />
+    <project
+        name="layers/meta-guf-distro"
+        revision="c402d855fc8d47e3742ada3b0a3fda4b9649e414"
+        remote="seco-ne"
+        path="sources/meta-guf-distro"
+    />
+</manifest>
+```
+
+Each project is listed with a **name**, a local **path** where the [repo][3]
+tool will clone the source code to, and a fixed git **revision**. We do not use
+a branch name here in order to have a defined, taggable manifest state at all
+times.
+
+### External source repositories used by BitBake
+
+Some bitbake recipes are set to `SRCREV = "${AUTOREV}"`. To enable reproducible
+builds, the revision for these recipes is written to the `SRCREV.conf` file.
+This file is also part of the manifest repo and looks something like this:
+
+```ini
+SRCREV_pn-libmdb = "1442894df13d9b290cfb5d97183eb9a96c8e4eba"
+SRCREV_pn-egalaxi2c = "d2dfb014e8c2ed36801bd020894d8306f02ff146"
+SRCREV_pn-xconfig = "0fc1ea45b55e729d551bb7d40dd25fbde02ee1b6"
+```
+
+Each project is listed with its bitbake **recipe name** after `SRCREV_PN-` and
+the fixed git **revision**.
+
+[1]: https://git.seco.com/seco-ne/yocto/manifest
+[2]: https://git.seco.com/seco-ne/yocto/manifest/-/blob/dunfell/default.xml
+[3]: https://gerrit.googlesource.com/git-repo
+
+
+<!----------------------------------------------------------------------------->
+## Integrating project changes
+<!----------------------------------------------------------------------------->
+
+Everytime a developer pushes a change to one of the projects contained in the
+manifest or in the `SRCREV.conf` file, the project gets a new git revision. In
+order to have this change integrated into the manifest, the according `revision`
+attribute has to be updated. This process has been automated as part of our **CI
+pipeline** and runs automatically as soon as a **merge request is created**:
+
+![Project merge request pipeline](project-mr-pipeline.png)
+
+The pipeline consists of three jobs:
+
+1. **Integrate**
+
+   Create an integration branch on the manifest and update the project revision
+   on it.
+
+2. **Build**
+
+   Trigger a build on the integration branch, which contains the change.
+   This uses GitLab's [multi-project pipeline][4] functionality, which adds a
+   separate *Downstream* pipeline running on the manifest repository (seen on
+   the right).
+
+3. **Check**
+
+   Check if the integration branch is up to date with the current manifest
+   master (see ["Retrigger" job below][5] for why this is necessary).
+
+See the following diagram for a visualization of this process:
+
+![Create layer merge request](create-layer-mr.svg)
+
+Browse existing project merge requests (e.g. [here][6]) for real world examples
+of this pipeline.
+
+The above workflow makes project changes **buildable** and **testable** in the
+full manifest environment **before merging them**. Actually we even prevent the
+project merge requests from being merged unless the pipeline has run
+successfully.
+
+[4]: https://docs.gitlab.com/ee/ci/pipelines/multi_project_pipelines.html
+[5]: #merging-project-changes
+[6]: https://git.seco.com/seco-ne/yocto/layers/meta-guf-distro/-/merge_requests/321
+
+
+<!----------------------------------------------------------------------------->
+## Merging project changes
+<!----------------------------------------------------------------------------->
+
+As soon as a pushed project change is ready to be merged (i.e. the pipeline has
+successfully run on the merge request), a developer can merge it. This triggers
+a **pipeline** with only a **single job**:
+
+![Project master pipeline](project-master-pipeline.png)
+
+1. **Merge**
+
+   Create a merge request on the manifest, which merges the project integration
+   branch into the master branch, and merge it.
+
+This will however automatically trigger **another pipeline** on the **manifest
+master branch**, since there is a new commit now:
+
+![Manifest master pipeline](manifest-master-pipeline.png)
+
+This pipeline consists of two jobs:
+
+1. **Retrigger**
+
+   Retrigger the *"Check"* job on all open merge requests of all projects
+   contained in the manifest. This will invalidate the pipeline on these merge
+   requests. The manifest has a new commit now, so the already completed
+   pipelines on the open merge requests have run on a now outdated manifest
+   state.
+
+2. **Build**
+
+   Check if a build has already run for this commit. This is usually true,
+   because every merge request triggers a complete build. If so, just mirror
+   the build result from there.
+   There are corner cases however in which no pipeline has run in the merge
+   request, e.g. when two project merge requests from different projects are
+   merged simultaneously. In this case, perform a complete build on the commit.
+
+See the following diagram for a visualization of this process:
+
+![Merge layer merge request](merge-layer-mr.svg)
+
+Browse existing manifest merge requests (e.g. [here][7]) for real world examples
+of this pipeline.
+
+[7]: https://git.seco.com/seco-ne/yocto/manifest/-/merge_requests/545
+
+
+<!----------------------------------------------------------------------------->
+## Adding a new project to SRCREV.conf
+<!----------------------------------------------------------------------------->
+
+This workflow adds a gitlab pipeline to a project which automatically updates
+the git revision in the `SRCREV.conf` file on project changes.
+
+#### Source code
+
+1. Add the gitlab-ci repo as submodule to the project by using the correct
+   relative path:\
+   `git submodule add ../../yocto/infrastructure/gitlab-ci .gitlab-ci`
+
+2. Add an approriate `.gitlab-ci.yml` file. For example, copy it from the
+   repository [egalxi2c][8]. Modify the following variables in the file:
+   * `MASTER_BRANCH_PROJECT`: Set the project master branch
+   * `BB_RECIPE_NAME`: Set the name of the bitbake recipe
+
+3. Create a corresponding entry in the `SRCREV.conf` file of the manifest repo:\
+   `SRCREV_pn-<bitbake-recipe> = "<current-commit-hash>"`
+
+4. Add the project into the gitlab-ci project's `.gitlab-ci.yml`, so changes to
+   the CI are deployed into the new project.
+
+#### Repository settings
+
+1. Verify that the merge strategy is set to *Fast-forward merge*
+   (General -> Merge requests -> Merge method)
+
+2. Enable the CI/CD functionality for the repository
+   (General -> Visibility, project features, permissions -> CI/CD)
+
+3. Check that the default branch is protected and that *Maintainers + 
+   Developers* are allowed to merge (Repository -> Protected branches)
+
+[8]: https://gitlab.com/seco-ne/kernel/modules/egalaxi2c

docs/create-layer-mr.drawio

+<mxfile host="app.diagrams.net" modified="2022-05-10T15:50:28.875Z" agent="5.0 (Windows)" etag="tZCYtCye2vNUvOPvtp5c" version="17.4.6" type="device"><diagram id="fUzyECxa7j4ng1UTdmFO" name="Page-1">7V1bc5s4GP01nm0fksHcjB9jJ253pp3JNNvd9lEG2dYWgxfkxO6vXwkQBkkOlyIHu05mWvgAIXSOvqsgA2O63n2IwGb1OfSgP9A1bzcw7ge6PrQsjfxHJftU4thWKlhGyMtOOgie0E+YCbPrllvkwbh0Ig5DH6NNWeiGQQBdXJKBKApfyqctQr981w1YQkHw5AJflP6DPLxKpSNdO8g/QrRcZXe2tOzAGrBzM0G8Al74UhAZDwNjGoUhTrfWuyn06dixYUmvmx05mvcrggGuc8HnL39P/vx6Ez9/woExdPy7h923m7GdNvMM/G32wFlv8Z6NQBRuAw/SVrSBMXlZIQyfNsClR18I5kS2wmuf7A3J5gL5/jT0wyi51phN6S+RLyPgIdLTexQRlFAYkOMQxJheEga4cEn6Q+QxjsIfsHBk4tBf2pgP4jjrTz6syc4PiN1VtgMiN6PTMG+NQajdWkQmDmE2qs8wwnBXEGVD+gGGa4ijPTllV6bpvrz7UqAKY8SqQJOxkwlBRs9l3vIBQrKRodgA0ZGjGFFrNptNZPA4+tyw7QLW7FgQBvB1CnQAhO6UkRiZugCFKYNiaKmCwr5CkenEmlDk06d7KMYCFPMtIqaAx4M8Ii4P+vHB5eB4mMzuHu6p4vHRko6oS4YQkoMTOnSImJS77MAaeR69oRTvMiNOBIctQ0MVGEMBi4FugzV9+GAeb5Kn5vc1iegePkM/3JAhPj6rho1nlQegs3BlwNuuA+eLbkAxy5iYot0YyjAxVWGiV6sqGHh31KE6qBEPxKt8lIUpw2ytLpCaDFu0/5ab4WT3e3ZtsnO/K555v2fmXfAIkt/XEMEgWkJcTUboldxAEbcCMpYEGCaLoA8wei47jzK0sjs8hoj0+OBO6GVejG0O8DjcRi7Mrip6e1xDJkcw4oaUG0oHRmgoIU/+2O35ZMjmuE9Vq4eeyeaSbpKbgJv13gd7OofTw/OIHWUScv/CNWc413kF/OaT3bw6Jum+ZdR1TMaqoBCQWIMALWCMz5Dolt4zoltdWjUqfwSYuHRBItG1Q5SqwtaBCHM9a2/ljF5ZOcMu82RotLRyxsgsN8QHEIqtXI3sSZ+9pkYcSzF5TZVpNclo9oqMtsaFR3zcU5eMll4mo6DWFJNRDHEvl4w94Q6vyFq764JG5JNSirnD9GZN8rg0AYvcMmWOEaRID62CHr9g4ca9YgYfyA3bahWBGXxDqpkhZgYet/GKSOYRCNxVmppJAjVUiNz+29K6xgR43s0CAryN4EFYCOzQ0bBOTMTVT6tFMEY/wTxpirJqQwcnGS5rMrCSFN0WhzFfHOBikaILXaWPGhQMWP2LqYwsFiv6yDJeqsvGKS4BLRwXutLQY+5YptVRktOwytNEZ09RGFddMqx5dab7cR2du0plzOipSs0h/lWVKjSkWqU289RqMOPf7XrzlF0sc+zgDuGcOGT7e0F+oA3dqWRNZRjQM0Ms2M+2Lppg0fmGFLNGF1NFkpTqNCLWltha7fOXAV3tEHVvqFvmaS/WoBucNmEWpWR5TmnR9WZprx5aHhY590WHjMoQG5rRTofwuQahIdU6RObrHdchTHEcm/ESPUI6DpdRcvmsXN6ZNVYxv6lCMfgqBVu8UFQoQwnb+SxqZwrFqFEvOl+HpWfKRtARZktlI2gtviHVJeAabu75BZa5G1gRWKor3ooJma8bL1XYgmKWVdjLqpbQ9RnFtNJK7hsc1/XFsn1eodRwSP75+HBHeqGFi45dTF7/Cx15l/Q6gC9FJ3cRhWs67vPwGb6Xdee4+3q1QBnnx5zmsMRkynD8igrsnvR9d2l7Yjx4nW+29VStcUVDio2H2ay02i+XoydkEDDsypMQGlJNhhrVqf57EjwcefD3Zp5Eswz1eVeMK8MAO8uvnVtygo8XRo7ebpbzDdlcO4onuSVq/EIyodLzurhl+rLFiSddpm/XWsP7V4SWSxpcaBu0gT4KaBxSzju/UfromqE+klCydInhOWlCyb6687V0O+94WdaonW7nfQ+hIdULJa/ufOfuvO0Mb8fFn3bU4DlW0axiooxkL4d1pBluHb3keWq3pj6u0BDJ3iOMEHkwagF+sbDFXgntCaVs7vVly26pX2yuOG6N6zmPBEewL5yWmc3jHTbkHT7aL/78rF8H1qY96FbZySr1Zx+u5iajIlwdsvJz96+26410w3nHq9WqpN/L6FsvhZbUv2614o99WoMki4KUGSTD7tQe9YUbnJmxtbZmhm/IOI2ZYR2ua2ZYv5SamVENM3NlZkNmmpx2ac3MEdcXVczMOlybmVm/1DJTDO/bfW3jeFqkZ8k7my12YM4SG4IqZ0lVhmUkBtxY9lLxBSFgl4k+NG/fGgNZfUVIoHpw44f71inKc4WLlRAZXJJs92nBElcruSvo/mgMwQWUHmTfzjpp6YH1pwDGNAVDQ3SVUe8qC4U5i2LaQRATgHQtWaDkbqMEHPpRwJjg/Qc9I100xbf3DvgvYE+P42gLWQMLFCWrreAOulv6VYz31xpHRY1jZNVcNMuHqd1xWGEm85JrHI5ezj875QbbVjwqmlWcRWA1+Wv9o7v6h2N3QhSBf683q5ooF5k7dthTvNVSp7HoUFznX1NFza2EH3ejqIVP3jFoTvXZlBp2Ol6BDd3crv07F4dFL+kTmEP/kXg92dfK5iHG4VriRuGQm5nhFtPlMtP8w9XaKxHDm3wvTfhwqDCJZR6VsqiA1RQl8fTBRbcmWapJA2T8F8AlHqol8bR/649QmEb5IxTWSIRWtr5fHbTNqizhBganVc7k5jNEn+ksVLVQu+XL8W2Vc926fmfK+fhL78Upn6Y2yajFW/863yXznfsarC35Tr1z0vnebAHgdb43XBrG10Xbzve6BdbO3vS4yPDHVPemB9k9/AmRFIXD32ExHv4H</diagram></mxfile>
\ No newline at end of file

docs/gitlab-ci-deployment.md

+<!----------------------------------------------------------------------------->
+# `gitlab-ci` deployment
+<!----------------------------------------------------------------------------->
+
+<!----------------------------------------------------------------------------->
+## Deploy changes into all projects using 'gitlab-ci' as submodule
+<!----------------------------------------------------------------------------->
+
+The gitlab-ci repo is integrated into the other related projects as a submodule
+and as reference in their `.gitlab-ci.yml` file. Both references need to be
+updated when a change in the repo shall be deployed to the projects using it.
+
+To achieve this the CI pipeline is set up to update all these projects in one
+run. The submodule *does not* need to be updated manually, but the CI needs some
+manual steps to do the deployment.
+As this repo contains all the CI code, some extra care is needed in order not to
+break the whole thing. To allow testing the CI before everything is merged into the Yocto pipeline, a sandbox environment in the form of a set of test projects is
+available in the [ci-test group][1].
+
+[1]: https://git.seco.com/seco-ne/yocto/infrastructure/ci-test
+
+
+<!----------------------------------------------------------------------------->
+## Workflow to deploy a change
+<!----------------------------------------------------------------------------->
+
+* Commit and push change to the gitlab-ci repo, create a merge request.
+
+* The pipeline checks syntax and other stuff in the *Analyze* stage.
+
+* The *Integrate* stage allows testing the change in the complete setup.
+  There are two possible integrate steps, one to deploy the change into Merge
+  Request in the *ci-test* group, the other is for the *productive yocto* setup.
+
+* When the integrate stage has run, a build is started in the related manifest
+  project.
+
+* In all projects there are now integration branches created, named
+  `integrate/gitlab-ci/<gitlab-ci branch name>`.
+  These could be used to add additional changes if needed. But that should only
+  happen in very special cases.
+
+* If everything is OK until here, (build OK, CI works as expected ...)
+  the change can be merged into the main branch of gitlab-ci.
+  *This only adds the patch to the gitlab-ci repo's main branch, the reference
+    in the projects using this are not yet updated*
+
+* After the merge a new pipeline is created. In this pipeline there are two 
+  manual jobs: *'merge-ci-test'* and *'merge-yocto'*
+  These jobs update the reference in the project groups *'ci-test' *and
+  *'yocto'*. This is split up to allow more testing in cases where the CI
+  changes can only be tested when working on the master branch.
+
+* When the behaviour of the CI on the master branch may be effected of the
+  change only *'merge-ci-test'* should be executed. 
+  After this some tests may be done on the ci-test projects.
+  Probably you want to create a commit in *minimal-foo*, create a merge-request,
+  check the running pipeline, and merge it.
+  After that, check if the merge is succesfull and the manifest is updated as 
+  needed.
+
+* When everything is working the *'merge-yocto'* pipeline can be executed to 
+  deploy the change to the productive projects.
+
+* When the tests on *'ci-test'* have failed, just create a new patch fixing the
+  issue in the gitlab-ci repo and do the testing again.

docs/merge-layer-mr.drawio

+<mxfile host="app.diagrams.net" modified="2022-05-10T16:32:03.192Z" agent="5.0 (Windows)" etag="7UJfYfomj_BLGc_u5cve" version="17.4.6" type="device"><diagram id="fUzyECxa7j4ng1UTdmFO" name="Page-1">7Vxbc9o4FP41zOw+JOM79mNIQrczzU432ey2TzvCFqDGWNQWBPrrV8LyTRJgDCYmbZhp8bEly+d8+s5FMj3zdrb6EIP59AEHMOwZWrDqmXc9w9Bdw6L/McmaSzTdTSWTGAVcVgie0A+YXcilCxTApHIhwTgkaF4V+jiKoE8qMhDH+LV62RiH1bvOwQRKgicfhLL0XxSQaSo1Ha2Q/wHRZMrvbNn8xAxk13JBMgUBfi2JzPueeRtjTNJvs9UtDJnyMrWk7YZbzubjimFE6jR4ePxn8PH5Kll+IpGpu+HN/erLleek3SxBuOAPzEdL1pkGYryIAsh60Xrm4HWKCHyaA5+dfaVGp7IpmYX0SKdfxygMb3GI401bc3jLPlQ+iUGA6EjvUEythHBEz0OQENYER6TUJP2j8oTE+AWWzgxc9mGdhSBJ+HhytW4OXiDxp/wAxD6Hk573lplQu7apTFYh1+oSxgSuSiKu0g8QzyCJ1/SSVRWm6+rhawGVvsZl0xJMXJcLAYfnJO+5MCH9wq2otuhf9vLP74MX/HKz/O/549/P4PFJu9L1li1qD4fDgco8rjEyHadk6+xchCO4GwJbtK6wzVZDGG7VEn1DNoWlMoV3Akso55bjXqwljpoSoiVsp6YldLM1U3iSKWLaMZpMYCzZhD4mqSp+u4IFk9wPhjf3d4x7QjRhWvWpGuktzAFTH6Je5YafmKEgYDdU2ryKihOYxDH2m8RRWMRoyyAqknLAjD18NErmm6cWjzWF6A4uYYjnCisWOtQPnlkBgO7YVxne8V04Gp/GKFbVJpZsEl1lE6stmxj76QpGwQ2LqQoqCUAyzbUsTZnM3RoSqKna4vWX3BNvDr/ytpuDu1X5yrt15uGloGDz2WURAuIJJPvBCINKJCjbrWQZW2GYTBbDEBC0rMaPKmvxO3zGiI64iCiEuer2BYMneBH7kLcqB3xCR5YAMEPsKFWM1NEGPPljN8eTqZrjIaPWAC3p1wn7Sm8CrmbrEKzZHE5Pj+LsbCah9y+1ucC5LvrEN5/s1q/gJD22zXrBiaF7bZlCssQMRGgME3KBQLeNjgHdvgyvBmIijKENT2d2ytPlRZQseRbj/7qezuxb1Y7083q6GkWULmBsG55U6NuKsdQmu+hMqwlGq1NgdLQqGG0x96kLRtuoglGitpbBKKe67xeMAnaq6FSWx6wuEZ7nNCU8kTnt82Is49eaIPNZvRb5VWhtA1IZRtoeGB3hCb1OsY+Y9OlN2UdChthR28iQqwi3MQQEUtnDY48twNDsThvFIPKnaU1nk+GhUsr3fcHWRAYgCK7GtOkihoWwlBGirfmgXMGrX4+LYYJ+gNGmKwaxOdPURnf2oGdvansLghNxYUFIYsqx9z4SO2CxQcjoPZ7ElTBqKHOX1sp4LS8fjV0f+sqcZeTabKntJGo17apajewpKnpVqLXvtqXX/qXza4aMjvJrbuJj+VXqqG1+PSy8q4GMb4vZ/Ik3VkWDcIVIDhz6/WtJXsCGHexFzd7coWNeWXKmTeM1yb2LHbWMGkOuMSlqsQ+QDuLEflpZy807oY+LaefVIrA2Awk5ohr8br2/aVdB5HqSm9I9xWxobVnVOKy41kE3lQVUXSGcfpUnTM1sRjhiNUPqqG3CUQWGEuHsygy2TX4FAdFngJN409OwSiXDA3KIn5pbTHFZxJG5Jd/cdBZuMWssUF1uoNMx3pHowmrIOxKBiR21veZcIzy+vIQ0Dx/3JKTtrRbLVZ3D4seOsLk05CwAzRdc9wafqnv88iJqL2LaikKKyouI5evTAbfrEWpHHIDI21bTwNP29nTUsgOwDluL7VbY0BEwSDY8VTQgddQ2GGosU3U/GhDNkedybxYNHFadvuwl5r2hvMNra5dWaxBj/r5rNJvlYkeO0E/Lk9yWGX+WBqJ7o66L39dfZzfjWff1O6pC88Uzrm3Wy79Mcd/ZyfTaVyVg75Vy9zJphrKOMmnjHfuKMsy1Vv5zzsqsXo3w6VcsvSexEko5ludde6U/txlSxAVFW9POC40ab5kmUzBnXxez8MYnuOwtP4ERDD/jBPH9/SNMCJ4p3CnBgj/ACxKiiHJM9ra3tsNbv8kbBuLbj4oN8Of0yNa79MjWGXMg9WvWB1EjnsPoQF6krYeIjanqkXMXXL/qJG2Sbe5y38zDCq/ZW5m6D/awAm86NXmTGhKsS5fxquv2AZvqAW8dl6mOOAuIpiNoSuLb90J3gBmOeu9fdLBO9hRvxQzv9mWQQ3mk+/vvGwfqZ9x/r9RiowLciZc0KkApYv+NmHsuXfJjdZHUlXWQnE2OXQeROmoZIfLuicfiBy+0YvXUn0L/pbyYqn3DI/ovDYrpWDf32QQv0noor7FRhX9fwIT5Qu23dM03bTWP8TcaXSdySx9HBNAYPmDPwG5DpqyjbA32dwnKHV4uPc51iWWmvi65Lt3dAfbTR7Udea+n4g4vjjPEAnhjzmivkq62viFZf7RAYVBjQnahkn7UXBQTA9XPR7VVSVcbQ/XzGUdNxZ1Jpl6Zq9f9fiWcZAJ3z5zNa3dZY16/065NNztW1/DYwWcYI6o4hoSib1FYlwlqxKF6p1bY+457rdmnYQ1X61979QrFh2a1liMNe+fQpOu9M2S1upzWPiA6S1howumMmi9Z0KhECk3GMZ71+Aa2OZpDVmr8acIRS/gVhb5pyeGI6vXCBnuA6WHxy5ep5YvfDzXv/wc=</diagram></mxfile>
\ No newline at end of file