Skip to content
Snippets Groups Projects
Commit 264671ec authored by Lorenzo Pagliai's avatar Lorenzo Pagliai
Browse files

[DOC] Update documentation to latest code changes

* Update new stages description in manifest pipeline file
* Update layers integration description in a different file
* Update images
* Remove unused yaml files
parent eebd1a22
No related branches found
No related tags found
No related merge requests found
......@@ -2,21 +2,21 @@
# gitlab-ci Repository
<!----------------------------------------------------------------------------->
This repository contains **GitLab CI scripts** for Yocto infrastructure tasks,
This repository contains **GitLab CI scripts** for Edgehog OS infrastructure tasks,
e.g. automatic integration of project changes into the manifest. It is meant to
be included into all relevant Yocto repositories as a [git submodule][1].
be included into all relevant Edgehog repositories as a [git submodule][1].
[1]: https://git-scm.com/book/en/v2/Git-Tools-Submodules
<!----------------------------------------------------------------------------->
## The EdgeBot token
## The GitBot token
<!----------------------------------------------------------------------------->
Most scripts inside this repository need a [group GitLab access token][2] of
a user with access to all relevant repositories (manifest and all contained
projects) in order to work correctly.
We have created the **EdgeBot** group access token for this task. The token has
We have created the **GitBot** group access token for this task. The token has
maintainer permissions in the [`edgehog` group][3] and has all of the token
permissions enabled (read/write API access, read/write registry access, etc.).
......@@ -33,12 +33,21 @@ The group access token value is provided via the CI environment variable
## Automatic manifest integration
<!----------------------------------------------------------------------------->
See this chapter for information on how the automatic integration process works:
See this chapter for information on how the automatic manifest integration process works:
[Automatic manifest integration][5]
[5]: docs/automatic-manifest-integration.md
<!----------------------------------------------------------------------------->
## Automatic layers integration
<!----------------------------------------------------------------------------->
See this chapter for information on how the automatic layers integration process works:
[Automatic layers integration][8]
[8]: docs/automatic-layers-integration.md
<!----------------------------------------------------------------------------->
## Deployment
......
<!----------------------------------------------------------------------------->
# Automatic layers integration
<!----------------------------------------------------------------------------->
**Contents:**
- [Automatic layers integration](#automatic-layers-integration)
- [Source repositories used by BitBake](#source-repositories-used-by-bitbake)
- [Integrating BSP changes](#integrating-bsp-changes)
- [Merging project changes](#merging-project-changes)
- [Adding a new project to SRCREV.conf](#adding-a-new-project-to-srcrevconf)
- [Source code](#source-code)
- [Repository settings](#repository-settings)
## Source repositories used by BitBake
Some bitbake recipes, more precisely those regarding the Board Support Packages (BSPs) references, are set to
`SRCREV = "${AUTOREV}"`. To enable reproducible builds, the revision for these recipes
is written to the `SRCREV.conf` file contained in the `conf` folder of the
corresponding meta-layer. For instance, the [meta-seco-rk][4]
meta-layer recipes contain the reference to the Rockhip Board Support Package (BSP)
repositories and the corresponding [conf/SRCREV.conf][5] file looks something like this:
```ini
SRCREV:pn-u-boot-seco-rk = "7d205eec8c6d3e6d3e08390b3c276f9fafb67489"
SRCREV:pn-linux-seco-rk = "e6962a7c84438f36487ca8519f2b93b45efd59a6"
SRCREV:pn-linux-seco-px30-rk = "645936464a1d6641c2bab5686045d4fce281fdd3"
```
Each project is listed with its bitbake **recipe name** after `SRCREV:pn-` and
the fixed git **revision**.
[1]: https://git.seco.com/edgehog/seco-manifest
[2]: https://git.seco.com/edgehog/seco-manifest/-/blob/kirkstone/distro/seco-distro.xml
[3]: https://gerrit.googlesource.com/git-repo
[4]: https://git.seco.com/edgehog/layers/seco/meta-seco-rk
[5]: https://git.seco.com/edgehog/layers/seco/meta-seco-rk/-/blob/kirkstone/conf/SRCREV.conf
<!----------------------------------------------------------------------------->
## Integrating BSP changes
<!----------------------------------------------------------------------------->
Everytime a developer pushes a change to one of the projects referenced in the
recipes, the corresponding `SRCREV.conf` file needs to get a new revision.
In order for this to happen, a similar process to that described in the
[Automatic manifest integration][6] document occurs. This process has been automated
as part of our **CI pipeline** and runs automatically as soon as a
**merge request is created** in one of the recipe repositories (i.e. currently
those in the [bsp group][12]):
![Project merge request pipeline](recipe-mr-pipeline.png)
[6]: automatic-manifest-integration.md
The pipeline running in the recipe repositories is described in the `layers-integration.yml` [file][11].
[11]:https://git.seco.com/edgehog/infrastructure/gitlab-ci/-/blob/master/layers-integration.yml
[12]: https://git.seco.com/edgehog/bsp
The pipeline consists of three jobs:
1. **Integrate-into-layer**
Create an integration branch on the meta-layer and update the project revision
on it.
2. **Create-merge-request**
Create a Merge Requeste (MR) in the meta-layer from the newly created integration branch into the main one.
The creation of this MR automatically triggers all the CI jobs described in detail in the
[Automatic manifest integration][6] document.
1. **Check**
Check if the integration branch is up to date with the current meta-layer
master branch (see [Automatic manifest integration][6] for why this is necessary).
See the following diagram for a visualization of this process:
![Create recipe merge request](create-recipe-mr.jpg)
The above workflow makes project changes **buildable** and **testable** in the
full manifest environment **before merging them**. Actually, the user could even
[prevent the project merge requests from being merged unless the pipeline has run
successfully][7].
[6]: https://docs.gitlab.com/ee/ci/pipelines/multi_project_pipelines.html
[7]: https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html
[8]: #merging-project-changes
<!----------------------------------------------------------------------------->
## Merging project changes
<!----------------------------------------------------------------------------->
As soon as a pushed project change is ready to be merged, a developer can merge it.
This triggers a **pipeline** with only a **single job**:
![Project master pipeline](project-master-pipeline.png)
1. **Merge**
Merge the MR on the integration branch on the meta-layer created in the previous
step. This trigger a cascade process according to which a MR is opened and closed
in the manifest (see [Automatic manifest integration][6]).
<!----------------------------------------------------------------------------->
## Adding a new project to SRCREV.conf
<!----------------------------------------------------------------------------->
This workflow adds a gitlab pipeline to a project which automatically updates
the git revision in the `SRCREV.conf` file on project changes.
#### Source code
1. Add the gitlab-ci repo as submodule to the project by using the correct
relative path:\
`git submodule add ../../../infrastructure/gitlab-ci .gitlab-ci`
2. Add an approriate `.gitlab-ci.yml` file. For example, copy it from the
repository [linux-seco-rk][9]. Modify the following variables in the file:
* `MASTER_BRANCH`: the main branch of the project.
* `BB_RECIPE_NAME`: the name of the bitbake recipe that references to the project.
* `LAYER_PROJECT_PATH`: the path to the meta-layer repository that contains the recipe.
3. Create a corresponding entry in the `SRCREV.conf` file of the meta-layer repo:\
`SRCREV_pn-<bitbake-recipe> = "<current-commit-hash>"`
4. Add the project into the gitlab-ci project's `.gitlab-ci.yml`, so changes to
the CI are deployed into the new project.
#### Repository settings
1. Verify that the merge strategy is set to *Fast-forward merge*
(General -> Merge requests -> Merge method)
2. Enable the CI/CD functionality for the repository
(General -> Visibility, project features, permissions -> CI/CD)
3. Check that the default branch is protected and that *Maintainers* are
allowed to merge (Repository -> Protected branches).
[9]: https://git.seco.com/edgehog/bsp/rockchip/linux-seco-rk
\ No newline at end of file
......@@ -4,48 +4,34 @@
**Contents:**
- [Automatic manifest integration](#automatic-manifest-integration)
- [Defined source code state](#defined-source-code-state)
- [Manifest](#manifest)
- [External source repositories used by BitBake (Currently not used)](#external-source-repositories-used-by-bitbake-currently-not-used)
- [Manifest](#manifest)
- [Integrating project changes](#integrating-project-changes)
- [Skipping the build](#skipping-the-build)
- [Merging project changes](#merging-project-changes)
- [Adding a new project to SRCREV.conf](#adding-a-new-project-to-srcrevconf)
- [Adding a new project to Manifest](#adding-a-new-project-to-manifest)
- [Source code](#source-code)
- [Repository settings](#repository-settings)
<!----------------------------------------------------------------------------->
## Defined source code state
<!----------------------------------------------------------------------------->
### Manifest
## Manifest
Our [Yocto manifest][1] contains a list of projects which are used to build our
Yocto distribution. A simplified version of a manifest file [`default.xml`][2]
The [Edgehog manifest][1] contains a list of projects which are used to build
Edgehos OS distribution. A simplified version of a manifest file [`seco-distro.xml`][2]
(here containing only two projects) would look something like this:
```xml
<manifest>
<remote
name="seco-ne"
alias="origin"
fetch="ssh://git@git.seco.com:4444/seco-ne/yocto"
/>
<project
name="layers/meta-seconorth-machine"
revision="7aede82170ff7f92de0dcfeca89ed79849869214"
remote="seco-ne"
path="sources/meta-seconorth-machine"
/>
<project
name="layers/meta-seconorth-distro"
revision="c402d855fc8d47e3742ada3b0a3fda4b9649e414"
remote="seco-ne"
path="sources/meta-seconorth-distro"
/>
<default sync-j="8"/>
<remote name="seco-repos" alias="origin" fetch="https://git.seco.com/edgehog" />
<remote name="seco-layers" alias="origin" fetch="https://git.seco.com/edgehog/layers/seco" />
<project name="meta-seco-edgehog-things" revision="901e92783ffb6c5a36fee99efd57cb0955b53a18" path="layers/meta-seco-edgehog-things" remote="seco-layers" />
<project name="meta-seco-edgehog-embedded" revision="8fe5d96c85605645260baa6e78b198a3b69f607e" path="layers/meta-seco-edgehog-embedded" remote="seco-layers" />
</manifest>
```
Each project is listed with a **name**, a local **path** where the [repo][3]
......@@ -53,38 +39,28 @@ tool will clone the source code to, and a fixed git **revision**. We do not use
a branch name here in order to have a defined, taggable manifest state at all
times.
### External source repositories used by BitBake (Currently not used)
Some bitbake recipes are set to `SRCREV = "${AUTOREV}"`. To enable reproducible
builds, the revision for these recipes is written to the `SRCREV.conf` file.
This file is also part of the manifest repo and looks something like this:
```ini
SRCREV_pn-libmdb = "1442894df13d9b290cfb5d97183eb9a96c8e4eba"
SRCREV_pn-egalaxi2c = "d2dfb014e8c2ed36801bd020894d8306f02ff146"
SRCREV_pn-xconfig = "0fc1ea45b55e729d551bb7d40dd25fbde02ee1b6"
```
Each project is listed with its bitbake **recipe name** after `SRCREV_PN-` and
the fixed git **revision**.
[1]: https://git.seco.com/edgehog/seco-manifest
[2]: https://git.seco.com/edgehog/seco-manifest/-/blob/kirkstone/distro/seco-distro.xml
[3]: https://gerrit.googlesource.com/git-repo
[4]: https://git.seco.com/edgehog/layers/seco/meta-seco-rk
[5]: https://git.seco.com/edgehog/layers/seco/meta-seco-rk/-/blob/kirkstone/conf/SRCREV.conf
<!----------------------------------------------------------------------------->
## Integrating project changes
<!----------------------------------------------------------------------------->
Everytime a developer pushes a change to one of the projects contained in the
manifest or in the `SRCREV.conf` file, the project gets a new git revision. In
manifest, the project gets a new git revision. In
order to have this change integrated into the manifest, the according `revision`
attribute has to be updated. This process has been automated as part of our **CI
pipeline** and runs automatically as soon as a **merge request is created**:
![Project merge request pipeline](project-mr-pipeline.png)
The pipeline running in the meta-layers is described in the `manifest-integration.yml` [file][11].
[11]:https://git.seco.com/edgehog/infrastructure/gitlab-ci/-/blob/master/manifest-integration.yml
The pipeline consists of three jobs:
1. **Integrate**
......@@ -95,14 +71,14 @@ The pipeline consists of three jobs:
2. **Build**
Trigger a build on the integration branch, which contains the change.
This uses GitLab's [multi-project pipeline][4] functionality, which adds a
This uses GitLab's [multi-project pipeline][6] functionality, which adds a
separate *Downstream* pipeline running on the manifest repository (seen on
the right).
3. **Check**
Check if the integration branch is up to date with the current manifest
master (see ["Retrigger" job below][6] for why this is necessary).
master (see ["Retrigger" job below][8] for why this is necessary).
See the following diagram for a visualization of this process:
......@@ -111,11 +87,11 @@ See the following diagram for a visualization of this process:
The above workflow makes project changes **buildable** and **testable** in the
full manifest environment **before merging them**. Actually, the user could even
[prevent the project merge requests from being merged unless the pipeline has run
successfully][5].
successfully][7].
[4]: https://docs.gitlab.com/ee/ci/pipelines/multi_project_pipelines.html
[5]: https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html
[6]: #merging-project-changes
[6]: https://docs.gitlab.com/ee/ci/pipelines/multi_project_pipelines.html
[7]: https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html
[8]: #merging-project-changes
#### Skipping the build
......@@ -170,15 +146,11 @@ See the following diagram for a visualization of this process:
![Merge layer merge request](merge-layer-mr.svg)
<!----------------------------------------------------------------------------->
## Adding a new project to SRCREV.conf
## Adding a new project to Manifest
<!----------------------------------------------------------------------------->
**NOTE**: Currently the `SRCREV.conf` file is not used but is created in the
manifest repository for automation purposes. In any case, the steps below shall
be kept in mind to properly configure CI/CD on the yocto projects/manifest.
This workflow adds a gitlab pipeline to a project which automatically updates
the git revision in the `SRCREV.conf` file on project changes.
the git revision in the manifest file on project changes.
#### Source code
......@@ -187,11 +159,11 @@ the git revision in the `SRCREV.conf` file on project changes.
`git submodule add ../../../infrastructure/gitlab-ci .gitlab-ci`
2. Add an approriate `.gitlab-ci.yml` file. For example, copy it from the
repository [meta-seco-edgehog-things][8]. Modify the following variables in the file:
* `BB_RECIPE_NAME`: Set the name of the bitbake recipe
repository [meta-seco-edgehog-things][9]. Modify the following variables in the file:
* `MASTER_BRANCH`: the main branch of the project.
* `MANIFEST_FILE`: the manifest file where the project is present (e.g. `distro/seco-distro.xml` for [meta-seco-edgehog-things][9])
3. Create a corresponding entry in the `SRCREV.conf` file of the manifest repo:\
`SRCREV_pn-<bitbake-recipe> = "<current-commit-hash>"`
3. Create a corresponding entry in the manifest file of the manifest repo.
4. Add the project into the gitlab-ci project's `.gitlab-ci.yml`, so changes to
the CI are deployed into the new project.
......@@ -204,7 +176,7 @@ the git revision in the `SRCREV.conf` file on project changes.
2. Enable the CI/CD functionality for the repository
(General -> Visibility, project features, permissions -> CI/CD)
3. Check that the default branch is protected and that *Maintainers +
Developers* are allowed to merge (Repository -> Protected branches)
3. Check that the default branch is protected and that *Maintainers*
are allowed to merge (Repository -> Protected branches)
[8]: https://git.seco.com/edgehog/layers/seco/meta-seco-edgehog-things
[9]: https://git.seco.com/edgehog/layers/seco/meta-seco-edgehog-things
<mxfile host="app.diagrams.net" modified="2022-12-13T12:09:38.820Z" agent="5.0 (X11)" etag="YEs0r12hDfQW9MvCnzrW" version="20.6.2" type="device"><diagram id="fUzyECxa7j4ng1UTdmFO" name="Page-1">7Vxtc6JIEP41qbv9oAUMIH6MJu7uXVKVS273dj+OMCoXBG8YTdxffzMIOMyMigi+JIlVu9BAg91PP93TM3gF+tPXzxjOJveRh4IrQ/Ner8DNlWEAzbDof0yyXEl0o2usJGPse6lsLXjyf6FUqKXSue+huHAiiaKA+LOi0I3CELmkIIMYRy/F00ZRULzrDI6RJHhyYSBL//E9MllJO4a2ln9B/niS3tnS0gNTmJ2bCuIJ9KIXTgRur0AfRxFZbU1f+yhgxsvMsrpusOFo/lwYhaTMBfeP33tfv7XixR0Jge4E17evP1pde6VmAYN5+oXTpyXLzAI4moceYlq0K9B7mfgEPc2gy46+UKdT2YRMA7qn082RHwT9KIhwci0Y9NmHyscYej590hsfUy/5UUiPIxgTdkkUEu6S1R+VxwRHz4g70nPYhykLYBynz5ObNdl5RsSdpDsQuymc9Fxb5kKtbVGZbMLUqguECXrlRKlJP6Noighe0lPSo61OBtQU4JkBX9ZgsZ30lAkHlK7tpCBNATrOda+dSDdSP+7hU13hUhtOmaPCYTxLvre4rylEN2iBgmiG8BZE6HsjwoPIGbkq99qug4ajmtxid4tuMYHkFt1WuMXUGvKKsTvQUOhdM8aie2EUUmHPg/EktzNn0yKYqR16xSClhsPLHznOk92f6bXJzs0rf+bNMosfKeSSzzafEIjHiOyGI/IKPCt7jvOMpXBMJsMogMRfFNlZ5a30Dg+RT5+YA4YQr10bFHXE0Ry7KL2M51NRkyFAzOgImlamkTQl8Mm/eHVEAVWcB4xRPX9BN8dsk94EtqbLAC5ZHK8OD3F2NJPQ+3PXXGK802g+r3i36ox3Jn+AhCAcJhJDA42yAMREeLKNPjqXuNY7xWjME8DecS1pEjHScFxnBFUSOi4rhny3CJhNQOBhoO2AQXXKB93zwoYpcH5ekB2ODVFT09iQy4iHeTyhkiGGoTtZVXIJp/scyf83Z6OMHvS81ghBMsdoLeRygL8xA1AaJkWEwcAfMzJyKTpoagE9RtY+HTddpwemvuexy3sYxf4vOExUMVzNmHESc1m9K+uG6ZqTKBZL9SxTpATEp5VdnLRH3gC2VXSo5ciJQ4VNo6nEoXfrDv9/59PZU3qxKtGgV5/k7EC3f3LyNTewnZ3UsAqkC6IGMaA7ol9LU4NIMpKmhqnBMBW4kerBPqbxT6Nfu39MuiG4fuqoWGS+WYoxNQEYWR3KQdzQjkkxoATFHNL1GTkucpU1/9CxTGsrg+xhWN3ZkIwLhpXt2ljN31W1XgS7xhM4Y5vzaXDtkoiH9R0couCBwjTtjw0jQqKpAvckEuwfzUngh9TQWQ9Uk51y2xtc395wrTgB6ls6dHU4y7aKzrIUA7SGYuDpT31mzf6+64Xwh4H/uAv/uv+q6McMfTKEz6hFv75PXXSSQbBkUIXZtzCNYONjDoKVRla1KGrkmYE1GAx6KrM6xhDY9gFgr9cR+RCS84Spovy81KzdFarq4D26wnQ6JV2hg4ZcIbeG3KQqa01xiapos4E3EX75Okrl8yIqmnBJPmfGz5kcMxc0PQl2KbFBH6hcbBjZILV2V3R2u+KYbVOt2Dhtazqo0DxNznhA2Kc2YgHIRZIwDt7Zb9tcy5xsCA3sbgFFOhB0lB5Ci5ocQVF9I2ilGZ06sXf8KbrNbfvdIDsdeIT+S9dyqoJH0KSbzlHRU3vb7pCufS20ojsnhYY4U6vrtUFD1NQwNBQzOu+9a39Y4WLZZpE1ulLd0ljTXu3hEp2fo3TUDrMrEDpqucF2dNQ6WmOGlfs0F0+r57UARgqKyrQqaWqaVku0l85jpqwMbvips3NMx1ISrVypSYld1NQ0bj5mys4zrdvCTJnelfsRjc2UqaGy3zKuS0g/xqm7BUUfA61qt0AHOzQ1TSOqtuFmGsm4Y1PQK6hkOEwno1ohnFIdg7255b0yCQDFZT2GbUhMoquW5Tc2Fabv19m89GLlxCwjcYNZvSe5Q1PTLFOiK3mBI8u8BXOEtRpqu8r9um8zb0XVEiXnRKzmWgrXhR+z6Rl603AzxXNU/PTYf7z93najcMSSdkT/+XJ7TZ9Ci0Y115diDpCe5ffkwUP0wle4IxxNmd2H0QJ9Uj3O5tr1Iwtt5A5LBj1QoN5uCvVZiF1KPXs2ZapZvUw1d2hqOIHka7XeQOFxunpCdGJt9YSkqWk4GG+inhAdkg/+TlVPGPs1Iy9i9jgP5Laj6Vwwt7S2BsytEc12qq942Dm8SDt4597sEIchlmW1Ne7PqJpTBL1ivdA0h8gNVHonNMZJ+byzunt7K9gMS2afo65gM1R9yh0t7aTqr9CV4lw9KL5evH9/6qP3ndcFVrFjZWXtRT6hHTejlVgW+TFWUJR0VubKg8cKkqaGiR3sNzysr4hJ6wt+oWW9lUzDKzDNsyozqi+ikwoLR9tWsDQNR5mB3Alyn99ljWEr3pg6ao0B5BmM/sobmj+qVkocPsG1tcnqx+yxYEz9kpY77hwnDmC/zxVTN//Gzlg1XkV9v8PgBS7ZcYLnKFMw8nHMLkevyJ2zFfmfPkoZuZQRkGsr2p6ql/06jUF3v6XdH6VM/q4yqPrrSGIpI2lqOndczM8onDEcJCeKP3FVue0paWoYDmaJyvYC2555VJ2q7Zl1rz/C7BDWFSbHncqsKy6tlDQ1HWYNv0p5ojAzTz27YB713bSqswQWF5La1pDk5y+MwgyG1u52csGGMurA0XzJxUtlfujstNkZ6G1hmVvlV6Tylzb3e/WSQgEuudPScchezEJ317+CvDp9/VvS4PZ/</diagram></mxfile>
\ No newline at end of file
docs/create-recipe-mr.jpg

93.6 KiB

......@@ -4,7 +4,7 @@
The pipeline running in the manifest repository is the one that does the real
work, like building images, generating docs and deploying image to download
servers and triggering tests in LAVA (TO DO). The pipeline running in the manifest
servers and triggering tests in LAVA. The pipeline running in the manifest
repository is described in the `manifest-pipeline-yocto.yml` [file][1].
[1]:https://git.seco.com/edgehog/infrastructure/gitlab-ci/-/blob/master/manifest-pipeline-yocto.yml
......@@ -19,8 +19,8 @@ whenever a merge request is opened on a project or on the repository containing
CI/CD scripts. Eventually the asynchronous build can be skipped in case the
`skip build` label is specified when opening the merge request.
The `deploy` and `notify` stage job are currently triggered only on scheduled
pipeline.
The `deploy` and `notify` stage jobs are currently triggered only on scheduled
pipelines.
<!----------------------------------------------------------------------------->
## Build jobs
......@@ -32,7 +32,7 @@ files in the `boards` [folder][2]. To include another board within the compiled
one it will then be necessary to include at the beginning of the
`manifest-pipeline-yocto.yml` file the corresponding board configuration file.
The build jobs either compile the Egdehog image and the bundle.
The build jobs either compile the Egdehog image and the RAUC bundle for the update.
[2]:https://git.seco.com/edgehog/infrastructure/gitlab-ci/-/tree/master/boards
......@@ -53,6 +53,18 @@ defined at group level][3]:
[3]:https://git.seco.com/groups/edgehog/-/settings/ci_cd
[4]:https://learn.microsoft.com/it-it/azure/cognitive-services/translator/document-translation/create-sas-tokens?tabs=Containers
<!----------------------------------------------------------------------------->
## Changelog
<!----------------------------------------------------------------------------->
Every time a build is executed on the manifest, the changelog is also generated
from one of the scripts accessing the GitLab API in the [scripts folder][6]. In particular,
the job makes it possible to obtain a file in `.md` format containing the titles
and URL references of all the MRs merged into the various meta-layers. This gives an idea of the
various major changes between different versions of Edgehog.
[6]:https://git.seco.com/edgehog/infrastructure/gitlab-ci/-/tree/master/scripts
<!----------------------------------------------------------------------------->
## Notify Job
<!----------------------------------------------------------------------------->
......@@ -64,43 +76,16 @@ Edgehog group. The notification is allowed by using an [Incoming WebHook][5],
saved as a [group variable][3]
[5]:https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook
<!----------------------------------------------------------------------------->
## Test jobs
<!----------------------------------------------------------------------------->
TODO
## Release
<!----------------------------------------------------------------------------->
## Job generation, parent child jobs (NOT USED)
<!----------------------------------------------------------------------------->
As we support images for multiple machines/architectures we are building a lot
of different image variants, where each needs at least build, deploy and test
job.
To simplify the generation of all these jobs, without a lot of copy and paste
and the possibility to configure the actual build images, we are using gitlab's
dynamic-child-pipeline feature. [See gitlab docs.][1]
[1]: https://docs.gitlab.com/ee/ci/pipelines/parent_child_pipelines.html#dynamic-child-pipeline-example
There is a *'generate-build-jobs'* job, that creates a yaml file containing the
pipeline with all needed jobs.
There are the following CI variables in the 'generate-build-jobs' job controlling
the content (make sure these are not set in a more global scope, as this
would overwrite the settings in the generated yml):
* `CI_PARAM_MACHINES`: Space separated list of machines to build for, like "santaro santoka"
* `CI_PARAM_IMAGE`: The name of the image to build. If set to an empty string,
related jobs are not created.
* `CI_PARAM_DISTRO`: The name of the distro to build
* `CI_PARAM_IMAGE_FNG`: The name of the fngsystem image to build. If set to an
empty string, related jobs are not created.
* `CI_PARAM_DISTRO_FNG`: The name of the fngsystem distro to build
Only when the tag is executed will a final job also be executed that allows a [GitLab
release][7] associated with that tag to be created. This job makes it possible
to associate with the GitLab release all the links to the SW artifacts produced by
the build jobs in such a way that they are easily accessible, and allows the
previously generated changelog file to be associated with the release description.
It uses a python script called `generate_job_from_template.py` to convert the
`build-jobs.jinja2` to `build-jobs.yml`. This yml file is then used by the
*'trigger-build-jobs'* job, to setup the pipeline described by it.
[7]:https://docs.gitlab.com/ee/user/project/releases/
![Manifest's parent child pipeline](manifest-parent-child.png)
![Weekly pipeline](weekly_pipeline.png)
docs/recipe-mr-pipeline.png

24.7 KiB

docs/weekly_pipeline.png

259 KiB

---
# --------------------------------------------------------------------------------------
# Global
# --------------------------------------------------------------------------------------
include:
- local: common.yml
stages:
- testing
- manifest-integration
variables:
MANIFEST_FILE: "distro/seco-distro.xml"
# The BB_RECIPE_NAME is used for projects referenced in the SRCREV file
# to match the repository and the bitbake recipe name.
# We set it here to none, as every project needing it
# has to specify it in its own gitlab-ci.yml file.
# The BB_RECIPE_NAME is passed to the python scripts below anyway, but not
# used for projects referenced in the manifest file.
MANIFEST_PROJECT: seco-manifest
MASTER_BRANCH_MANIFEST: kirkstone
MASTER_BRANCH_PROJECT: main
BB_RECIPE_NAME: none
workflow:
rules:
# Explicitly allow externally triggered pipelines in every case
- if: $CI_PIPELINE_SOURCE == "pipeline" || $CI_PIPELINE_SOURCE == "api"
# Do not run pipelines on forked projects.
# The pipelines would not work anyway because of the users permissions.
# There are two cases catched here:
# 1. The project is forked into someones gitlab namespace and a MR to
# include a change into this forked project is created. In this case
# is the CI_PROJECT_ROOT_NAMESPACE not seco-ne but the
# namespace the fork lives in.
# 2. The MR from the forked project is created to merge the change into this
# the project in the seco-ne namespace (customer sending
# change to us). Here the the IDs used below differ.
#
- if: $CI_PROJECT_ROOT_NAMESPACE != "seco-ne"
when: never
- if: $CI_MERGE_REQUEST_SOURCE_PROJECT_ID != $CI_MERGE_REQUEST_PROJECT_ID
when: never
# FIXME: Unfortunately we cannot use variables in regular expressions due to a
# GitLab limitation: https://gitlab.com/gitlab-org/gitlab/-/issues/209904
# As soon as this get fixed, use the regex based rules below instead of checking
# against the MASTER_BRANCH variable.
# Run pipeline if target branch of the merge request has an integration target, i.e.
# INTEGRATION contains a line beginning with the target branch followed by a colon.
# This also implies that the pipeline runs in merge request context only.
# - if: $INTEGRATION =~ /^$CI_MERGE_REQUEST_TARGET_BRANCH_NAME:/m
- if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $MASTER_BRANCH
# Run pipeline on target branch after merging a merge request.
# - if: $INTEGRATION =~ /^$CI_COMMIT_BRANCH:/m
- if: $CI_COMMIT_BRANCH == $MASTER_BRANCH
- if: $CI_MERGE_REQUEST_IID
- if: $CI_PIPELINE_SOURCE == "schedule"
when: always
.skip-for-gitlab-ci-integrations:
rules:
- if: $CI_COMMIT_REF_NAME !~ /^integrate\/gitlab-ci\/.*/
# --------------------------------------------------------------------------------------
# Manifest integration jobs
# --------------------------------------------------------------------------------------
testing:
tags:
- infrastructure
rules:
- if: $CI_PIPELINE_SOURCE == "schedule"
when: always
stage: testing
timeout: 10 m
image: secodocker/manifest-test:vtemp
needs: []
generate-pipelines:
rules:
- if: $CI_PIPELINE_SOURCE == "schedule"
when: always
extends:
- .infrastructure
- .skip-for-gitlab-ci-integrations
stage: manifest-integration
script:
# The job generation script implicitly passes the OS environment to the template, so
# that the template has access to all GitLab CI variables. Hence there is no need
# to explicitly pass any of them as command line arguments.
- .gitlab-ci/scripts/generate_job_from_template.py
--template=.gitlab-ci/manifest-integration-pipelines.yml.jinja2
> manifest-integration-pipelines.yml
artifacts:
expire_in: 4 weeks
paths:
- manifest-integration-pipelines.yml
trigger-pipelines:
rules:
- if: $CI_PIPELINE_SOURCE == "schedule"
when: always
extends:
- .skip-for-gitlab-ci-integrations
stage: manifest-integration
needs:
- generate-pipelines
trigger:
include:
- artifact: manifest-integration-pipelines.yml
job: generate-pipelines
strategy: depend
check:
extends:
- .infrastructure
- .skip-for-gitlab-ci-integrations
stage: manifest-integration
rules:
- if: $CI_MERGE_REQUEST_IID && $CI_COMMIT_REF_NAME !~ /^integrate\/gitlab-ci\/.*/
needs:
- trigger-pipelines
allow_failure: true
script:
- cd ${CI_PROJECT_DIR}
# Loop over all integrations and check each integration branch
- while read -r integration; do
SOURCE_BRANCH=$(echo $integration | cut -d':' -f1);
TARGET_PROJECT=$(echo $integration | cut -d':' -f2);
TARGET_BRANCH=$(echo $integration | cut -d':' -f3);
if [[ "$SOURCE_BRANCH" == "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME" ]]; then
.gitlab-ci/scripts/check_if_integration_branch_is_up_to_date.py
--gitlab-url=${CI_SERVER_URL}
--token=${GITBOT_TOKEN}
--target-project=${TARGET_PROJECT}
--target-branch=${TARGET_BRANCH}
--source-project=${CI_PROJECT_PATH}
--merge-request=${CI_MERGE_REQUEST_IID}
;
fi;
done <<< "$INTEGRATION"
yamllint:
extends: .yamllint
stage: manifest-integration
---
include:
- local: manifest-pipeline.yml
variables:
# The id of the gitlab project used in the rules section to not run pipelines in
# forked projects. Using variable here, to allow override in other projects including
# this file.
MANIFEST_PROJECT_ID: 1725
# In the manifest, the remotes are specified by an identifier. This is used to find
# out included projects for the retrigger job. In custom manifests, the remote may be
# named differently, so we need a variable that may be overriden.
CI_PARAM_SECO_REMOTE: seco-ne
# GitLab group to search for projects to retrigger
RETRIGGER_GROUP: ${CI_PROJECT_ROOT_NAMESPACE}
BUILD_TIMEOUT: 1h
# This is the jinja2 template file used to generate the build jobs
BUILD_JOBS_TEMPLATE: build-jobs-yocto.yml.jinja2
generate-build-jobs:
variables:
# Default image and distro
CI_PARAM_IMAGE: guf-image
CI_PARAM_DISTRO: guf-wayland
# Flash-N-Go image and distro
# In the past, the buildfng job overwrote the image and distro itself. Due to the
# transition to the new seconorth names, image and distro for the buildfng must be
# settable from outside of the job.
CI_PARAM_IMAGE_FNG: fngsystem-image
CI_PARAM_DISTRO_FNG: guf-fngsystem
# List of machines to build images for
CI_PARAM_MACHINES: imx6guf imx6ullguf imx8mguf imx8mpguf
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment