Integrate gitlab-ci/lpag/jinja2_rendering and 18 more
[CACHE][TAG] Use downloads and sstate-cache on tag
- In a first implementation of the build logic, the downloads and sstate-cache were supposed to be used only on sheduled and MR builds. However, as the number of supported boards increase, there is more necessity of CPU and time to get all Yocto images built when a tag is performed. For this reason we decide to remove the differentiation at least as long as the build server is not upgraded.
--
[REPORT] Fix on link report naming on tag
- The report creation included an extra string on the link to the artifacts that was not really present and gave rise to not working URLs.
--
[DEPLOY-CI] Fix on projects to be CI-integrated
In the previous implementation the deploy-ci job was looking only for projects with the INTEGRATION project variable explicitly referring to the manifest repository. With this fix, the script will allow searching also forprojects with the reference to the meta-layer inside the INTEGRATION variable, thus allowing to integrate CI also for 'bsp' group projects.
--
[BSP][RETRIGGER] Retrigger pipelines for BSP MRs
Every time a BSP related MR is merged, the SRCREV.conf file in the corresponding meta-layer is updated and all other integration MRs in that meta-layer can no longer be merged due to conflicts. For this reason we add this steps every time a BSP related MR is merged:
- Search for all open MRs in the same BSP group of the project on which the MR occurred.
- Retrigger the pipelines for all these MRs, that in turn updates the integration branches in the meta-layer.
- All the integration MRs in the meta-layer are now mergeable so that if two or more MRs are merged (tipically U-Boot and Kernel MRs are merged almost at the same time) no conflict are expected to occur.
- Moreover, this procedure allows to have an integration MR in the meta-layer always updated with the latest BSP version in the main branch.
To make this work we also need to change the retrigger job, avoiding the possibility to run 'check' jobs that are already running and that would throw an error.
--
[TIMEOUT] Increase timeout in notify, deploy jobs
- The timeout in that jobs need to scale up as the number of boards and thus build jobs increases.
- Increase timeout also in .infrastructure generic job for CI tasks since sometimes it failed because it took too much to clone/fetch the repository.
--
[DEPLOY-CI] Fix on srcrev conf file in manifest
In the current implementation of Edgehog project, manifest repositories do not have to contain SRCREV.conf files which are instead in the conf/SRCREV.conf file of the meta-layer. For this reason we want to avoid to throw an error when the file is not found inside the manifest.
--
[DEPLOY-CI] Insert jinja2 rendering for deploy CI
- Since the number of custom projects and thus of GitLab groups and repositories to which the CI/CD steps need to be deployed, jinja2 rendering is introduced in the same way as it was done for the manifest integration steps.
- Define the INTEGRATION_CI project variable for this repository in order to define a list of group, manifest repository and branch name to which the gitlab-ci shall be integrated.
- Use the mechanism of dynamic child pipeline creation.
- Update documentation according to the modifications.
--
[DOC] Update docs according to latest changes
- Insert new section describing all the necessary steps and settings to include a new project in the CI/CD integration flow
- Insert description of dynamic child pipelines creation for manifest pipeline
- Insert instructions on how to deploy the GitLabCI to custom projects
--
[RULES] Change rules for dynamic manifest pipeline
These changes are necessary since the mani manifest pipeline is triggered from a parent pipeline and the rules need to be adapted according to this.
--
[VARIABLE] Redefine variable assignment for custom
- Change name of DEFCONFIG_FILE and BUILD_DIRECTORY variables in order to allow compatibility of the CI/CD also for custom projects
- Insert graphic backend information in the IMAGE_NAME variable
- Fix onvarible assignment for layers integration
--
[RULES] Add "pipeline" and "api" trigger rules
- Inserting these rules allow to retrigger pipelines in MR in case some of the jobs failed
- Insert yamllint control also on masifest pipeline
- Remove some debug echos
- Minor bug fix
--
[YAML][SYNTAX] Fix on yaml syntax
- As of now no control were performed over yaml syntax and some fixes were necessary.
- Increase yaml file line length max value to 200
--
[PIPELINE] Insert cancel pipeline and yamllint job
- The cancel-previous-pipeline job aims at reducing dead time. Indeed, everytime we push a new commit to a branch, a new pipeline is created. Often the pipeline for the previous commit has not finished yet, and GitLab does not automatically cancel it. It consumes valuable build time, even if we don't need the results anymore. This adds a MR pipeline job to search for previous pipelines on the same branch and cancel them explicitly.
- Add yamllint job allowing to check for correct yaml syntax in the dynamic child pipeline created in the layers and manifest integration jobs.
- Add yamllint job also on .gitlab-ci pipeline.
--
[LAYERS] Major refactoring of layers integration
This commit introduces major changes in the layers integration steps performed every time a new MR is opened in one of the BSP repositories. More in detail:
- Introduces jinja2 templating tool to perform dynamic child pipeline and dinamically assign the INTEGRATION variable value to the CI jobs.
- The main layers-integration.yml file only contains check and trigger pipelines to create a dynamic downstream pipeline.
- Introduces layers-integration-jobs.yml that is dinamically assigned with the variables from the INTEGRATION variable and that executes all the main tasks of layer integration pipelines
--
[MANIFEST] Major refactoring of manifest pipeline
The ideas behind this refactoring are the following:
- Introduce dynamic child pipelines also for pipeline running on the
manifest. This in particular allows to dynamically creates a .yml
configuration file containing all the board jobs via jinja2 templating
tool. With this approach it will only be necessary to define three main
variables in the manifest or in the CI/CD variables of the manifest
repository:
- MACHINES: a list of the machines of which the build is necessacy
- DISTRO: a list of the Edgehog distros
- GRAPHIC_BACKEND: a list of the graphic backends supported
- Introduce a generic manifest-pipeline.yml that is used both by the pipeline on the test and general manifest and that simply generates the final .yml file doing all manifest CI/CD tasks.
- Introduce a generic build-pipeline.yml file containing only the definition of the main jobs used in the manifest CI/CD (i.e. build, deploy, test, changelog, etc.).
- Refactor manifest-pipeline-yocto.yml to simply stores variables and include the other .yml files thus allowing the other steps to be better tested in the testing group.
--
[CLEANUP] Remove unused and legacy files
- Delete all the content of the boards/ folder containing the job configuration files for the different boards. The idea behind it is to use a dynamic child pipeline from a dinamically created .yml file (containing all the board jobs), thus simplifying maintenance, avoiding copy and paste errors and allowing to dinamically define as a global variable the list of machines to be built/deployed/testes.
- Delete files initially imported from SECO-NE original project and no longer necessary.
--
[TEST] Include integration jobs for ci-test repos
Up to now, the ci-test group repositories were used for sporadic tests but in the optic of a continuous CI/CD improvement they need to be treated as 'production' repositories. With this approach, every time a MR is created in the gitlab-ci repo, the modifications introduced can be tested in the ci-test group repositories.
--
[CODE][REFACTOR] Insert usage of jinja2 templating
- The jinja2 templating is inserted for manifest integration steps in order to use dynamic child pipelines.
- A new job is generated before the integration actually starts that create the manifest integration jobs, allowing to correctly use the INTEGRATION global variable.
- Move manifest-integration-jobs to a dedicated yaml file.
--
[CUSTOM] Add custom project to gitlab-ci deploy