Integrate gitlab-ci/CI-deploy-yocto and 99+ more
-- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/04cd24f23863acb3ff54535ec166c0c7734abe4e CI: Remove debug, enable yocto manifest deploy -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/89bed7e6841bb0bba9a8dcc221060b8f772f59c2 CI: Add function to merge the gitlab-ci MRs -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/b9cb9a29a80973721956fb6c153c5bc9bfb51dde CI: Add function to merge the gitlab-ci MRs -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/25d7cabdb130955c0754c3742a5be2e4b747c491 CI: Harmonize foobar and yocto CI files Also set yocto deploy job to manual to test merge. -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/2cdf882e1e25e6c9e072d3b4e4160a3f3a1b6c54 CI: Enable analyze stage again in .gitlab-ci.yml again -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/26bd3edfe3ba62745cb5484422b366cccd609aab CI: Add common.yml containing common code for manifest and integration. -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/8bfa443a1550db2b464166f0a69db6f01bcb6c0b CI: Use only one stage for integration to allow parallel execution -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/bb392f5037b495a3c9f7dfa3ea7b340cf0163754 CI: Rewrite check job python file to use gitlab api (no clone anymore) For projects like the kernel the git checkout needs a log of time and space. Directly using the gitlab api only loads the infomation needed. -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/e60e50b1ff159331a257fb02fa77b2f38952c526 CI: Limit clone depth to 1 for speed -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/56b4cacb6f2a62b0dc325275585fe6673d453ba3 CI: Adapt manifests-integration.yml and manifest.yml for yocto projects -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/7cc7fa7701ec2713d3af7e8e21563163e600b4d7 CI: Add all yocto projects to the job list -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/b097a25350e1b4192c0b84aa6e7bb5a77477e3b6 CI: Generate .gitlab-ci.yml with include commands in deploy step This change should be reverted after once used. It is needed to initially add the include .gitlab-ci.yml in all subprojects. -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/4f194e2141fc72023dcf0ace4651475839392f5b CI: Check: integration branch may have multiple commits As all projects are commited in the same branch the 'up-to-date' check may not only check if the first parent commit points to the master/dunfell branch. Now it is needed to loop through the history until the integration branch's commit is found. On fail a message is displayed which merge request needs to be retriggered manually. This can now also be the 'parent'-MR that triggered the complete chain. The check job is used pipeline again. The retrigger job also looks in the .gitlab-ci project for check jobs to retrigger. -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/873ae97fa52ff62e515cf7681a383ed9a9ef608b CI: Disable foobar manifest build for integration branch BCS 746-000646 -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/e081a8d8d4d4b6096f2b64fe450fa0d50de159b2 CI: Create integration branch optionally also when no submodule change is detected -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/4452c5cbe26b80b4559c95543c6258cbd8382f25 CI: Implement that a merge-request may also be specified with project!number Before only the ID was used but the CI_OPEN_MERGE_REQUESTS variable has the complete path. -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/e86d320a42e60f253a534a66e8cee4afe93b30e2 CI: Multideploy: Detect exiting integration branch -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/0ee24df09955c771ab2d06591ab98873a1bc115a CI: Deploy: create integration branches and trigger pipelines in subprojects Allow deploy_gitlab_ci to change multiple projects at once. Use it to create integration branches and merge requests in all projects. Add a python file to generate a job yml from jinja2 template. Add a template for the jobs to trigger. These execute the actual integration in all 'subprojects'. Create the yml file in the deploy job and trigger it in a new trigger job. -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/6b6d57180c086b313399d268605e39f903d96358 CI: Add common gitlab-ci file for all yocto repositories. Create a merged version of the gitlab-ci files from the layers and the repositoried referenced by the SRCREV file to be included in the projects, to replace the project-individual implementation. BCS 746-000646 -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/369e59b3f333487c032a21a69eb4f1445f821c52 CI: Update include file's ref in gitlab-ci.yml when deploying changes Add update gitlab-ci file with function to adapt the include ref to a given revision. Add a 'pre-commit-hook' to the update-submodule function. Adapt deploy_gitlab_ci to use these to update the include statement in the base project .gitlab-ci.yml to use the same ref as the submodule is set to. BCS 746-000646 -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/ce59c40987d7a247d6e0eae713fc806237ace207 CI: deploy_gitlab_ci: Move one-time code to own function as reference Initially the moved code was needed to create the files including the new implementation. -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/b86fe23805678b5183fe3411dda3a42d697cbe38 CI: merge_into_manifest: Check if a commit is already merged -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/daa4cbeed2fad173a436d3d5c484b43dbddaa25a CI: Move the loop to check if a revision is related to another to common -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/3e491ba0735c42e431609468a1b06a212e5905c4 CI: Add new file to query the revision for a project from the manifest. -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/633e40df8078a6ffcb7b937ac0ecec66e78d485d CI: Remove foobar projects from the yocto project list -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/7ce0fd7430aa162965ea7fa411806b8379ebf220 CI: merge_gitlab_ci: Improve error handling. -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/7381d5983371f0bedd89d6c40db6c600932909e6 CI: merge_into_manifest: Fix already merged detection -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/5bb84a04cd6de815350f7dbcf59dd04c2b94a6f7 CI: Merge_into_manifest: Fixes for merge detection -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/285ce1951cc2ce2f6a0c390a6ece5042d930e95a CI: Merge_into_manifest: Fixes for merge detection -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/2aeb899ae3a3ad32ccac3b15f7025d42de5d5dac CI: merge_gitlab_ci: Fix some post merge issues -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/3f293145ca987e3ea3bfb7b1530485167d011a77 CI: merge_gitlab_ci: Fix for merge. -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/0a7d845ea64bf85363f1ce58229223bb0f9ac0d1 mainfest-CI:package_release: Increase timeout for ftp upload Added additional log output for sdk upload. -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/5cb383e6e87bb5860dcfd9a3d2b4726e8202dd42 alphaplan_fwr:manifest-CI: Cleanup Remove alphaplan_fwr debug code and run black and pylint. -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/adddfca255410c5b211631cef79a604c17b3ddaf package_release: Move metadata generation function to a separate file To improve the readability of the package_release.py script. BCS 746-000734 -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/5a60fd6a844e2f161aee2aa1121974b7a300b922 package_release: Improve md5sums.txt file creation Now md5sums.txt can be assembled from different copy processes. The build md5sums file is not overwritten by the license copy process anymore. BCS 746-000637 746-000734 -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/68f1959f3948d0b1ed18a6d9229eb4efa5730545 package_release:manifest-CI: split alphaplan FWR generation and package_release The alphaplan FWR generation is now separated to make the manifest gitlab-ci file more easy to understand and reduce the complexity of the package_release file. BCS 746-000637 -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/5c3ac7a26ba751f31a2832f06778f5135b07f021 CI: merge_into_manifest: Fix detection of gitlab-ci integration branch -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/3dbeefbdf9c50c2ad55e8e39ebfe3b7eb983e93f CI: merge_gitlab_ci: Remove deletion extra deletion of integration branch. -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/227bccb6f9def1cde91174afef585c8ffdc12e1b CI: Debug messages -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/98c8fdf5daee69b0ed21c357c9bc5502faa6b7d7 CI: merge gitlab-ci: manifest is merged at last The merge job of a subproject include in the merge request merged by gitlab ci, should not merge the integration branch in the manifest by themself. This is now disabled by checking the source branch name. This allow to merge the manifests commit at last. -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/e8c2f2214402955c1dc3fd2b43e5e2a642545d03 CI: Enable analyze stage again -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/c7f8bf0db4e4eecc12b5710fb0f9c8bfceba4e2f CI: Integration job is now optional, remove check job -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/55e34b981856d953a0dc0b73c1f3bda608cdbbf9 CI: Rename gitlab-ci jobs Tim suggested to use the same name scheme for all jobs and use the group names yocto and gitlab-ci as base. -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/3318487d518476135ce978fe3fa3a113ca29295f CI: integrate_into_manifest: Fix bug introduced when splitting functions -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/4c74c580f08a1432d7bd89cf2967abbb7e071bf4 CI: disable 'check' job for gitlab-ci integration as integrate is already disabled -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/5d1fb5c4a3613690375c76cb97204eddf9dcd4ac CI: Enable MR pipeline for integration of gitlab-ci again Our settings refuse to merge if no pipeline has been run. -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/69d604a4fe09fefe779eb1599da7ff29d86c7c0c CI: Fix merge pipeline -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/31d6d7dad54ab322345a6f44b3ff913341500f1c CI: deploy_gitlab_ci: Rework the deployment of gitlab-ci into the child projects The deploy_gitlab_ci now creates the integration commit and branch in each passed subproject and create an integration commit in the manifest containing all these new revisions. A build is then triggered on this commit to test the functionality. Split the update_submodule functions to reuse them in different ways. Remove some previously used files. BCS 746-000740 -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/ab3189781f0b0e97c23c7719a5fdbccbf0c4db39 CI: integrate_into_manifest: Fix integration for recipes in SRCREV file -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/976aaec9d9c6ef10defd4b3a7329d366452bd522 CI: deploy_gitlab_ci: Fix merge, integration branch was used after delete -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/65c6f273b63e4d1beca064f6edb3073321fd5cea CI: Rename foobar to ci-test, like the job names -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/60c311abd4930bf490f13ab13bf2534e160bf842 CI: Reuse existing integration branch preserving manual changes -- Commit: https://gitlab.com/SECO-Northern-Europe/yocto/infrastructure/gitlab-ci/-/commit/3e948f5db0c926a1226746fd597b5a53163a0796 manifest: check if docker build folder is empty BCS DevOps-000035 -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@c917791b Fix remote name on retrigger stage -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@1ba52853 Move repositories to git.seco.com Rename variable JENKINSGUF_SSH_PRIVATE_KEY / SSH_PRIVATE_KEY to GITLAB_PRIVATE_KEY on this occasion, because it contains a private key that was generated exclusively for this use case. The according public key has been added as a deploy key in GitLab to all repositories that this repository needs access to. Add more detailed documentation concerning this configurationdirectly in the gitlab-ci files. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@6c3a8ff7 CI:package_release: Remove "GUF-" prefix from gitlab artifacts dir BCS 746-000637 -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@d809d630 manifest-CI: Refactor manifest pipeline files * Split up build and package steps in separate files * Add additional comments * Add log output BCS 746-000637 -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@f2b7ec90 CI: Remove build for dual espresso again The pipeline is not setup to build from different manifest and so on, and the dual_espresso manifest will probably go to a seperate repo to keep it private. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@b81d573a CI: Remove '--verbose' from deploy step -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@c1ebf70c manifest: Added manifest file selection to ci-test -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@1285aad7 manifest: Added manifest file name to integration script BCS 746-000518 -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@dc6d7858 build: Added DualEspresso -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@0941262a layers: Added nogplv3 and dual-espresso layer to CI/CD -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@3b5e9be0 CI: Pull new changelog_generator image, add token to the call. BCS 746-000785 -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@968886c0 Update README.md: Add description on the gitlab-ci CI and how to deploy -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@bcb6076b CI: Add check if integration branch is on top of target branch before reusing it The integration branch was reused even if there where new commits on the target branch, which could lead into reverting some changes. This adds a check, if the existing integration branch is directly based on the target branch and deletes it if not. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@6f09c66c CI: deploy-gitlab-ci: Don't stop integration when one project is already updated Prior to this change the integration stop when one of the projects had already the latest ref of gitlab-ci in to submodule. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@6c39166f CI: deploy-gitlab-ci: Don't stop integration when one project is already updated Prior to this change the integration stop when one of the projects had already the latest ref of gitlab-ci in to submodule. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@aaac8d1c changelog_generator: Use logging instead of print, fix black -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@79afb233 CI: Move changelog_generator to gitlab-ci project The script can also be run in the normal python container, so this slightly reduces the complexity. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@383a5aca Added qt-multi-screen-compositor to project list -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@3e88ad8e buildfng: Moved the image and distro variables to global The image and distro for the buildfng job can now be set via global/external variables. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@751fc360 Improve CI documentation BCS 746-000636 -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@2d2210b6 CI: deploy-gitlab-ci: Don't stop integration when one project is already updated Prior to this change the integration stop when one of the projects had already the latest ref of gitlab-ci in to submodule. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@67b0195a CI: Increase infrastructure timeout to 10min -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@a06956de Rename layers to seconorth -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@6fbbf338 CI: Add fake lava test to ci-test, reduce fetch depth in update submodule -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@d820110c CI: Add variable containing the port for ssh access of the gitlab instance Our server installation uses port 4444 for ssh access. The lava tests didn't took this into account. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@ad9d367e Allow check job to fail This makes merging a project MR possible even if the manifest has received new commits since the MR pipeline build has run. The developer is responsible for deciding whether it is safe to merge it without having a build based on the latest manifest. BCS 746-000794 -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@81c2ad85 merge_into_manifest: fix error message if MR already exists -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@5e514076 Add possibility to skip MR pipeline check via label -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@ee151673 Add possibility to skip MR pipeline build via label -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@0885629d Document "skip build" label -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@e6980d48 Update meta-seconorth-dual-espresso path -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@83002e07 accept_merge_request: explicitly check if MR has merge conflicts According to the documentation [1] we should get a 406 error from the API if we try to merge a merge request which has a merge conflict. We are experiencing a different behaviour, though, where we are getting a 405 error in this case. There is an open GitLab issue on this topic [2]. In our error handling we assumed that the reason for a 405 error is most likely a required pipeline which is still running. This assumtion, however, leads to wrong behaviour in cases where no pipeline is started at all. Fixing this now by explicitly checking for merge conflicts at the very beginning of the error handling sequence. [1]: https://docs.gitlab.com/ee/api/merge_requests.html#merge-a-merge-request [2]: https://gitlab.com/gitlab-org/gitlab/-/issues/364102 -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@1bc683d5 Move python scripts to subfolder BCS 746-000808 -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@2e6c1b4f Build jobs: Use parent-child jobs to generate needed build jobs * Move all build, deploy and test stubs to the manifest-build.yml * Create a new job to generate and trigger the build jobs dynamically * Add the base jinja2 file for the build jobs. * Add initial docs for the manifest pipeline * Remove unused files -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@d9660a80 build-jobs.jinja2: Specify master branch variable again If the MASTER_BRANCH_MANIFEST variable is set in the manifest's .gitlab-ci.yml file, it may not find its way to the generated job, if it is not explicitly set during generation. That is done with this patch. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@686323c6 Remove yaml redundancies in manifest-integration Leave all common yaml parts in manifest-integration.yml and add manifest-integration-yocto.yml and manifest-integration-ci-test.yml containing the different variable assignments for each environment. The repos including these files have to be updated with the new names. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@4c751096 Harmonize manifest-integration yaml files for yocto and ci-test This is the first step for reducing code duplicates. BCS 746-000807 -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@c877f9c9 retrigger: Allow to specify the manifest's remote identifier in a variable Costum manifest may override this variable -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@2b2bc220 Define yamllint job explicitly in each pipeline The yamllint job was included in the pipeline implicitly via the inclusion of common.yml. This is not very transparent. Make the common definition a base job and extend it explicitly in each pipeline now. Add infrastructure stage comment header into both yaml files. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@7f68f307 Yocto manifest build: re-enable build:merge_request job This job was accidentally disabled in 2e6c1b4, because it was included in the generated child pipeline. This pipeline is not created on the master, though, so the condition for the job was never met. Move the job to the parent pipeline again, so that it gets executed on the master like before. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@970c7fd7 Re-add yamllint stage to ci test manifest pipeline Was accidentally removed in previous commit. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@9b8cf51c Share manifest pipeline for ci-test and yocto Combine all common yaml parts in manifest-pipeline.yml and add manifest-pipeline-yocto.yml and manifest-pipeline-ci-test.yml containing the different variable assignments for each environment. This change implicitly introduces parent-child build job generation in the ci-test pipeline, like it is done in the yocto pipeline already. The ci-test build jobs have been moved to build-jobs-ci-test.jinja2 accordingly. Furthermore rename GITLAB_CI_CURRENT_REV to GITLAB_CI_REVISION and remove the run conditions from all generated build jobs, since these are already present in the upstream trigger job. The repos including these files have to be updated with the new file and variable names. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@c1b939fe Use CI variables in build template directly The job generation script implicitly passes the OS environment to the template, so that the template has access to all GitLab CI variables. Hence there is no need to explicitly pass any of them as command line arguments. This change makes the "generate-build-jobs" job more generic, so that it can be shared with the ci-test pipeline in the future. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@775c09c1 Add gitlab_backup script to download all project exports of a gitlab group -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@0dd147c8 SDK: Remove package step from build sdk, add sdk to artifacts, create deploy_sdk -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@ecab2962 Manifest: include yamllint in full pipeline -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@a4071423 Manifest pipeline: run full build pipeline when triggered via web UI -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@8862b213 Manifest refactoring: move job rules into classes This way they can be reused instead of redefining them in each job. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@d604b330 Fix wrong remote name in ci-test pipeline -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@e810c2ac Fix wrong manifest project ID for ci-test pipeline -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@f39e65ab Manifest: really include yamllint in the full pipeline Previous commit did not have the effect due to wrong extend order. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@dc0503b6 CI: Hardcode the artifacts path in the generated job description Some gitlab update(?) seem to have changed the behaviour in variable passing. In the scripts the variables are available but in the artifacts path does not resolve them any more. May be it has something to do with the introduction of the trigger:forward keyword. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@ea3aa816 CI: Readd allow_failure:true to fix not finished pipelines According to the docs a manual job always has allow_failure set to true, but we see a lot of jobs staying in an unfinished state since the removal of the allow_failure: true keyword for the sdk and test jobs. This should fix the broken behaviour. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@c503a6d4 CI: pipeline yocto: Change scope of CI_PARAM_IMAGE, ... to fix fngsystem build The variables CI_PARAM_IMAGE, CI_PARAM_DISTRO, CI_MACHINE have been defined outside the job in the '.gitlab-ci.yml' file. This has a lower precedence then variables defined inside the job, but it seems that these variables are used in the scope of 'Trigger variables' when available in the trigger job. These override the value set in the template. As main results this made FNGSystem jobs build the normal yocto. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@6508e800 CI: Add hardcoded artifact's path generation also for fng system -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@075aeccb Refactoring: share build and test job basics for ci-test and Yocto This shares YAML code for the following procedures: - Build environment - Source code checkout via repo - SSH key and known hosts setup - LAVA test submission - Docker check if build folder is empty -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@ab09db0b CI: Lava Test: Allow to install test image directly from gitlab The images from the build job can be directly installed from gitlab. To achive this some changes in the complete pipeline have been needed. 1. The variables used in the build job, like CI_PARAM_IMAGE, ... and related variables like BUILDPATH are only valid in the build job now. 2. The build job writes every variable needed in a follow up job into build.env. This also includes the url to the fng-install.sh of the final image. 3. The build.env file is used as dotenv artifact, as well as normal file artifact. The dotenv make the written variables automatically available in follow up jobs, that are using the aritfacts, like the deploy job. The normal file artifact is available via artifact download. (I did't found a way to download the dotenv file instead) 4. Some scripts have been added: - Find a job inside the pipeline by name, as the id is not known in advance. - Download all artifacts or one file of the artifacts from a given job - Download one file of the latest job by name 5. The scripts are used to download the build.env into the test job (where not artifacts are needed anymore) 6. The script is sourced and all variables are available inside the script. Additionally this adds a fake build job to the ci-test pipeline, that copies an image from srv73 and stores it as artifact in a way that a test-job can run on it, like in the normal yocto pipeline. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@52bae4a1 Lava Testreporting: Add script to generate markdown report for given jobids Adds scripts for mark down generation. Adds scripts for lava test result queries. Adds script to generate a mark down report from this data Call script from submit_test to generate a report directly after the testrun. Integrate gitlab-ci/repo-do-not-exclude-tags and 30+ more -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@91dc115a repo: do not exclude tags We have created tags in our poky repo which are needed by repo, e.g. https://git.seco.com/seco-ne/3rd-party/yocto/poky/-/tags/seconorth%2Fkirkstone-4.0.0 Do not explicitly exclude tags in order to fetch them. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@5c2defce CI: prepare to work also with renamed machine names lime seco-mx6 -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@ef08f4ef deploy: Increase deploy timeout to 30min Especially when downloading the SDK the deploy takes longer than 10min. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@968aaadd build-jobs: Remove dump_install_command from buildsdk job When the dump_install_command script is executed in the SDK build step an error occurs because the fng-install script/vars are not available. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@fd742017 deploy_gitlab_ci: fix target branch in integration merge request We need to pass the correct target branch to the merge request creation instead of using the project's default branch. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@38379173 Run yamllint job on gitlab-ci integrations Otherwise no pipeline is created at all in these merge requests, which prevents the MR from being merged. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@dce4f8a3 changelog_generator: Fix machine project ID We have deleted the original project due to git corruption. Replace with the new project's ID. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@a7aa7c26 deploy_gitlab_ci: fix project argument passing -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@8a38f194 Put all manifest intgeration jobs into a single stage -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@a09564c1 deploy_gitlab_ci: explicitly pass gitlab-ci project and branch -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@172e7625 Improve error messages -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@f0f5a370 Improve naming of functions and variables -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@c8b484ca Retrigger MRs from all integrating projects Instead of getting the projects from the manifest file, search them via their INTEGRATION variable. Improve log messages on this occasion: print URL instead of just IDs. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@5e75bfc1 Deploy gitlab-ci to multiple branches Add different integration jobs for different manifest branches. The merge stage only has one job for each manifest, though, because otherwise we cannot guarantee a consistent state for all branches. Extend the deploy script for this purpose, so that it can deploy to multiple manifest branches at once. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@771dcff1 Fix temporary dir path We were passing the complete TemporaryDirectory object to the repo clone function instead of just the path string, resulting in the repo being cloned into a local dir "<TemporaryDirectory '/tmp/tmphwakypf8'>". Fix this to actually use the generated temp dir. This change makes it necessary to keep the TemporaryDirectory object reference until we don't need the directory anymore, otherwise it will be removed immediately. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@dce219c8 Print less important output in grey color -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@5e36261d deploy_gitlab_ci: remove hard-coded integration targets Get targets dynamically using the INTEGRATION variable instead. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@94cd6781 Recurse to child pipelines on retrigger This makes it possible to retrigger jobs that are not part of the given pipeline but in any nested child pipeline of it. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@fb1219a7 Ignore __pycache__ in scripts directory -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@0c29c995 Refactoring: rename arguments for consistency -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@d461090b Remove hard coded project and branch variables MANIFEST_PROJECT, MASTER_BRANCH_MANIFEST, MASTER_BRANCH_PROJECT -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@a4aae414 Add rules for integration pipeline -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@bfa38b24 Improve log messages -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@1c9670d9 commit_and_push: do not pass branch Branch can be implicitly determined via the repo. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@3eef782e deploy_gitlab_ci: integrate only correct projects into the manifest Previously we integrated all projects into the manifest, even those that belonged to a different manifest branch. That led to all manifests having the same project revisions, even if they should be different due to different intgrated project branches. Save a list of integration sources for each manifest branch separately now, so that we can check which project belongs to which manifest branch later. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@2b851ac6 Fix potential error message in retrigger job TypeError: 'NoneType' object is not subscriptable Example: https://git.seco.com/seco-ne/yocto/infrastructure/ci-test/minimal-manifest/-/jobs/127610 -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@c3fd0ecc Check if variable is defined before checking it value -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@c1be0f4b Fix jinja2 "or" operator -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@8da534ec Run integration pipeline for target branch only When we're integrating a single branch into a single manifest branch, there is no need to run the intgeration pipeline for all configured integrations, but only for those that match the target branch of the current merge request. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@8c94b725 ci-test: add job to check for correct foo branch This should catch an error where during a gitlab-ci merge process the wrong foo branch is integrated into the manifest, which happened in Yocto. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@163c7e1e Add yml extension to jinja2 templates -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@59523aa0 check job: disable verbose output -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@28bf0065 retrigger: add job states to log message -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@d25cdebf Really fix potential error message in retrigger job TypeError: 'NoneType' object is not subscriptable Example: https://git.seco.com/seco-ne/yocto/manifest/-/jobs/129201 -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@a0f7e84f Rename pipeline stages to reflect YAML filenames -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@f8d6de86 Make check error message more verbose -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@21e7edeb Improve ci-test build -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@76bd81d8 Fix check job rules -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@a6721243 Move check job to parent pipeline -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@593784e1 Refactor check_if_integration_branch_is_up_to_date - Use "source" and "target" terminology in the script, just like in the GitLab CI variables. - Spefify data types for function arguments. - Rename function arguments to better reflect their data types. - Move determination of integration branch into main function, so that the successfull status message can be more verbose. - Remove parent_merge_request feature and checking for gitlab-ci integrations, because the check job does not run on gitlab-ci integration branches at all. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@45b49ee3 Remove build stage on master We don't need to explicitly trigger a build after merging changes in gitlab-ci. A master pipeline on the target projects runs automatically. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@5e3dfaea CI:deploy: fix fngsystem deployment path The match to detect fngsystem still contained 'guf'. Changed to 'seco'. BCS 746-001043 -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@c831d1da get_integration_sources: check if CI is enabled Trying to query CI variables on a project which does not have the CI feature enabled leads to an HTTP 403 (forbidden) error. Check whether CI is enabled using the "jobs_enabled" flag. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@543a2170 manifest_package: add release suffix Customer specific manifest projects can now set the RELEASE_SUFFIX variable, which then gets appended to the deployment folder names. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@c763c43f alphaplan_fwr: change variable "release_name_local" to "release_name" Otherwise it can be assumed that it might be correlated with outputdir_local, which is not the case. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@2d8d26a5 package_release: more refactoring - Make output-dir an argument that can be passed multiple times instead of hard-coding two dirs with certain meanings. - Remove LOCALDIR variable from package.env, because it is not used anywhere (only occurence was job class ".uploadsdkftp" which wasn't used anywhere either). -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@dc3698c8 package_release: refactoring Improve readability of package_release.py and its submodule generate_release_metadata.py. - Add describing comments throughout the file. - Remove duplicate code from functions. Simplify them, so that they do what their names say (and nothing more). - Rename variables and function arguments to make their purpose clearer. - Harmonize variable names with their corresponding command line arguments. - Add pydoc for functions. - Add type declarations to function arguments. - Generate MD5 and metadata files in the source directories so that they can be copied along with the rest of the files. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@a5e27f6e FTP upload: Add flag to hide FTP upload stage This is a workaround to prevent customer releases being uploaded to our public FTP area. It should be removed as soon as we support uploading to different FTP target folders. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@e48b4c0b package_release: Do not fail if a file does not exist -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@9982711f Move deploy jobs to deploy runner -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@d404f257 Alphaplan: allow failure on import Otherwise the parent pipeline is stuck in "running" state until the manual action has been performed. See for reference: https://gitlab.com/gitlab-org/gitlab/-/issues/361574 -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@dd2a6d7d Fix error message -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@04ac29ad CI-Test: fix deploy-files job -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@22c519f1 Move FTP jobs to FTP runner -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@84d0616b Add new deploy pipeline to Yocto -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@f0436868 Move reusable parts from CI-Test to common -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@3d3b4d09 Yocto: fix SDK package -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@37360288 Yocto: fix SDK build -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@2b6fc739 Package: add comment -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@95d6e368 alphaplan_fwr_import: add error handling -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@277d4af7 Package & deploy: do not allow missing dotenv file -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@3c68ccab Alphaplan: add script for importing alphaplan data The script has mostly been copied from alphaplan_fwr.py, but with different input data. Adding a job to the CI-test environment for testing this. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@3b3238a1 Alphaplan: add script for generating alphaplan data This script has mostly been copied from alphaplan_fwr.py, but with different input data (artifacts from the SoftwareStore deploy job) and without actually importing the data to the Alphaplan database. Adding a job to the CI-test environment for testing this. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@387b8184 Deploy: save variables in deploy.env -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@902c9276 Deploy: save rsync file list for later stages -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@15622149 CI-test: use caching between the pipeline stages -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@a96a923e CI-test: add deploy stages for SoftwareStore and FTP Rename all stages to uppercase on this occasion, since we're adding stages with names containing multiple words. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@056f1af1 Add package class and stage -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@a1972724 Add script to query job logs filtered by status, tag, and log pattern -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@d501fed4 Update isort config -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@28906544 Reduce artifact expiration of build jobs to 1 week -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@bab6f811 Package: move SDK files to subfolder "sdk" -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@8bcaa426 scripts: Add delete_job_artifacts script to delete all artifacts from untagged jobs For a given project all artifacts may be deleted. This was needed due to space limitations on the gitlab server. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@ea1f3bad cancel_pipelines: include downstream pipelines -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@c8f97623 Add job to cancel outdated pipelines in merge requests Everytime we push a new commit to a branch, a new pipeline is created. Often the pipeline for the previous commit has not finished yet, and GitLab does not automatically cancel it. It consumes valuable build time, even if we don't need the results anymore. Adding a MR pipeline job to search for previous pipelines on the same branch and cancel them explicitly. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@897ba059 Manifest integration: remove dynamic job generation The dynamic job was only necessary because GitLab did not expand variables in the "trigger:project" keyword: https://gitlab.com/gitlab-org/gitlab/-/issues/347469 This has been fixed in GitLab 15.3, so we don't need it anymore: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/92346 -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@c657f31f mirror_mr_pipeline: improve debug output -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@473f12b1 mirror_mr_pipeline: add main call -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@88708b64 Convert CRLF to LF -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@169b2fa7 Improve mirroring MR pipeline Refactor three scripts into one in order to support triggering a new pipeline in the case of a skipped MR pipeline. This was not possible to implement with the previous three-script design. The single scripts were not used in other places anyway. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@0264fe09 Changelog generator: get all merge requests This fixes older releases not appearing in the changelog. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@e69d303e Changelog generator: remove debug output -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@ed36b465 mirror_mr_pipeline: include pipeline-not-found message in output -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@5ee69ab9 mirror_mr_pipeline: fix error when no pipeline found This case should trigger a new pipeline instead of reporting an error. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@8edbf2a6 Changelog generator: fix typo -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@e6ae06dd Changelog generator: pass projects as arguments Instead of hard-coding the project list in the changelog generator script, make them passable via arguments. This way we can have different changelogs in different envrionments. Hence we're adding the changelog job in the ci-test environment as well. -- Commit: seco-ne/yocto/infrastructure/gitlab-ci@a59a93a0 Package: Add needs to changelog, to package changelog also
Please register or sign in to comment