templates/monthly.md.erb

## First steps - [ ] Change `#f_upcoming_release` topic with `/topic <%= version.to_patch %>: <link_to_this_issue>` - [ ] Check for any holiday, [Family & Friends day] or [planned PCLs] scheduled for this release: - [ ] Adjust the [release preparation](#release-preparation) days if necessary - [ ] If there are Family and Friends day add notes for pausing deployments before the day starts and unpausing them before the next business day, see the [auto-deploy documentation] for details. - [ ] Update this issue with two planned dates for [recurring Staging rollback practice](https://gitlab.com/gitlab-org/release/docs/-/blob/master/runbooks/recurring-rollback-practice.md) and a planned date for [Hot Patching Production practice](https://gitlab.com/gitlab-org/release/docs/blob/master/general/deploy/post-deployment-patches.md#monthly-release-hot-patching-production-practice). Consider spreading these across timezones to share the knowledge. - [ ] Set Due Date for this Issue to the first practice session - [ ] Check for any [deprecations](https://gitlab.com/gitlab-org/release/docs/-/blob/master/runbooks/deprecation-evaluation.md) ## First Staging Rollback Practice **Date to be Completed:** - [ ] [Perform Staging Rollback Practice](https://gitlab.com/gitlab-org/release/docs/-/blob/master/runbooks/recurring-rollback-practice.md) - [ ] Set Due date for this issue to the date of the next practice session. If none, set it to the release date. ## Second Staging Rollback Practice **Date to be Completed:** - [ ] [Perform Staging Rollback Practice](https://gitlab.com/gitlab-org/release/docs/-/blob/master/runbooks/recurring-rollback-practice.md) - [ ] Set Due date for this issue to the date of the next practice session. If none, set it to the release date. ## Hot patching production practice **Date to be Completed:** *The work here is to be done by the next release managers. Please tag them to the item below.* - [ ] `@name` - [ ] [Perform hot patching Practice](https://gitlab.com/gitlab-org/release/docs/blob/master/general/deploy/post-deployment-patches.md#monthly-release-hot-patching-production-practice) - [ ] Hot Patching practice is documented: `<link to comment>` - [ ] Set Due date for this issue to the date of the next practice session. If none, set it to the release date. ## Check Pre environment **Date to be Completed:** <%= preparation_start_day %> - [ ] The `pre` environment should be checked automatically by the [scheduled pipeline `monthly-pre-deploy` in the release-tools repo](https://ops.gitlab.net/gitlab-org/release/tools/-/pipeline_schedules). Please ensure that it ran successfully. ## Up until <%= preparation_start_day %> - Ensure any deploys that do not make it to canary are investigated. [Disable canary if necessary]. - Push any successful deploy to canary into production after some time has passed (preferably 1h). - Should any deployment blockers prevent automatic promotions to production, this requires approval by the SRE On-Call. 1. Ask for permission to promote the release in #production - provide the necessary context to the Engineer 1. If permission is granted, set the `OVERRIDE_PRODUCTION_CHECKS_REASON` as a variable in the manual `promote` job. The value of the variable should be the reason why production checks are being overridden. See [gitlab-com-deployer.md#skipping-production-promotion-checks](https://gitlab.com/gitlab-org/release/docs/-/blob/6d9db5984c5de288d1be54e202d0253fedb50b82/general/deploy/gitlab-com-deployer.md#skipping-production-promotion-checks) for more info. 1. This will post a comment into this issue and begin the deployment 1. Ask the SRE On-Call to respond to the comment with their approval for auditing purposes ## Release Preparation ### Initial preparation day: <%= preparation_start_day %> - [ ] [Find the latest `sha`](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#auto-deploy-status) that made it into production successfully: `sha` - [ ] Make sure to [execute the post-deploy migration pipeline] to ensure that all post-deploy migrations have been executed: `/chatops run post_deploy_migrations execute`. - [ ] [Manually create the pipeline to update monthly release status to `announced`](https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/new?var%5BMONTHLY_RELEASE_METRIC%5D=announced). - [ ] Notify Engineering Managers and developers that this is the `sha` that is guaranteed to be released on the <%= ordinalized_release_date %>: ``` /chatops run notify ":mega: This is the most recent commit running on GitLab.com and this is guaranteed to be released on the <%= ordinalized_release_date %>. https://gitlab.com/gitlab-org/security/gitlab/-/commits/<SHA>. You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] <%= version.to_minor %>` Please see the following documentation on what this means: * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release` * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release` * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases` * When is the next Release? Check it on the 'Release Information' dashboard: https://dashboards.gitlab.net/d/delivery-release_info/delivery3a-release-information?orgId=1" ``` - [ ] Verify that the [release information dashboard] reflects the accurate status "announced" for the active release version (<%= version.to_minor %>). - The updated status can take up to 15 minutes to display on the dashboard once the pipeline to create the metric is finished. ### Candidate announcement day: <%= candidate_selection_day %> - [ ] Log latest auto-deploy branch: **BRANCH_NAME** - [ ] Ensure this build makes it through into production - [ ] Make sure to [execute the post-deploy migration pipeline] to ensure that all post-deploy migrations have been executed: `/chatops run post_deploy_migrations execute`. - [ ] Grab the `sha` from this new auto-deploy branch and notify Engineering Managers and developers that this is the candidate `sha` for the release: ``` /chatops run notify ":mega: This is the _candidate_ commit to be released on the <%= ordinalized_release_date %>. https://gitlab.com/gitlab-org/security/gitlab/-/commits/<SHA> You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] <%= version.to_minor %>` Further deployments may result in the final commit being different from the candidate. Please see the following documentation on what this means: * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release` * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release` * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases` * When is the next Release? Check it on the 'Release Information' dashboard: https://dashboards.gitlab.net/d/delivery-release_info/delivery3a-release-information?orgId=1" ``` ### RC tag day: <%= rc_tag_day %> - [ ] Determine what is the last auto deploy branch to have deployed to production and add it here: `BRANCH` - [ ] If you plan to use the latest commit deployed to production to create the RC, make sure: - To [execute the post-deploy migration pipeline] to ensure that all post-deploy migrations have been executed: ```text /chatops run post_deploy_migrations execute ``` - To verify if there are [production incidents blocking deployments](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?sort=created_date&state=opened&label_name%5B%5D=Incident%3A%3AActive&label_name%5B%5D=blocks%20deployments&first_page_size=20) - [ ] If there was an [Internal release](https://gitlab.com/gitlab-org/release/tasks/-/issues/?state=all&label_name%5B%5D=internal%20releases) after the last patch release and it includes a security fix: - [ ] Link the Internal release issue: - [ ] Use a commit before the internal release MR for the next tagging step: <SHA> <% if early_branch_creation? %> - [ ] Start the `monthly_release_initial_rc:start` job of the monthly release pipeline: <%= release_pipeline.web_url %> <% else %> - [ ] Create a RC version to ensure that the final version builds correctly ```sh # In Slack: /chatops run release tag <%= version %>-rc42 ``` This will use the latest commit deployed to production for the various components that we release. If a different commit is necessary for a component, such as GitLab, you should run the following instead: ```sh /chatops run release tag <%= version %>-rc42 --gitlab-sha=XXX ``` This will then use `XXX` as the SHA to create the GitLab stable branches. **NOTE:** this SHA is only used if the stable branch has yet to be created. If it already exists, the branch is left as-is. - [ ] Verify that the [CE stable branch] contains the right commits - There should be at least two commits: the last commit from the previous stable branch (usually a version update), and the sync commit created by the merge train. - The sync commit will have the message "Add latest changes from gitlab-org/gitlab@<%= version.stable_branch(ee: true) %>" - [ ] Verify that the pipelines are green ```sh # In Slack: /chatops run release status <%= version %>-rc42 ``` - [ ] Verify that the RC has been deployed to the [pre environment](https://pre.gitlab.com/help) - Deployment to pre will start automatically. It can take 2 hours to start once the RC is tagged. A notification will be sent to the `#announcements` channel in Slack when it starts. - If required to deploy manually, follow the steps in [pre-and-release-environments.md#manual-deployments](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/pre-and-release-environments.md#manual-deployments). <% end %> - [ ] Verify that the [release information dashboard] reflects the accurate status "RC Tagged" for the active release version (<%= version.to_minor %>). - The updated status can take up to 15 minutes to display on the dashboard once the RC is tagged. - [ ] Notify Engineering Managers and developers that final candidate has been created: ``` /chatops run notify ":mega: The stable branch has been created and the release candidate is tagged. Barring any show-stopping issues, this is the final commit to be released on the <%= ordinalized_release_date %>. https://gitlab.com/gitlab-org/security/gitlab/-/commits/<%= version.stable_branch(ee: true) %> You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] <%= version.to_minor %>` * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases` * When is the next Release? Check it on the 'Release Information' dashboard: https://dashboards.gitlab.net/d/delivery-release_info/delivery3a-release-information?orgId=1" ``` ### Tag day: <%= tag_day %> - [ ] Start the `monthly_release_tag_day:prepare` job in the monthly release pipeline: <%= release_pipeline.web_url %> - Jobs to verify the integrity of the stable branches (`monthly_release_tag_day:ensure_stable_branches_green`) and the mirror status (`monthly_release_tag_day:mirror_status`) will start automatically. - [ ] Once the above jobs are green, trigger the `monthly_release_tag_day:start_tag` job: <%= release_pipeline.web_url %> - This job will tag the final `<%= version.to_patch %>` version. - [ ] The `monthly_release_tag_day:check_omnibus_packages_tagging` job in the monthly release pipeline: <%= release_pipeline.web_url %> will be triggered 90 minutes after tagging completes. However, if possible, you can start it earlier. In case the job fails, troubleshoot the issue, and rerun the job. - [ ] Check that the CNG Images are built: [CNG builds](https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines?ref=<%= version.tag(ee: true) %>). - [ ] Check that the Helm chart is built by making sure the `publish_tagged_package` job succeeded: [<%= "#{helm_tag}" %>](https://dev.gitlab.org/gitlab/charts/gitlab/-/pipelines/?ref=<%= "#{helm_tag}" %>) - [ ] After tagging, the release.gitlab.net instance was automatically updated with the latest monthly release. Corroborate if the deployment was successful by verifying that the `monthly_release_tag_day:verify_release_deploy` job succeeded. - A Slack notification with `Validate release.gitlab.net running <%= version.to_patch %> has successfully executed` will be sent to `#f_upcoming_release`. - [ ] After tagging, the [GitLab QA update scenario](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/developer-experience/test-coverage/#gitlab-qa-update-scenario) runs. Corroborate if the QA was successful by verifying that the `monthly_release:update_paths` stage succeeded. - A Slack notification with `Update paths QA` will be sent to `#f_upcoming_release`. - This step is optional at the moment. Ignore and continue the release process if this step fail. - [ ] Validate `<%= version.to_patch %>` has been passed automated QA by ensuring the `release-gitlab-qa-smoke` job from the release.gitlab.net deploy pipeline is green. - Find the pipeline with the name `auto-deploy: release - <%= version.to_patch %>-ee.0` from the [deployer pipelines page](https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines?page=1&scope=all&source=trigger&ref=master). Past this point, no new code can be added to the release that was not included in the final RC. ### Release day: <%= release_day %> Final release is tagged, so any changes will have to initiate a patch release. [Reminder: We have a soft PCL today](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#production-change-lock-pcl), coordinate with the EOC before deploying to production. Consider not running the post-deploy migration pipeline to keep rollback options available. - [ ] At 13:00 UTC, start the `monthly_release_release_day:start` job in the monthly release pipeline: <%= release_pipeline.web_url %>. An announcement will be sent and the publish will happen 10 minutes later automatically. In case you want to publish the release immediately, skip the delay time of the job `monthly_release_release_day:publish`. - :warning: **Make sure that neither packages nor the blog post get published earlier than 13:10UTC without approval by the [messaging lead](https://about.gitlab.com/handbook/marketing/blog/release-posts/managers/) of the release post. Mind that you don't need their approval if you're on time** :warning: - If anything goes wrong and the release is delayed, ping the [release post manager](https://about.gitlab.com/handbook/marketing/blog/release-posts/managers/) on Slack to make them aware of the issue. Cross-post the slack message to the #marketing channel to notify them too - At 14:10 UTC: - [ ] Start the `monthly_release_verify:start` job in the monthly release pipeline: <%= release_pipeline.web_url %> - [ ] Make sure the `monthly_release:verify` stage has completed successfully. - [ ] Check that the CNG Images are published: [CNG builds](https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines?ref=<%= version.tag(ee: true) %>). Each version has 4 tags for `ee`, `fips`, `ubi` and the branch itself. - [ ] Check that the Helm chart is published by making sure the `release_package` job succeeded: [<%= "#{helm_tag}" %>](https://dev.gitlab.org/gitlab/charts/gitlab/-/pipelines/?ref=<%= "#{helm_tag}" %>) - [ ] Post an update about the status in `#f_upcoming_release` ``` :mega: <%= version.to_patch %> is published and publicly available ``` - [ ] Once all packages are available publicly and GitLab.com is up and running on the release version, ping the [release post manager](https://about.gitlab.com/handbook/marketing/blog/release-posts/managers/) on Slack ([#release-post channel]) to give them a go to merge the release post at ~14:20 UTC, so that it will be live at 15:00 UTC - [ ] Start the `monthly_release_finalize:start` job in the monthly release pipeline: <%= release_pipeline.web_url %> ## Release Certification The [release certification process](https://about.gitlab.com/handbook/ceo/chief-of-staff-team/jihu-support/release-certification.html) may apply to this release. cc @gitlab-com/gl-security/product-security/federal-application-security [CE stable branch]: https://gitlab.com/gitlab-org/gitlab-foss/-/commits/<%= version.stable_branch %> [Deploy]: https://gitlab.com/gitlab-org/takeoff#deploying-gitlab [staging.gitlab.com]: https://staging.gitlab.com/ [getting help]: https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/monthly.md#getting-help [#release-post channel]: https://gitlab.slack.com/messages/C3TRESYPJ [Disable canary if necessary]: https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/canary.md#how-to-stop-all-production-traffic-to-canary [execute the post-deploy migration pipeline]: https://gitlab.com/gitlab-org/release/docs/-/tree/master/general/post_deploy_migration#how-to-execute-post-deploy-migrations [planned PCLs]: https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#production-change-lock-pcl [auto-deploy documentation]: https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#do-we-deploy-on-weekends-or-holidays [Family & Friends day]: https://handbook.gitlab.com/handbook/company/family-and-friends-day/#upcoming-family-and-friends-days [release information dashboard]: https://dashboards.gitlab.net/d/delivery-release_info/delivery3a-release-information?orgId=1 /milestone %"<%= version.to_minor %>" /due <%= Date.today + 3.days %>

templates/monthly.md.erb (181 lines of code) (raw):