How We Switched to a Continuous Delivery Pipeline in 3 months

 7 min read

First MonthBackgroundOur deployment strategy was very old and manual due to several reasons:Deployment was manual (build, publish/promote, release and deploy)Functional and technical tests were manualThe release pipeline took long times to be pushed to productionOften defect and rollback for a releaseHigh coupling between Infrastructure & dev teams & QAA lot of release specificities, hard to maintain and operateHave a server was too long and require each time specificationsNo release management was in placeNo artifact was created and packagedThe repository was on Subversion with multiple branch principles

Our old deployment platform

In order to avoid defect deployment, ease releases and post-tests and be more close to the market, we have decided to turn our current pipeline to a CI/CDsOrganizationWe organize the project by the Agile methodology. We have begun by a “kickoff” page under confluence to initiate the project. A formal kick-off meeting must be set up if at least one of the following conditions is met:3+ developers are going to work on the project for at least 2 sprints,The project has an impact on shared resources or on the performance of a web application,The rollout or rollback of the project does not follow the usual process,The project adds a new item to the company stack,The project involves the creation of a new service, application or tool,The project requires additional hardwares or resources,The project will be hosted on machines with a new kind of configuration,The project adds or modifies cookies, or modifies URLs,…Once we had our first clean draft, we chose a name for the project, in our case it was Jaune Doe (John Doe is everywhere). We also sent it to the R&D team for a review. This step took 2 weeks.We got the go-ahead from our fellow stakeholders during the kickoff meeting then we began working on the technical part. We collaborated on the migration to Git (one of the biggest parts of this project). We adopted a “one-branch organisation”.In order to be fully dedicated to the project, we organized a virtual team of 3 engineers (also called vteam): One PM/Lead, one SRE and one QA engineer.The project organization was simple and we’ve been continuously asking and replying to these questions:What needs to be done this week ? A follow-up every Monday for the weekly “menu”What’s keeping you from doing your tasks ? A follow-up every Thursday to identify the pain points that keeps us from doing our jobWhat did we accomplish since the last week ? A demo meeting every Friday morning to show our progressActing as a project manager, I made the choice of using mind mapping (e.g to share ideas and a to-do list. Our mindmap was separated into 3 main branches: Build, Promote and Deploy. At the end, each final branch was represented by a Jira ticket and a Kanban board with all the to-do actions.We ended up having around 50 tickets:

GoalsThe aim of our new deployment methodology is:Deploying an IIS applicationDeploying a Docker applicationDeploying Windows servicesAll deployments should use artifacts, same for SQL with DacPacDeploying infrastructure services (configuration, servers, pools)Deployment could be done in the Cloud or BaremetalEach release/deployment should be trackedDeployment should be fully automatedYou write it, you own it: Developers should deploy their own applicationsDevelopment should be test-drivenDelivering every change to production as soon after it happensSeparation of config from codeReducing our Time to MarketNon GoalsWe will not manage the code pipeline, we will only focus on how infrastructure can make continuous integration smoother.We will not propose the hosting for stateless services.DesignThe idea is to streamline a release through the different stages from the build to production with automation, which will also require the completion of those steps:Service isolationAgnostic environmentHorizontal ScalingOrchestrationAutomatic monitoringA release has a ticketRolling UpdateTest suite provided to run at every step of the deploymentAble to deploy on Cloud and Baremetal servers
Our new deployment platform

We would use this kind of pipeline:Check out code and buildUnit testing and quality controlDeploy to test environmentFetch latest buildsPackagingIntegration testingDeploy to productionProduction testing

Functionalities of the pipeline
Release definitionA release is for us one or more deployable artifact and all other results of the pipelinedeployable one or more artifacts,manifesto (README, INSTALL, VERSION, CHANGELOG),logs,debug artifacts,runtime/buildtime dependencies and their versions.For Msbuild, artifacts are detected using a custom property.Build definitionBuild branch (no concurrent builds & partial build)Build is scheduled as soon as possible with a quiet period. If no baseline is found, a complete build is triggered:For each known repositories, get hash of branch/HEAD of all repositories (master must exist and default)For each modified repositories, clone repository for given branchesScan projects dependencies and alter dependencies graph (from baseline)Clone missing repositories to build from bottom to topGenerate a artefact with all cloned repositoriesBuild the artefactRun unit tests on the artefactPublish & push artifacts for projectsUnit test report is stored in TeamCityBuild log is stored in TeamCityUpdate latest baseline for branch if everything is OKBuild on PR (ephemeral & concurrent)Build is scheduled immediately. If no baseline found, the build fails.Get latest baseline for branchClone repository and apply patchScan projects dependencies and alter dependencies graph (from baseline)Clone missing repositories to build from bottom to topGenerate the artefact with all cloned repositoriesBuild the artefactRun unit tests on the artefactUnit test report is stored in TemCityBuild log is stored in TeamCityBootstrap repositoryA bootstrap build repository will be defined and will contain:Build scriptsTeamCity build definitionRepositories directoryAny modifications in this repository will trigger a build.Version definitionDeploy a release means which artifact, which version, what deployment order, what mandatory compatibilitySRE has a full view of version in production and also a daily reportingThe version package is the same from DEV to PROD. A configuration file is used to declare an environmentTested version means at each stage (development, integration, production) we have a test (system and applicative/functional)Deployment WorkflowEvery day, here is what happens:A job takes a snapshot of latest artifacts and dacpac files then deploy to a SandboxThe code is tested (E2E during 30 minutes max) in a Sandbox for integrationIf the tests pass, the code is published by creating a baseline of all build artifacts and storing them in a repository like for DockerA deployment happens when a developer decides to take a release and put it on production servers by using a ticketThe code is deployed progressively in the day on the production servers (10/50/100%)At the end of deployment, business and technical metrics are measured by instant metric after deployment (10–15min)The deployment to production is progressive (10/50/100%) in order to stop the deployment whenever we detect any incident before the full release. The progressive deployment also enables a worldwide release instead of releasing to one datacenter first, then the other ones.Deployment rulesIt is not possible to reliably release all the applications at the same time.Every deployment must be done within local business hours and current time restriction must be respected (10:00 AM — 04:00 PM local time)The production deployment can be stopped at any time by production teamNo limit to deploy one or more release in the day. It depends on the health of the production platformNo concurrent deployment.Ensure rollback can be done at anytimeEnsure deployment is trackedNo deployment on Friday, on Weekend or Bank HolidaysAll deployments have a # versionCollect team features, communicate, check everything is OK before proceedingStack for Core FunctionsAnsible-Vault for managing secretsKubernetes for managing, organizing, and scheduling containers. We will use it to manage resources and containers deploymentDocker container runtime. Ansible will be the Swiss knife of the platform, it will deploy all the configuration triggered by event that we want ( example: New Docker image, Heavy load in on premise platform)Ansible for provisioning and standard server operations. We use it for legacy deploymentEtcd for a built in framework service discovery. Etcd will be our key configuration host, will manage keys and specific needPrometheus added withPRTG for the monitoring and alertingSCM with Git and BitbucketOur CI use TeamCityAnsible-Vault allows keeping sensitive data such as passwords or keys in encrypted files, rather than as plaintext in your playbooks or roles.Stack for Other FunctionsJira for trackingSlack for collaboration+communicationGrafana for dashboardPortainer UI for DockerDashboard UI for Kubernetes

Second MonthAll development, configuration and installation has been done during this month and also the arrival of the new servers.Hardware and Software installation12 servers Dell R430, 2,2GHz, 32 and 64Go RAM, 200Go SSD.3 servers per DCAll Software are Open sourceLegacy deploymentWe use Ansible to handle and manage legacy applications:ansible-playbook -i windows windows_services.yml -e "env=dev app=appbot url= release="$env → development or production environment$app → generic app name, must be equal to the executable$url → path to the artifact$release → release versionLegacy deployment is done by Ansible following a trigger from TeamCity.Container deploymentDocker deployment is done by Kubernetes following a trigger from TeamCity.Configuration deploymentA shared repository was created and file populated to handle applicative deployment.Manifest creationWe have created a manifesto to ensure the collaboration between teams. Our manifest looks like this:Deployment recipe and artefact organisationThe artifact contains:APP folder which contains binaries and application files. The folder is copied to the targeted hostAPP/ --> Deployed under C:\smart\Release\{{ app }}\{{ app }}-{{ env }}-{{ release }}\App or /App (Linux)CONFIG folder which contains the infra configuration set in which references are in following format “{{ http_user_password }}”. It’s transformed using placeholders and manifesto file.CONFIG/ ---> Deployed under C:\smart\Release\{{ app }}\{{ app }}-{{ env}}-{{ release }}\config | /Config/app_name-env.(json|config|xml)APP.yml file which contains specifications and set value for Kubernetes orchestration or AnsibleApp.yml ---> Deployed under Kubernetes sets (deployment), docker running command, IIS specific configuration

Third MonthThe last month was dedicated to finish all tasks like tracking on a confluence page, implement rules or use Selenium for applicative tests, add notification into Slack and perform the progressive deployment with one application and a canary environment.

ConclusionThis project was very useful for the company and permit to ease releasing, time to market is improved. We were on time, on early time. We handle new application with Kubernetes and also all legacy deployment (IIS and Windows Service) prior to Ansible. We switching from a mono branche to multiple, migrating our SCM to Git and built an artifact for each release with a version number.

Tags: ,


Leave a Comment