Last Updated on February 22, 2024 by Arnav Sharma
A short blog on the key concepts and components that make up a pipeline. And understanding how to get started, what tools to use and what links to follow.
Pre-requisites:
- A DevOps project and a blank YAML file in the repo.
- VS Code installed.
- Azure Pipelines extension installed (Ref. https://marketplace.visualstudio.com/items?itemName=ms-azure-devops.azure-pipelines)
- Some patience with code ?
From Microsoft:
- A trigger tells a Pipeline to run.
- A pipeline is made up of one or more stages. A pipeline can deploy to one or more environments.
- A stage is a way of organizing jobs in a pipeline and each stage can have one or more jobs.
- Each job runs on one agent. A job can also be agentless.
- Each agent runs a job that contains one or more steps.
- A step can be a task or script and is the smallest building block of a pipeline.
- A task is a pre-packaged script that performs an action, such as invoking a REST API or publishing a build artifact.
- An artifact is a collection of files or packages published by a run.
Getting Started – Creating a Pipeline:
Step 1: Clone the DevOps repo (where you have the YAML file created) to your system and open VS Code.
and ensure the pipeline (YAML file) is visible:
Step 2: Choosing Triggers & Pool
Triggers to run a pipeline automatically. Azure Pipelines supports many types of triggers. Based on your pipeline’s type, select the appropriate trigger from the list below:
- Scheduled triggers are independent of the repository and allow you to run a pipeline according to a schedule.
- Pipeline triggers in YAML pipelines and build completion triggers in classic build pipelines allow you to trigger one pipeline upon the completion of another.
- Continuous deployment triggers help you start classic releases after a classic build or YAML pipeline completes.
- Pull request release triggers are used to deploy a pull request directly using classic releases.
Agent pool is needed to build your code or deploy your software using Azure Pipelines (you need at least one agent). As you add more code and people, you’ll eventually need more.
When your pipeline runs, the system begins one or more jobs. An agent is a computing infrastructure with installed agent software that runs one job at a time.
These can be of two types: Microsoft-hosted agents and Self-hosted agents
More details here: Azure Pipelines Agents – Azure Pipelines | Microsoft Docs
Step 3: Using Resources for Azure Pipelines
Azure DevOps Pipelines provide you with the ability to share inputs, secure your data and create connections. Resources help in a few ways:
- Ways to share something such as a secure file or password across pipelines.
- Examples of using resources for sharing are variable groups, secure files, and service connections. In all cases, you’re using a resource as a way for a pipeline to access and consume something.
- A tool for enhancing security through access checks and other restrictions.
- For example, you can limit a service connection to only run on one pipeline. You could also make sure that a repository can only be accessed from a pipeline after a manual approach check.
- Ways to improve traceability for your pipeline and make it easier to troubleshoot environments.
- For example, you can see the number of the last run that was deployed to an environment.
Step 4: Understanding what’s needed – ie. multiple stages or multiple steps with tasks or a mix of both.
Like in the below example, we have different stages and each stage has a task related to it.
Option 1: Single-Stage Single Job Pipeline.
For single jobs, you don’t have to specify the stages, instead, you can straightway define the steps and then the task under it.
In the below code, I am trying to deploy multiple Azure Resources, starting with APIM and then networking component for a subnet and will gradually peer it with the main subnet. Then will deploy some random resources like KeyVault and SQL.
In the below example, the top 4 inputs (green block) remains the same for any type of pipeline/number of the jobs but the inputs below that would change depending on the tasks.
Option 2: Single-stage, multiple jobs:
In this type of pipeline, the first part remains the same but has multiple tasks as shown below:
Option 3: Single-stage, multiple jobs:
This can be useful when deploying in multiple environments, like non-prod state followed by prod etc.
And we can also have control, checks and approvals between two stages.
Step 5: The complete Pipeline !!
In the last few steps we covered:
- What tool to use (VS Code/Extension)
- How Pipeline would run (triggers)
- Where Pipeline will run (Agent)
- What options do we have to run the code or execute steps
Now, the last step is: What all we can do?
Using VS Code + the extension that we discussed earlier, we have an option of Intellisense
As you type the ‘task’ using IntelliSense, you will have all the options for tasks that you can do using the pipeline.
So, let’s take an example of ARM Deployment:
I choose the template deployment option:
And then supply the inputs.
Using IntelliSense – I will get all the options that are needed:
Though the tool does not informs you about what’s required or optional, but you can use the task name and perhaps search GitHub or MS Docs to see what’s needed.
For example, for AzureResourceGroupDeployment@2 task you can use MS Docs and check what’s required to run the task:
List of all tasks: Catalog of the built-in tasks for build-release and Azure Pipelines & TFS – Azure Pipelines | Microsoft Docs
Once you have completed the pipeline, you can run it and any issues should be reported in the Azure DevOps screen with an error.
FAQ: Authoring Azure DevOps Pipelines
Q: What is Azure DevOps pipeline?
A: Azure DevOps pipeline is a continuous delivery tool that simplifies the process of building and deploying code to any platform. It allows you to create, test, and deploy your code automatically.
Q: What is a YAML file?
A: YAML is a markup language used to describe data. In Azure DevOps pipeline, YAML files are used to describe the pipeline configuration.
Q: How can I create a YAML file?
A: You can create a YAML file in any text editor. You just need to follow the syntax of YAML language to define the stages and tasks in your DevOps pipeline.
Q: What is a task in Azure DevOps pipeline?
A: A task in Azure DevOps pipeline is a predefined unit of work. It can be used to perform a specific action such as compiling code or running tests.
Q: How can I trigger a DevOps pipeline?
A: You can trigger a DevOps pipeline manually or automatically. Manual triggers can be initiated by a user, while automatic triggers can be set up to run when changes are made to certain branches or when pull requests are created or merged.
Q: How do I specify a branch to target in a DevOps pipeline?
A: You can specify a branch to target in your DevOps pipeline by setting the appropriate property in your YAML file.
Q: How do I install and configure Git for a DevOps pipeline?
A: You can install and configure Git by installing the Git command-line tools on your local machine and setting the appropriate access and authentication protocols.
Q: What is a pull request in Azure DevOps?
A: A pull request is a request to merge changes in a branch to another branch. It allows developers to review and discuss code changes before they are merged into the main branch.
Q: How do I checkout code in a DevOps pipeline?
A: You can check out code in a DevOps pipeline by using the “Checkout” task in your YAML file. This task will fetch the latest code from the specified branch and set up the necessary environment for your pipeline to run.
Q: How do I configure a release pipeline in Azure DevOps?
A: You can configure a release pipeline in Azure DevOps by setting up the appropriate stages and tasks in your YAML file. You will need to define each step of your release process, from building and testing your code to deploying it to your target environment.
Q: How can one create and deploy their first Azure pipeline using YAML?
A: For those new to Azure DevOps, creating and deploying your first Azure pipeline using YAML can be a bit daunting. I’ll explain how to create a basic pipeline using YAML in Azure DevOps. This process involves authoring a YAML file in your repository, which describes the steps and tasks your pipeline will execute. You can use the YAML pipeline editor in Visual Studio Code to aid in this process. As a step-by-step guide, start by configuring the default YAML pipeline and adding pipeline tasks and variables as needed. Throughout 2024, you can also find additional resources and recommended articles from Medium to enhance your understanding and efficiency in using YAML pipelines.
Q: What are the key components of a YAML pipeline in Azure DevOps?
A: A YAML pipeline in Azure DevOps consists of several key components. Firstly, the pipeline uses a YAML file (azure-pipelines.yml) as its source code, allowing for version control and collaboration. This file specifies a list of pipeline tasks, which are the individual steps that the pipeline will execute. Pipeline variables can be defined to manage configuration and settings across the pipeline. Additionally, it’s important to configure the default branch and directory where the YAML file is located. For those who want to edit or view and manage your pipelines, Visual Studio Code provides an excellent interface. Also, for more complex scenarios, you can specify environment variables and use task inputs to handle different runtime conditions.
Q: How can one publish and share a custom task in Azure DevOps?
A: To publish and share a custom task in Azure DevOps, you first need to create a development environment for your task. This often involves writing the task code, typically in PowerShell or JavaScript, and defining the task inputs in a manifest file, such as vss-extension.json. Once your task is ready, you can upload your task to your Azure DevOps organization and publish your extension to the marketplace. This allows others to use your task in their pipelines by simply adding it to their YAML file. Ensure to include a detailed readme or a blog post for someone who wants to author their own tasks, providing step-by-step instructions and testing strategies. This not only helps in sharing your work but also in receiving feedback through the new feedback system in Azure DevOps.
Q: What are the advantages of using a self-hosted agent in Azure Pipelines?
A: Using a self-hosted agent in Azure Pipelines offers several advantages. Firstly, it allows you to create a more customized build and release environment that can meet specific security updates and technical support requirements of your organization. With a self-hosted agent, you can leverage the runtime environment of your own infrastructure, which can be crucial if your pipeline needs to access resources that are only available in your internal network or if you need to use specific software that is not available in the hosted agents. Additionally, by using a self-hosted agent, you can potentially reduce costs associated with the pipeline execution, especially if you have existing infrastructure that is underutilized.
Q: What are the best practices for managing pipeline variables in Azure DevOps?
A: Managing pipeline variables in Azure DevOps efficiently is key to maintaining a secure and organized pipeline. It is recommended to specify and configure pipeline variables carefully to control various aspects of the pipeline behavior. Best practices include using environment variables for sensitive data to ensure security, and utilizing the pipeline’s UI features in Azure DevOps to view and manage these variables. When modifying the azure-pipelines.yml file directly, ensure that variables are clearly defined and documented. Use task inputs to dynamically assign values during runtime. Additionally, keep an eye on related articles and blog posts for updated strategies and best practices, as Azure DevOps continues to evolve throughout 2024.