• Key Vault Authorized Application
• Vault Authentication Methods
• Vault Generate Key Application Authentication Invalid

To acquire an access token with managed identity for azure key vault, you just need to: var azureServiceTokenProvider = new AzureServiceTokenProvider;var keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback)); The code string accessToken = await azureServiceTokenProvider.GetAccessTokenAsync(VaultUrl);is not necessary here, and it should be the root cause of your issue. NET Core web application to access key vault. We are going to perform below steps: Register web application which will create service principal for the application; Generate client secret for new app registration. This client secret is kind of authenticating identity of application. This is a code walkthrough to show you how to create a.Net console application to authenticate to Azure Active Directory using OAuth2 Client Credentials flow to get an access token to Azure Key Vault. It then uses the access token to call Azure Key Vault to get a secret.

Azure Key Vault is a cloud service that safeguards encryption keys and secrets like certificates, connection strings, and passwords. Because this data is sensitive and business critical, you need to secure access to your key vaults by allowing only authorized applications and users. This article provides an overview of the Key Vault access model. It explains authentication and authorization, and describes how to secure access to your key vaults.

1. Vault can create a one-time password (OTP) for SSH authentication on a network every time a client wants to SSH into a remote host using a helper command on the remote host to perform verification.
2. Authenticate with Certificate to Azure Key Vault. Azure AD Application authenticates to Key Vault by using a Client Id and an X509 Certificate instead of Client Secret. Create or Get a Certificate; Configure Azure AD and Associate the Certificate; Access Azure Key Vault from.NET Client using X509 Certificate; Create or Get a Certificate.

For more information on Key Vault, see About Azure Key Vault; for more information on what can be stored in a key vault, see About keys, secrets, and certificates.

Access model overview

Access to a key vault is controlled through two interfaces: the management plane and the data plane. The management plane is where you manage Key Vault itself. Operations in this plane include creating and deleting key vaults, retrieving Key Vault properties, and updating access policies. The data plane is where you work with the data stored in a key vault. You can add, delete, and modify keys, secrets, and certificates.

Both planes use Azure Active Directory (Azure AD) for authentication. For authorization, the management plane uses Azure role-based access control (Azure RBAC) and the data plane uses a Key Vault access policy and Azure RBAC for Key Vault data plane operations (preview).

To access a key vault in either plane, all callers (users or applications) must have proper authentication and authorization. Authentication establishes the identity of the caller. Authorization determines which operations the caller can execute. Authentication with Key Vault works in conjunction with Azure Active Directory (Azure AD), which is responsible for authenticating the identity of any given security principal.

A security principal is an object that represents a user, group, service, or application that's requesting access to Azure resources. Azure assigns a unique object ID to every security principal.

• A user security principal identifies an individual who has a profile in Azure Active Directory.

• A group security principal identifies a set of users created in Azure Active Directory. Any roles or permissions assigned to the group are granted to all of the users within the group.

• A service principal is a type of security principal that identities an application or service, which is to say, a piece of code rather than a user or group. A service principal's object ID is known as its client ID and acts like its username. The service principal's client secret or certificate acts like its password. Many Azure Services supports assigning Managed Identity with automated management of client ID and certificate. Managed identity is the most secure and recommended option for authenticating within Azure.

For more information about authentication to Key Vault, see Authenticate to Azure Key Vault

Key Vault authentication options

When you create a key vault in an Azure subscription, it's automatically associated with the Azure AD tenant of the subscription. All callers in both planes must register in this tenant and authenticate to access the key vault. In both cases, applications can access Key Vault in three ways:

• Application-only: The application represents a service principal or managed identity. This identity is the most common scenario for applications that periodically need to access certificates, keys, or secrets from the key vault. For this scenario to work, the objectId of the application must be specified in the access policy and the applicationId must not be specified or must be null.
• User-only: The user accesses the key vault from any application registered in the tenant. Examples of this type of access include Azure PowerShell and the Azure portal. For this scenario to work, the objectId of the user must be specified in the access policy and the applicationId must not be specified or must be null.
• Application-plus-user (sometimes referred as compound identity): The user is required to access the key vault from a specific application and the application must use the on-behalf-of authentication (OBO) flow to impersonate the user. For this scenario to work, both applicationId and objectId must be specified in the access policy. The applicationId identifies the required application and the objectId identifies the user. Currently, this option isn't available for data plane Azure RBAC (preview).

In all types of access, the application authenticates with Azure AD. The application uses any supported authentication method based on the application type. The application acquires a token for a resource in the plane to grant access. The resource is an endpoint in the management or data plane, based on the Azure environment. The application uses the token and sends a REST API request to Key Vault. To learn more, review the whole authentication flow.

The model of a single mechanism for authentication to both planes has several benefits:

• Organizations can control access centrally to all key vaults in their organization.
• If a user leaves, they instantly lose access to all key vaults in the organization.
• Organizations can customize authentication by using the options in Azure AD, such as to enable multi-factor authentication for added security.

Resource endpoints

Applications access the planes through endpoints. The access controls for the two planes work independently. To grant an application access to use keys in a key vault, you grant data plane access by using a Key Vault access policy or Azure RBAC (preview). To grant a user read access to Key Vault properties and tags, but not access to data (keys, secrets, or certificates), you grant management plane access with Azure RBAC.

The following table shows the endpoints for the management and data planes.

Management plane and Azure RBAC

In the management plane, you use Azure role-based access control (Azure RBAC) to authorize the operations a caller can execute. In the Azure RBAC model, each Azure subscription has an instance of Azure AD. You grant access to users, groups, and applications from this directory. Access is granted to manage resources in the Azure subscription that use the Azure Resource Manager deployment model.

You create a key vault in a resource group and manage access by using Azure AD. You grant users or groups the ability to manage the key vaults in a resource group. You grant the access at a specific scope level by assigning appropriate Azure roles. To grant access to a user to manage key vaults, you assign a predefined Key Vault Contributor role to the user at a specific scope. The following scopes levels can be assigned to an Azure role:

• Subscription: An Azure role assigned at the subscription level applies to all resource groups and resources within that subscription.
• Resource group: An Azure role assigned at the resource group level applies to all resources in that resource group.
• Specific resource: An Azure role assigned for a specific resource applies to that resource. In this case, the resource is a specific key vault.

There are several predefined roles. If a predefined role doesn't fit your needs, you can define your own role. For more information, see Azure built-in roles.

You need to have Microsoft.Authorization/roleAssignments/write and Microsoft.Authorization/roleAssignments/delete permissions, such as User Access Administrator or Owner

Important

If a user has Contributor permissions to a key vault management plane, the user can grant themselves access to the data plane by setting a Key Vault access policy. You should tightly control who has Contributor role access to your key vaults. Ensure that only authorized persons can access and manage your key vaults, keys, secrets, and certificates.

Data plane and access policies

You can grant data plane access by setting Key Vault access policies for a key vault. To set these access policies, a user, group, or application must have Key Vault Contributor permissions for the management plane for that key vault.

You grant a user, group, or application access to execute specific operations for keys or secrets in a key vault. Key Vault supports up to 1,024 access policy entries for a key vault. To grant data plane access to several users, create an Azure AD security group and add users to that group.

You can see the full list of vault and secret operations here: Key Vault Operation Reference

Key Vault access policies grant permissions separately to keys, secrets, and certificates. Access permissions for keys, secrets, and certificates are at the vault level.

For more information about using key vault access policies, see Assign a Key Vault access policy

Important

Key Vault access policies apply at the vault level. When a user is granted permission to create and delete keys, they can perform those operations on all keys in that key vault.Key Vault access policies don't support granular, object-level permissions like a specific key, secret, or certificate.

Data plane and Azure RBAC (preview)

Azure role-based access control is an alternative permission model to control access to Azure Key Vault data plane, which can be enabled on individual key vaults. Azure RBAC permission model is exclusive and once is set, vault access policies became inactive. Azure Key Vault defines a set of Azure built-in roles that encompass common sets of permissions used to access keys, secrets, or certificates.

When an Azure role is assigned to an Azure AD security principal, Azure grants access to those resources for that security principal. Access can be scoped to the level of the subscription, the resource group, the key vault, or an individual key, secret, or certificate. An Azure AD security principal may be a user, a group, an application service principal, or a managed identity for Azure resources.

Key benefits of using Azure RBAC permission over vault access policies are centralized access control management and its integration with Privileged Identity Management (PIM). Privileged Identity Management provides time-based and approval-based role activation to mitigate the risks of excessive, unnecessary, or misused access permissions on resources that you care about.

For more information about Key Vault data plane with Azure RBAC, see Key Vault keys, certificates, and secrets with an Azure role-based access control (preview)

Firewalls and virtual networks

For an additional layer of security, you can configure firewalls and virtual network rules. You can configure Key Vault firewalls and virtual networks to deny access to traffic from all networks (including internet traffic) by default. You can grant access to traffic from specific Azure virtual networks and public internet IP address ranges, allowing you to build a secure network boundary for your applications.

Key Vault Authorized Application

Here are some examples of how you might use service endpoints:

• You are using Key Vault to store encryption keys, application secrets, and certificates, and you want to block access to your key vault from the public internet.
• You want to lock down access to your key vault so that only your application, or a short list of designated hosts, can connect to your key vault.
• You have an application running in your Azure virtual network, and this virtual network is locked down for all inbound and outbound traffic. Your application still needs to connect to Key Vault to fetch secrets or certificates, or use cryptographic keys.

For more information about Key Vault firewall and virtual networks, see Configure Azure Key Vault firewalls and virtual networks

Note

Key Vault firewalls and virtual network rules only apply to the data plane of Key Vault. Key Vault control plane operations (such as create, delete, and modify operations, setting access policies, setting firewalls, and virtual network rules) are not affected by firewalls and virtual network rules.

Private endpoint connection

Vault Authentication Methods

In case of a need to completely block Key Vault exposure to the public, an Azure Private Endpoint can be used. An Azure Private Endpoint is a network interface that connects you privately and securely to a service powered by Azure Private Link. The private endpoint uses a private IP address from your VNet, effectively bringing the service into your VNet. All traffic to the service can be routed through the private endpoint, so no gateways, NAT devices, ExpressRoute or VPN connections, or public IP addresses are needed. Traffic between your virtual network and the service traverses over the Microsoft backbone network, eliminating exposure from the public Internet. You can connect to an instance of an Azure resource, giving you the highest level of granularity in access control.

Common scenarios for using Private Link for Azure services:

• Privately access services on the Azure platform: Connect your virtual network to services in Azure without a public IP address at the source or destination. Service providers can render their services in their own virtual network and consumers can access those services in their local virtual network. The Private Link platform will handle the connectivity between the consumer and services over the Azure backbone network.

• On-premises and peered networks: Access services running in Azure from on-premises over ExpressRoute private peering, VPN tunnels, and peered virtual networks using private endpoints. There's no need to setup public peering or traverse the internet to reach the service. Private Link provides a secure way to migrate workloads to Azure.

• Protection against data leakage: A private endpoint is mapped to an instance of a PaaS resource instead of the entire service. Consumers can only connect to the specific resource. Access to any other resource in the service is blocked. This mechanism provides protection against data leakage risks.

• Global reach: Connect privately to services running in other regions. The consumer's virtual network could be in region A and it can connect to services behind Private Link in region B.

• Extend to your own services: Enable the same experience and functionality to render your service privately to consumers in Azure. By placing your service behind a standard Azure Load Balancer, you can enable it for Private Link. The consumer can then connect directly to your service using a private endpoint in their own virtual network. You can manage the connection requests using an approval call flow. Azure Private Link works for consumers and services belonging to different Azure Active Directory tenants.

For more information about private endpoints, see Key Vault with Azure Private Link

Example

In this example, we're developing an application that uses a certificate for TLS/SSL, Azure Storage to store data, and an RSA 2,048-bit key for encrypting data in Azure Storage. Our application runs in an Azure virtual machine (VM) (or a virtual machine scale set). We can use a key vault to store the application secrets. We can store the bootstrap certificate that's used by the application to authenticate with Azure AD.

Vault Generate Key Application Authentication Invalid

We need access to the following stored keys and secrets:

• TLS/SSL certificate: Used for TLS/SSL.
• Storage key: Used to access the Storage account.
• RSA 2,048-bit key: Used for wrap/unwrap data encryption key by Azure Storage.
• Application Managed Identity: Used to authenticate with Azure AD. After access to Key Vault is granted, application can fetch the storage key and certificate.

We need to define the following roles to specify who can manage, deploy, and audit our application:

• Security team: IT staff from the office of the CSO (Chief Security Officer) or similar contributors. The security team is responsible for the proper safekeeping of secrets. The secrets can include TLS/SSL certificates, RSA keys for encryption, connection strings, and storage account keys.
• Developers and operators: The staff who develop the application and deploy it in Azure. The members of this team aren't part of the security staff. They shouldn't have access to sensitive data like TLS/SSL certificates and RSA keys. Only the application that they deploy should have access to sensitive data.
• Auditors: This role is for contributors who aren't members of the development or general IT staff. They review the use and maintenance of certificates, keys, and secrets to ensure compliance with security standards.

There's another role that's outside the scope of our application: the subscription (or resource group) administrator. The subscription admin sets up initial access permissions for the security team. They grant access to the security team by using a resource group that has the resources required by the application.

We need to authorize the following operations for our roles:

Security team

• Create key vaults.
• Turn on Key Vault logging.
• Add keys and secrets.
• Create backups of keys for disaster recovery.
• Set Key Vault access policies and assign roles to grant permissions to users and applications for specific operations.
• Roll the keys and secrets periodically.

Developers and operators

• Get references from the security team for the bootstrap and TLS/SSL certificates (thumbprints), storage key (secret URI), and RSA key (key URI) for wrap/unwrap.
• Develop and deploy the application to access certificates and secrets programmatically.

Auditors

• Review the Key Vault logs to confirm proper use of keys and secrets, and compliance with data security standards.

The following table summarizes the access permissions for our roles and application.

The three team roles need access to other resources along with Key Vault permissions. To deploy VMs (or the Web Apps feature of Azure App Service), developers and operators need deploy access. Auditors need read access to the Storage account where the Key Vault logs are stored.

Our example describes a simple scenario. Real-life scenarios can be more complex. You can adjust permissions to your key vault based on your needs. We assumed the security team provides the key and secret references (URIs and thumbprints), which are used by the DevOps staff in their applications. Developers and operators don't require any data plane access. We focused on how to secure your key vault.

Note

This example shows how Key Vault access is locked down in production. Developers should have their own subscription or resource group with full permissions to manage their vaults, VMs, and the storage account where they develop the application.

Resources

Next steps

This week I needed to execute some actions against an Office 365 environment from within a YAML-based deployment pipeline in Azure DevOps, naturally PnP PowerShell and CLI for Microsoft 365 came to mind as the perfect toolset.
I absolutely wanted to avoid username/password authentication, as the account would require Global Administrator permissions to execute the commands but couldn't have multi-factor auth applied and that could pose a major security risk. I decided to use Service Principal authentication with my own Azure AD app, but I also didn't want to keep my certificate in the repository.

Luckily, all of this can be achieved with an out of the box functionality in Azure DevOps: Variable Groups. A Variable Group can be connected to a Key Vault and can connect the secrets & certificates in there to variables in your pipeline. All of this happens securely in the backend and your secret/certificate will never be exposed, not even in the logging of the DevOps action.
Let's see how that works!

Preparation

Service Principal & Service Connection

I usually create one Service Principal in my customers Azure AD for my DevOps automated deployment pipelines, called '{MyCompany} DevOps Pipeline'. This one is used to create the Service Connection to the Azure environment of my customer so we can install the application from our DevOps pipelines. More information about creating this Service Principal for a Service Connection can be found on the Microsoft Docs website.

Extend Service Principal for Office 365 connection

I decided to reuse that existing Service Principal and use it for my connection to Office 365 from the pipeline as well. I need to make some changes in https://portal.azure.com to my Service Principal for this to work.
Go to Azure Active Directory -> App registrations and find your app in the list. After opening it, in the left navigation find API permissions. Depending on the actions you want to execute you have to add some permissions in here to signify that your DevOps pipeline is allowed to do these actions. Since I want an unattended run, I had to choose from the Application scopes:

Make sure you hit the 'Grant admin consent for {Your Organisation}' button to actually allow all the permission scopes. The status should turn green for all of them:

Key Vault

Secrets and certificates need to be stored securely in a Key Vault. I either reuse an existing Key Vault that contains some other shared secrets already, or I create a new one in a new resource group called '{MyCompany}-DevOps-Pipeline':

Next up: allow our Azure AD app to access the secrets in this Key Vault. Select 'Access Policies' (NOT Access Control) in the Key Vault management pane and add an access policy:

Select Get and List for Secret permissions, and next to 'Select principal' you choose the Azure AD application we extended in the previous step. Click add.
Don't forget to click Save in the following screen, or nothing of the previous steps get actually saved:

Now the Azure AD app can read all the secrets from your Key Vault!

Certificate

I wanted to do App-only authentication to SharePoint and since this is only allowed through the Client ID/Certificate flow, I also need a certificate. If you'd only connect to Microsoft Graph you could get away with using just a Client Secret.
There is an easy way and a hard way to get this type of self-signed certificate. In the hard way you'd generate it locally from the command prompt, but in the easy way you just let the Key Vault create one for you:

In your Key Vault, go to Certificates and click Generate/Import button. These are the settings I used, but you can choose whatever fits your purpose (or security guidelines):

Your certificate will be generated and will show up in the overview list.

Add certificate to Service Principal

Last step is to assign this certificate to your Service Principal so it can be used in the actual authentication process to Office 365. In the Key Vault, go to Certificates and find your certificate in the list. Click on it and click again to open the latest version. In that last screen, you can download a PFX/PEM version of your certificate.

Go back to Azure Active Directory -> App registrations and find your app in the list, and open it. In the left navigation find 'Certificates & Secrets', upload your certificate in there:

Azure DevOps Variable Group

Time to switch over to your Azure DevOps environment and open op the repository where you will run your pipeline. Click open 'Pipelines' in the left navigation and select 'Library'. In there we will create a new Variable Group:

Trick here is to enable the toggle 'Link secrets from an Azure Key Vault as variables'. You get to select the Service Connection and the key vault containing your certificate:

Under 'Variables' click 'Add'. It will show you a list of all secrets and certificates that exist in the Key Vault. From this list, select the certificate we created in one of the previous steps:

Azure DevOps Pipeline

It is finally time to create your YAML-based pipeline. In Azure DevOps, underneath Pipelines, click Pipelines. A button 'New pipeline' will be top right. Select where your code lives (I picked Azure Repos Git), and a predefined starter template (I chose 'Starter pipeline'). You need to save it before the next steps work.

Start editing your pipeline again and from the menu behind the three dots on the top right, select 'Triggers':

On the new screen, switch to the 'Variables' tab. It'll show you all available variables, but also an option for 'Variable groups'. Select it, and now you can finally link our Variable Group from the preparations into this pipeline.

Make sure you click Save in the tab bar. You see that the certificate from the Key Vault is now available as a variable in our pipeline! Let's use it to connect to Office 365 by means of PnP PowerShell and CLI for Microsoft 365.

UPDATE 08/12

You can also achieve this link from directly within the YAML-file, by adding the following snippet:

PnP PowerShell

I decided to use the new PowerShell Core version of PnP PowerShell, currently in preview and slated for a GA release very soon. In my YAML pipeline I added a PowerShell task, made sure I selected to use PowerShell Core and used the inline mode for my script:

I added the installation of the PnP.PowerShell module in there, and opened the connection to my SharePoint site. This is where the magic happens: I provide the clientId and the tenantId of my Azure AD application/Service principal that we prepared. The last parameter is -CertificateBase64Encoded, and there I pass the variable from the linked Variable Group which fetches the data from my Azure Key Vault.
This results in the following task:

For demo purposes I just fetch the title of the site in SharePoint, but in my actual pipeline I install an SPFx application into the app catalog. And all of this without having secrets or certificates exposed anywhere!

CLI for Microsoft 365

What PnP PowerShell can do, can also be done with CLI for Microsoft 365 albeit a little (a lot?) more complicated.
I once again start with a PowerShell Core task, where I put everything inline. This is because we need to do some transformation on our base64 encoded certificate. The final steps would be this:

Firstly we make sure to use Node 12, as this is the required version to use CLI for Microsoft 365, after that we do the following actions in the PowerShell task:

1. Get the current location of the PowerShell execution
2. Fetch the certificate as base64 encoded
3. Convert the base64 encoded certificate to byte array
4. Write the byte array certificate to disk as a .pfx
5. Open the byte array certificate as an X509Certificate2 object in memory
6. Store the certificate thumbprint and dispose of the object in memory
7. Use openssl tool to convert the the .pfx to a .pem certificate file, containing the private key and passing in an empty import password.
   This is necessary because CLI for M365 doesn't accept an empty password for an .pfx file, and leaving out the parameter assumes it to be a .pem file.
8. Install CLI for Microsoft 365
9. Load clientId and tenantId of your application in memory for CLI to use, instead of using their default information
10. Log in by using the .pem file and the stored certificate thumbprint
11. Load some data from Office 365
12. Log out from CLI
13. Remove the stored certificates from disk

As you can see, this process is a lot more complex than when you'd use PnP PowerShell but the end result is the same: an app-only connection to Office 365 without exposing secrets or certificates!

Conclusion

Using both PnP.PowerShell and CLI for Microsoft 365 it is possible to do app-only, unattended authentication to your Office 365 tenant and execute the necessary actions. Combine this with Variable Groups that connect to your Azure Key Vault, and you've got yourself a very secure pipeline.

At this time, it is a bit easier with PnP.Powershell but with some changes to CLI for Microsoft 365 it could be simplified too:

• Most of the additional steps can be skipped if they allow direct usage of the base64 encoded certificate. (Feature request)
• Don't pass in the --thumbprint parameter but derive it from the passed in certificate. (Feature request)
• Allow passing in an empty password when providing a .pfx file. (Feature request)
• Allow the m365 login command to accept clientId and tenantId as a parameter, so it doesn't depend on environment variables. (Feature request)

UPDATE 10/12: if you need these improvements for CLI in your pipeline, please read the follow-up blogpost!