[BUG] DefaultAzureCredential does not fall back to other credential types unless ManagedIdentityCredential is disabled

[v-uugwuoke]

Describe the bug
The DefaultAzureCredential is supposed to fall back to other credential types when ManagedIdentityCredential is failed, but it doesn't.

Details
attempting to connect to app configuration with DefaultAzureCredential fails to fall back to other credential options. It was working fine until 21st of december. I am using an Azure VM as my dev environment

This code below fails until managedIdentityCredential is disabled

builder.Configuration.AddAzureAppConfiguration(options =>
{
    options.Connect(new Uri(endpoint), new DefaultAzureCredential());
 
});

Error details

System.TimeoutException
  HResult=0x80131505
  Message=The provider timed out while attempting to load.
  Source=Microsoft.Extensions.Configuration.AzureAppConfiguration
  StackTrace:
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<LoadAsync>d__31.MoveNext()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationManager.AddSource(IConfigurationSource source)
   at Microsoft.Extensions.Configuration.ConfigurationManager.Microsoft.Extensions.Configuration.IConfigurationBuilder.Add(IConfigurationSource source)
   at Microsoft.Extensions.Configuration.AzureAppConfigurationExtensions.AddAzureAppConfiguration(IConfigurationBuilder configurationBuilder, Action`1 action, Boolean optional)
   at Program.<Main>$(String[] args) in C:\Users\Uche\source\repos\WebApplication1\WebApplication1\Program.cs:line 23
 
  This exception was originally thrown at this call stack:
    [External Code]
 
Inner Exception 1:
AggregateException: One or more errors occurred.
 
Inner Exception 2:
RequestFailedException: Service request failed.
Status: 403 (Forbidden)
 
Headers:
Date: Tue, 07 Jan 2025 14:07:27 GMT
Connection: keep-alive
x-ms-request-id: c5da4250-739f-4f90-a58d-44a6795158a3
x-ms-client-request-id: 445a7254-fb5d-4d94-b430-3d3320919ecc
x-ms-correlation-request-id: c5da4250-739f-4f90-a58d-44a6795158a3
Strict-Transport-Security: max-age=31536000; includeSubDomains
Access-Control-Allow-Origin: *
Content-Length: 0

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[christothes]

Hi @v-uugwuoke
Could you provide a bit more information about which credential you are expecting to be utilized? Did something change in the environment when this problem started occurring, such as a managed identity being enabled for this VM?

To see what is going on, it would be helpful to get some logging of the repro. Below are the instructions:

For console output, you would just need to add the following line to your program.

using AzureEventSourceListener listener = AzureEventSourceListener.CreateTraceLogger();

If you need to log the output somewhere other than the console, this can be done as described in these docs.

For example:

using AzureEventSourceListener listener = new AzureEventSourceListener(
    (args, message) => myLogger.Log("[{0:HH:mm:ss:fff}][{1}] {2}", DateTimeOffset.Now, args.Level, message),
    level: EventLevel.Verbose); 

[github-actions]

Hi @v-uugwuoke. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[v-uugwuoke]

@christothes The VM is an AVD host machine. No changes were made prior to the issue being observed. I expect it to use the visual studio Credential locally

These are the logs related to the app configuration failure after adding AzureEventSourceListener.

Azure-Identity: [Informational] False MSAL 4.66.1.0 MSAL.NetCore .NET 8.0.8 Microsoft Windows 10.0.19045 [2025-01-08 07:30:14Z - 3362be99-05f9-4176-ad1b-96c8616570f1] [Managed Identity] Successful response received.
'APIapp.API.exe' (CoreCLR: clrhost): Loaded '<Apiapp>.API\bin\Debug\net8.0\System.Text.Encodings.Web.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'APIapp.API.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\8.0.8\System.Numerics.Vectors.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
Azure-Identity: [Informational] False MSAL 4.66.1.0 MSAL.NetCore .NET 8.0.8 Microsoft Windows 10.0.19045 [2025-01-08 07:30:14Z - 3362be99-05f9-4176-ad1b-96c8616570f1] Checking client info returned from the server..
Azure-Identity: [Informational] False MSAL 4.66.1.0 MSAL.NetCore .NET 8.0.8 Microsoft Windows 10.0.19045 [2025-01-08 07:30:14Z - 3362be99-05f9-4176-ad1b-96c8616570f1] Saving token response to cache..
Azure-Identity: [Informational] False MSAL 4.66.1.0 MSAL.NetCore .NET 8.0.8 Microsoft Windows 10.0.19045 [2025-01-08 07:30:14Z - 3362be99-05f9-4176-ad1b-96c8616570f1] [SaveTokenResponseAsync] ID Token not present in response. 
Azure-Identity: [Informational] False MSAL 4.66.1.0 MSAL.NetCore .NET 8.0.8 Microsoft Windows 10.0.19045 [2025-01-08 07:30:14Z - 3362be99-05f9-4176-ad1b-96c8616570f1] Cannot determine home account ID - or id token or no client info and no subject 
Azure-Identity: [Informational] False MSAL 4.66.1.0 MSAL.NetCore .NET 8.0.8 Microsoft Windows 10.0.19045 [2025-01-08 07:30:14Z - 3362be99-05f9-4176-ad1b-96c8616570f1] [SaveTokenResponseAsync] Saving AT in cache and removing overlapping ATs...
Azure-Identity: [Informational] False MSAL 4.66.1.0 MSAL.NetCore .NET 8.0.8 Microsoft Windows 10.0.19045 [2025-01-08 07:30:14Z - 3362be99-05f9-4176-ad1b-96c8616570f1] Looking for scopes for the authority in the cache which intersect with <AppConfigEndPoint>
Azure-Identity: [LogAlways] False MSAL 4.66.1.0 MSAL.NetCore .NET 8.0.8 Microsoft Windows 10.0.19045 [2025-01-08 07:30:14Z] [Internal cache] Total number of cache partitions found while getting access tokens: 0
Azure-Identity: [Informational] False MSAL 4.66.1.0 MSAL.NetCore .NET 8.0.8 Microsoft Windows 10.0.19045 [2025-01-08 07:30:14Z - 3362be99-05f9-4176-ad1b-96c8616570f1] Intersecting scope entries count - 0
'APIapp.API.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\8.0.8\System.Collections.Immutable.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
Azure-Identity: [Informational] False MSAL 4.66.1.0 MSAL.NetCore .NET 8.0.8 Microsoft Windows 10.0.19045 [2025-01-08 07:30:14Z - 3362be99-05f9-4176-ad1b-96c8616570f1] 
	=== Token Acquisition finished successfully:
Azure-Identity: [Informational] False MSAL 4.66.1.0 MSAL.NetCore .NET 8.0.8 Microsoft Windows 10.0.19045 [2025-01-08 07:30:14Z - 3362be99-05f9-4176-ad1b-96c8616570f1]  AT expiration time: 1/9/2025 7:07:01 AM +00:00, scopes: <AppConfigEndPoint>. source: IdentityProvider
Azure-Identity: [Informational] False MSAL 4.66.1.0 MSAL.NetCore .NET 8.0.8 Microsoft Windows 10.0.19045 [2025-01-08 07:30:14Z - 3362be99-05f9-4176-ad1b-96c8616570f1] Fetched access token from host login.microsoftonline.com. 
Azure-Identity: [LogAlways] False MSAL 4.66.1.0 MSAL.NetCore .NET 8.0.8 Microsoft Windows 10.0.19045 [2025-01-08 07:30:14Z - 3362be99-05f9-4176-ad1b-96c8616570f1] 
[LogMetricsFromAuthResult] Cache Refresh Reason: NoCachedAccessToken
[LogMetricsFromAuthResult] DurationInCacheInMs: 0
[LogMetricsFromAuthResult] DurationTotalInMs: 160
[LogMetricsFromAuthResult] DurationInHttpInMs: 0
Azure-Identity: [LogAlways] False MSAL 4.66.1.0 MSAL.NetCore .NET 8.0.8 Microsoft Windows 10.0.19045 [2025-01-08 07:30:14Z - 3362be99-05f9-4176-ad1b-96c8616570f1] TokenEndpoint: ****
Azure-Identity: [Informational] ManagedIdentityCredential.GetToken succeeded. Scopes: [ <app config Endpoint> ] ParentRequestId: cde3d04b-77cc-4166-a0b3-b7f916a72e1c ExpiresOn: 2025-01-09T07:07:01.6098325+00:00
Azure-Identity: [Informational] DefaultAzureCredential credential selected: Azure.Identity.ManagedIdentityCredential
Azure-Identity: [Informational] DefaultAzureCredential.GetToken succeeded. Scopes: [ <app config Endpoint> ] ParentRequestId: cde3d04b-77cc-4166-a0b3-b7f916a72e1c ExpiresOn: 2025-01-09T07:07:01.6098325+00:00
Azure-Core: [Informational] Request [cde3d04b-77cc-4166-a0b3-b7f916a72e1c] GET <AppConfigEndPoint>/kv?key=REDACTED&label=REDACTED&api-version=2023-10-01
Accept:application/vnd.microsoft.appconfig.kvset+json, application/problem+json
Correlation-Context:REDACTED
User-Agent:Microsoft.Extensions.Configuration.AzureAppConfiguration/8.0.0+0afb56a4ec7b628e0b771a4a34ac7f7694ea8cfd,azsdk-net-Data.AppConfiguration/1.4.1 (.NET 8.0.8; Microsoft Windows 10.0.19045)
x-ms-client-request-id:cde3d04b-77cc-4166-a0b3-b7f916a72e1c
x-ms-return-client-request-id:true
Authorization:REDACTED
client assembly: Azure.Data.AppConfiguration
'APIapp.API.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\8.0.8\System.Threading.ThreadPool.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
Azure-Core: [Warning] Error response [cde3d04b-77cc-4166-a0b3-b7f916a72e1c] 403 Forbidden (00.1s)
Date:Wed, 08 Jan 2025 07:30:12 GMT
Connection:keep-alive
x-ms-request-id:89113d68-59bb-4f1f-a089-994f5f368c4b
x-ms-client-request-id:cde3d04b-77cc-4166-a0b3-b7f916a72e1c
x-ms-correlation-request-id:89113d68-59bb-4f1f-a089-994f5f368c4b
Strict-Transport-Security:max-age=31536000; includeSubDomains
Access-Control-Allow-Origin:*
Content-Length:0
 
Microsoft-Extensions-Configuration-AzureAppConfiguration-Refresh: [Warning] Failed to get configuration settings from endpoint '<AppConfigEndPoint>'.
Azure-Core: [Informational] Request [62cefb52-8467-45f8-bbf3-f2051e7f9dab] GET <AppConfigEndPoint>/kv?key=REDACTED&label=REDACTED&api-version=2023-10-01
Accept:application/vnd.microsoft.appconfig.kvset+json, application/problem+json
Correlation-Context:REDACTED
User-Agent:Microsoft.Extensions.Configuration.AzureAppConfiguration/8.0.0+0afb56a4ec7b628e0b771a4a34ac7f7694ea8cfd,azsdk-net-Data.AppConfiguration/1.4.1 (.NET 8.0.8; Microsoft Windows 10.0.19045)
x-ms-client-request-id:62cefb52-8467-45f8-bbf3-f2051e7f9dab
x-ms-return-client-request-id:true
Authorization:REDACTED
client assembly: Azure.Data.AppConfiguration
Azure-Core: [Warning] Error response [62cefb52-8467-45f8-bbf3-f2051e7f9dab] 403 Forbidden (00.0s)
Date:Wed, 08 Jan 2025 07:31:47 GMT
Connection:keep-alive
x-ms-request-id:bc047309-a921-4a22-ba9f-d4665cef823d
x-ms-client-request-id:62cefb52-8467-45f8-bbf3-f2051e7f9dab
x-ms-correlation-request-id:bc047309-a921-4a22-ba9f-d4665cef823d
Strict-Transport-Security:max-age=31536000; includeSubDomains
Access-Control-Allow-Origin:*
Content-Length:0
 
Microsoft-Extensions-Configuration-AzureAppConfiguration-Refresh: [Warning] Failed to get configuration settings from endpoint '<AppConfigEndPoint>'.
Exception thrown: 'System.TimeoutException' in Microsoft.Extensions.Configuration.AzureAppConfiguration.dll
An unhandled exception of type 'System.TimeoutException' occurred in Microsoft.Extensions.Configuration.AzureAppConfiguration.dll
The provider timed out while attempting to load.

[christothes]

Hi @v-uugwuoke
The logs indicate that you are successfully receiving a token from the managed identity endpoint.

[Managed Identity] Successful response received.
Fetched access token from host login.microsoftonline.com. 
Azure-Identity: [Informational] DefaultAzureCredential credential selected: Azure.Identity.ManagedIdentityCredential

And then the call to the resource endpoint failed with a 403, meaning your managed identity does not have permissions to the resource.

Azure-Core: [Informational] Request [cde3d04b-77cc-4166-a0b3-b7f916a72e1c] GET <AppConfigEndPoint>/kv?key=REDACTED&label=REDACTED&api-version=2023-10-01
Azure-Core: [Warning] Error response [cde3d04b-77cc-4166-a0b3-b7f916a72e1c] 403 Forbidden (00.1s)

To resolve this, you can either remove the managed identity configuration from the VM or grant the managed identity access to the resource.

[github-actions]

Hi @v-uugwuoke. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.