Process Crash in ASP.NET Framework 4.8

[marafiq]

Stack overflow exception randomly crashed the process. It only happened twice in last few days. Did it simply got stack overflow because the number of retries? Appreciate the help

Packages used

<PackageReference Include="Azure.Identity" Version="1.13.2" />
        <PackageReference Include="Microsoft.Extensions.Caching.Abstractions" Version="9.0.3" />
        <PackageReference Include="Microsoft.Extensions.Caching.Memory" Version="9.0.3" />
        <PackageReference Include="Microsoft.Extensions.Configuration" Version="9.0.3" />
        <PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="9.0.3" />
        <PackageReference Include="Microsoft.Extensions.Configuration.AzureAppConfiguration">
            <Version>8.1.1</Version>
        </PackageReference>
        <PackageReference Include="Microsoft.Extensions.Configuration.Binder" Version="9.0.3" />
        <PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="9.0.3" />
        <PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="9.0.3" />
        <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="9.0.3" />
        <PackageReference Include="Microsoft.Extensions.Options" Version="9.0.3" />
        <PackageReference Include="Microsoft.Extensions.Primitives" Version="9.0.3" />
        <PackageReference Include="Microsoft.FeatureManagement">
            <Version>4.0.0</Version>
        </PackageReference>
        <PackageReference Include="System.Configuration.ConfigurationManager">
            <Version>9.0.3</Version>
        </PackageReference>

How It is configured

var builder = new ConfigurationBuilder();
            IConfigurationRefresher refresher = null;
            builder.AddAzureAppConfiguration(options =>
            {
                var siteNameAsLabelFilter = System.Configuration.ConfigurationManager.AppSettings["system.environment"].ToLower();
                var azureAppConfigConn = System.Configuration.ConfigurationManager.ConnectionStrings["AzureAppConfiguration"].ConnectionString;
                refresher = options.Connect(azureAppConfigConn)
                    .ConfigureStartupOptions(startupOptions => startupOptions.Timeout = TimeSpan.FromSeconds(5))
                    .UseFeatureFlags(featureFlagOptions =>
                    {
                        // Label filters order is important. The last filter will override the value of the first filter if the key exists in last filter.
                        featureFlagOptions
                            .Select(KeyFilter.Any, "global")
                            .Select(KeyFilter.Any, siteNameAsLabelFilter)
                            .SetRefreshInterval(TimeSpan.FromMinutes(1));
                    })
                    .ConfigureClientOptions(clientOptions =>
                    {
                        clientOptions.Retry.MaxRetries = 10;
                        clientOptions.Retry.MaxDelay = TimeSpan.FromSeconds(30);
                        clientOptions.Diagnostics.ApplicationId = siteNameAsLabelFilter;
                    })
                    // Label filters order is important. The last filter will override the value of the first filter if the key exists in last filter.
                    .Select(KeyFilter.Any, "global")
                    .Select(KeyFilter.Any, siteNameAsLabelFilter)
                    .ConfigureRefresh(refresh =>
                    {
                        // app configurations will only be refreshed if the sentinel key is updated
                        refresh
                            .Register("Sentinel", "global", true)
                            .SetRefreshInterval(TimeSpan.FromMinutes(1));
                    })
                    .GetRefresher();
            }, true);

Stack Trace

FaultingExceptionFrame
HelperMethodFrame
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
Microsoft.Extensions.Configuration.AzureAppConfiguration.TracingUtils+d__3.MoveNext
System.Threading.ExecutionContext.RunInternal
System.Threading.ExecutionContext.Run
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction
System.Threading.Tasks.Task.FinishContinuations
System.Threading.Tasks.Task.Finish
System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult mscorlib]].TrySetException
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult mscorlib]].SetException
Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider+<>c__DisplayClass38_2+<b__0>d.MoveNext
HelperMethodFrame
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider+<>c__DisplayClass38_2+<b__0>d.MoveNext
System.Threading.ExecutionContext.RunInternal
System.Threading.ExecutionContext.Run
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run
System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1[[System.Boolean mscorlib]].InvokeContinuation
System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1[[System.Boolean mscorlib]].SignalCompletion
Azure.Data.AppConfiguration.ConditionalPageableImplementation+d__10.MoveNext
HelperMethodFrame
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
Azure.Data.AppConfiguration.ConditionalPageableImplementation+d__10.MoveNext
System.Threading.ExecutionContext.RunInternal
System.Threading.ExecutionContext.Run
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction
System.Threading.Tasks.Task.FinishContinuations
System.Threading.Tasks.Task.Finish
System.Threading.Tasks.Task`1[[System.__Canon mscorlib]].TrySetException
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon mscorlib]].SetException
Azure.Data.AppConfiguration.ConditionalPageableImplementation+d__14.MoveNext
HelperMethodFrame
Azure.Data.AppConfiguration.ConditionalPageableImplementation+d__14.MoveNext
HelperMethodFrame
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
Azure.Data.AppConfiguration.ConditionalPageableImplementation+d__14.MoveNext
System.Threading.ExecutionContext.RunInternal
System.Threading.ExecutionContext.Run
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction
System.Threading.Tasks.Task.FinishContinuations
System.Threading.Tasks.Task.Finish
System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult mscorlib]].TrySetException
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult mscorlib]].SetException
System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder.SetException
Microsoft.Extensions.Configuration.AzureAppConfiguration.UserAgentHeaderPolicy+d__3.MoveNext
HelperMethodFrame
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
Microsoft.Extensions.Configuration.AzureAppConfiguration.UserAgentHeaderPolicy+d__3.MoveNext
System.Threading.ExecutionContext.RunInternal
System.Threading.ExecutionContext.Run
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction
System.Threading.Tasks.Task.FinishContinuations

[amerjusupovic]

@marafiq From what I can tell you're right, lowering the MaxRetries should significantly lower the chance of this happening. There may be some steps we can take from the provider side to mitigate this issue, but I'm not entirely sure what that would include yet.

[drago-draganov]

The retry settings shouldn't cause StackOverflow exception.

@marafiq, can you share a request-id to your AppConfiguratoin store instance?

[marafiq]

I don't have that id but I am using premium azure app config with replication enabled. But even then why it can cause stackoverflow- it happened one time in prod with 3 retries but I can not get more details than that. Continuously running process crash monitor slow down the app so much and this happens from no where

[marafiq]

If it helps I m running the app in P2V3 - App service plan

[drago-draganov]

@marafiq
It's definitely not expected. Can you reach out to (AzureAppConfig at microsoft.com)? I will be happy to assist you.