A production issue tale

The system has been in production for 6 years. Recently, it was upgraded to .NET10. All related NuGet packages were upgraded too.

A support ticket came in. Users experienced an error message occasionally. A search in the Application Insights showed:

And the exception trace:

at System.ThrowHelper.ThrowInvalidOperationException_ConcurrentOperationsNotSupported (System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.Filters.SaveTempDataFilter.SaveTempData (Microsoft.AspNetCore.Mvc.ViewFeatures, Version=10.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters (Microsoft.AspNetCore.Mvc.Core, Version=10.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)

The exception was thrown after the application code had finished processing.

It is a classic race condition problem, usually involving using Dictionary, which is not thread-safe, in parallel.
With Claude Code support, the candidate emerged. The application code stores information in HttpContext, which in general does not encourage. The "HttpContet.Items" is a dictionary, not thread-safe.

We need "OperationId" for troubleshoot purposes. We had it since the beginning with Application Insights since 1.0 to 2.x. The above is for Application Insights 3.x The manipulation of "context.Items" caused the problem.

Before the upgrade, the log was called once in the HttpRequest thread. From 3.x, Application Insights changed to Open Telemetry (Otel). In the new model, to support distributed logs, every external calls are logged, SQL, Cosmos, Http, ... calls. The "OnEnd" is called in the continuation thread of Otel; AKA another thread.

The application has a few SQL calls for some endpoints; as seen the timeline image, the first two calls ended at the same time. With enough "luck", they both modified the dictionary on the same key. The dictionary implementation marks "the internal state corrupted."

In the Response step of ASPNET MVC pipeline, it stores View state by looking for View state from HttpContext. The corrupted state was triggered.

I have had a few incidents regarding careless use of Dictionary. Hopefully, I can remember it next time. Until then, a few notes:

Do not try to modify the internal state of framework from the application code. If must do, think twice, and think again, and again.
Dictionary and thread-safe. It is easy to careless use it in the development and then forget.
Upgrade framework. Read and understand all breaking changes and what each of them might impact the application.

By the way, the solution is simple. Remove those lines of code and use Activity.Current.TraceId directly.