Hi Everyone, i have the Following issue:
Environment
The scenario is the following
7 application servers running Microsoft Windows Server 2008 SP2
These servers originally had .net Framework v3.5
1 database server running Microsoft SQL Server 2008 R2 with Microsoft Windows Server 2008 SP2, acting as principal DB with mirroring
For the communication between clients and servers, we are using WCF, it is not ASP .net (we do not use IIS)
For the queries to the DB we are using entity Framework 3.5
Problem
The short story is that we fell in the situation of the size of the thread pool in .net Framework 3.5 which is limited by the number of cores and in v.4.5.2 it is 100 times bigger. So we decided to upgrade it because the system was too slow after few days of continuous running.
Now we have a problem that seems to point to a faster application server sending much more requests to our database and the database is generating timeout exceptions under heavy load.
The .net Framework was upgraded from v3.5 to v4.5.2 in the 7 application servers and eventually we started to experience SQL timeout exceptions like the following one:
System.Data.EntityCommandExecutionException: An error occurred while executing the command definition. See the inner exception for details. -àSystem.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. This failure occurred while attempting to connect to the Principle server. -àSystem.ComponentModel.Win32Exception: The wait operation timed out
--- End of inner exception stack trace ---
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.EntityClient.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
--- End of inner exception stack trace ---
at System.Data.EntityClient.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
at System.Data.Objects.Internal.ObjectQueryExecutionPlan.Execute[TresultType](ObjectContext context, ObjectParameterCollection parameterValues)
at System.Data.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)
at System.Data.Objects.ObjectQuery`1.System.Collections.Generic.Ienumerable<T>.GetEnumerator()
at Thales.Echo.Services.IMS.Services.IncidentService.GetSubscriberHistory(String callerNum, DateTime callerDateTime, eDataBaseType databaseType, List`1& incidentCalls) 0
Possible workaround for the problem
We tried to fix it by setting the command timeout for the methods having the exception but there are a lot of methods that can have this problem in our source code.
ANY IDEAS OF WHAT COULD BE HAPPENING???