Windows Server Tuning - Expanding TCP/IP socket/port connection pool

 This support article provides solutions on what to do when you see an error in the Windows Server event viewer that contains:

Error Message : Only one usage of each socket address (protocol/network address/port) is normally permitted".

The full error message is at the bottom of this article - or you can jump to that point directly.

Explanation of Issue:

On Windows Server with ASP, .NET, IIS, etc - there is  normally only one (1) permitted usage of each socket address (protocol/network address/port)..  When a client initiates a TCP/IP socket connection to a server, the client typically connects to a specific port on the server and requests that the server respond to the client over a short lived, temporarily used TCP or UDP port. On Windows Server 2003 and Windows XP the default range of ports used by client applications is from 1025 through 5000. Under certain conditions it is possible that the available ports in the default range will be exhausted, meaning that there are no additional ports available at that moment.

Why this happens:   TCP/IP port exhaustion can occur on a client computer if the client computer is engaging in an unusually high number of TCIP/IP socket connections with a Server application. When a FileHold Server application is very busy with client requests, each FileHold client is making repeated, authenticated calls to the FileHold server application, and consequently there is repeated and continuous open and closing of connections. A client connection is comprised of 5 parts ( Protocol, Local IP, Local Port, Remote IP, Remote Port).

By default there are 4076 ports available in Windows 2003 because the usable port range is between port 1024 and port  5000. The FileHold web services communicate with the authenticated client repeatedly, and on a very busy server, the FileHold application can run into these port bottlenecks within the Windows server.  

On Windows 2008 servers, by default the operating system allows socket connections to be established between the ports 49152 – 65535; this equates to a little over 13,000 user ports.  This is a larger amount than the default of Windows 2003, but may need to be increased on busy server.

If all of the available ports are allocated to FileHold client sessions, and none are available, then the client experiences a condition known as TCP/IP port exhaustion. When TCP/IP port exhaustion occurs, client port reservations cannot be made and errors will occur in client applications that attempt to connect to a server via TCP/IP sockets. TCP/IP port exhaustion is more likely to occur under high load conditions than under normal load conditions. It can also be an indicator that the server hardware is underpowered and not adequate for the job.

At the bottom of this support article is a listing of event ID's that can be an indicator that this is occuring.  Each of these events will display a similar error:  TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host. 

EventID's:  100, 141, 177, 223, 288, 291, 293, 312, 323

Resolution for Windows 2003 Server and Windows 2008 Server:
Adjust the web server TCP/IP socket parameters to provide the IIS/ASP.NET FileHold Windows Server environment with adequate network sockets at a sufficient reusable rate. To do this, perform the following tasks in the Windows registry: 

Warning: Incorrect use of the Windows registry editor for Windows 2003 may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry.  Registry modifications should only be carried-out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry be made prior to making any registry changes.
 

Windows 2008 Server:

• On Windows 2008 servers, by default the operating system allows socket connections to be established between the ports 49152 – 65535; this equates to a little over 13,000 user ports.  This should be increased if you see the warning signs in the Windows event viewer logs.
• To display the current ports for the TCP protocol use the netsh command
• Open command prompt as administrator
• Then type

netsh int ipv4 show dynamicport tcp

-To increase the ports use the net shell command

netsh int ipv4 set dynamicport tcp start=1500 num=63000

This will provide 63,000 TCP user ports.

• When increasing the MaxUserPort, Microsoft recommends that port 1434 be reserved for use by the SQL Server Browser service (sqlbrowser.exe). 
• On a Windows 2008 server we simply start the port allocation at 1500 as shown in the previous Windows 2008 netsh command.

 

 

Windows 2003 Server:
1.  On the FileHold Application Server please locate the following key in the Windows registry: 

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

2.  Update the following values, or create them if they do not already exist: 

Name	Type	Default	Recommended (decimal)
MaxUserPort	DWORD	5,000	64,512
TCPTimedWaitDelay	DWORD	240	120
MaxFreeTcbs	DWORD	2,000	65,536
MaxHashTableSize	DWORD	512	16,384

 

NOTE:  
•When increasing the MaxUserPort, Microsoft recommends that port 1434 be reserved for use by the SQL Server Browser service (sqlbrowser.exe).

•On a Windows 2003 server add the following registry key

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

Add a Multi-string Value key.

Rename the new key to ReservedPorts

Add the values: 1434-1434

3.  For changes to take effect, REBOOT the server. 

 

Sample Snippet from a Windows Event viewer

MachineName : FILEHOLD1

TimeStamp : 6/30/2011 5:57:05 AM

FullName : Microsoft.Practices.EnterpriseLibrary.ExceptionHandling,

Version=2.0.0.0, Culture=neutral, PublicKeyToken=89d498f6b8d04fcc

AppDomainName :

/LM/w3svc/1/root/FH/FileHold/LibraryManager-1-129538815002500000

ThreadIdentity : Joe Tester

WindowsIdentity : TESTDOMAIN\FH_Service

      Inner Exception

      ---------------

      Type : System.Net.Sockets.SocketException, System, Version=2.0.0.0,

Culture=neutral, PublicKeyToken=b77a5c561934e089

      Message : Only one usage of each socket address (protocol/network

address/port) is normally permitted 10.53.17.10:80

      Source : System

      Help link :

      ErrorCode : 10048

      SocketErrorCode : AddressAlreadyInUse

      NativeErrorCode : 10048

      Data : System.Collections.ListDictionaryInternal

      TargetSite : Void DoConnect(System.Net.EndPoint,

System.Net.SocketAddress)

      Stack Trace :    at System.Net.Sockets.Socket.DoConnect(EndPoint

endPointSnapshot, SocketAddress socketAddress)

         at System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP)

         at System.Net.ServicePoint.ConnectSocketInternal(Boolean

connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address,

ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout,

Exception& exception)

Listing of Event ID's

Source	Event ID
Value	199, 100
Description	Only one usage of each socket address (protocol/network address/port) is normally permitted

 

Source	Event ID
Value	5000
Description	P1 acceleratorservice.exe, P2 7.5.2.0, P3 475edacb, P4 mscorlib, P5 2.0.0.0, P6 471ebc5b, P7 416b, P8 27, P9 system.runtime.remoting.remoting

 

Source	Event ID
Value	293
Description	TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.

 

Source	Event ID
Value	288
Description	TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.

 

Source	Event ID
Value	323
Description	TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.

 

Source	Event ID
Value	312
Description	TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.

 

Source	Event ID
Value	223
Description	TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.

 

Source	Event ID
Value	291
Description	TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.

 

Source	Event ID
Value	141
Description	TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.

 

Source	Event ID
Value	177
Description	TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.