gcmcnutt
New Member
- Joined
- Jul 13, 2008
- Messages
- 3
- Reaction score
- 0
We've isolated a primary memory leak issue on our TFS server. The scenario is this:
- we have a fairly dense training area in the map
- folks come into the training area, start an attack
- they leave the terminal running
- AND, they are on a relatively slow link [perhaps < 10kbit download during intervals]
- Then, the server gets into a state where the send messages are sent faster than can be transmitted to the client
I've simulated this in a dense training area: by adding a network bandwidth limiter to the client, setting up an attack and instrumeting the server to see the connection output grow without bounds.
On our production server, we've seen up to 1MB of VM growth per *second* on the server -- growing to beyond the VM limit of the host, thus crashing the server.
I've prototyped a connection buster for when an output queue grows beyond a certain message count. If this happens we force a disconnect [to trigger the equivalent behavior as a lost link would]. Here's a code snipped from connection.cpp::send
#ifdef __DEBUG_NET__
fprintf(stdout, "Connection: send another [0x%x][%d:%d]\n", this, m_pendingWrite, m_outputQueue.size());
#endif
m_outputQueue.push_back(msg);
m_pendingWrite++;
if (m_pendingWrite > 500) {
fprintf(stdout, "Force closing slow connection [0x%x][%d]\n",
this, m_pendingWrite);
closeConnection();
}
If anyone has worked on this behavior or wondered why their memory grows over time, please check around this area.
- we have a fairly dense training area in the map
- folks come into the training area, start an attack
- they leave the terminal running
- AND, they are on a relatively slow link [perhaps < 10kbit download during intervals]
- Then, the server gets into a state where the send messages are sent faster than can be transmitted to the client
I've simulated this in a dense training area: by adding a network bandwidth limiter to the client, setting up an attack and instrumeting the server to see the connection output grow without bounds.
On our production server, we've seen up to 1MB of VM growth per *second* on the server -- growing to beyond the VM limit of the host, thus crashing the server.
I've prototyped a connection buster for when an output queue grows beyond a certain message count. If this happens we force a disconnect [to trigger the equivalent behavior as a lost link would]. Here's a code snipped from connection.cpp::send
#ifdef __DEBUG_NET__
fprintf(stdout, "Connection: send another [0x%x][%d:%d]\n", this, m_pendingWrite, m_outputQueue.size());
#endif
m_outputQueue.push_back(msg);
m_pendingWrite++;
if (m_pendingWrite > 500) {
fprintf(stdout, "Force closing slow connection [0x%x][%d]\n",
this, m_pendingWrite);
closeConnection();
}
If anyone has worked on this behavior or wondered why their memory grows over time, please check around this area.