High CPU/RAM usage after 8h+

[skysz]

Hello everyone!

I'm facing a problem that I don't know how to solve by myself anymore. I'm running a Global TFS 1.2 server in a Xeon E-2286G 8vCPU 4Ghz/8GB RAM dedicated VPS and the server used to have just ~100 players. Since I've put it on Otservlist, we started to get ~200 players and since then we are facing problems related to CPU and RAM Leaks.

The problem is this leak comes only after 8-10h of uptime. When I restart the tf2 process, it stays on 40-50% of CPU and ~60% of RAM usage even if those 200 players stay online. But after that 8-10h, the CPU goes to 90%+ all the time, lagging the server and RAM usage keeps going UP until it gets 90%+ and the SO kills the tfs2 process.

I think I can "solve" the RAM issue by just buying more RAM, so after SS it goes down again. But the CPU I don't how what to do, since is a really nice processor with a good clock speed. I don't know if I can afford a better processor.

Here is a perf top that I did:


And htop:


RAM usage (I'm restarting the server twice a day to avoid a new episode of Linux killing the PID and rollbacking the server, so thats why since 24/05 the RAM is not going anymore to 90%+):


Could anyone help me to find out what can I do to start a better investigation? I can pay via Paypal/donate to TFS project/whatever you guys find better in gratitute.

Thanks!!!

 

[platano]

Have you seen any other unrecognized PIDs using ps aux?

 

[skysz]

Hello,

there's not strange PID. I've checked this as well.

Seems like I'm having CPU/RAM rising as a player logs in or logs out, and after a lot of episodes the CPU goes to a level that there's no other way then restarting the server . I just didn't figure it out what I have to look into to fix.

 

[roriscrave]

its a tibia server? u have a custom things?

Hello,

there's not strange PID. I've checked this as well.

Seems like I'm having CPU/RAM rising as a player logs in or logs out, and after a lot of episodes the CPU goes to a level that there's no other way then restarting the server . I just didn't figure it out what I have to look into to fix.

 

[Lessaire]

You've introduced a memory leak somewhere. There is no way to really determine what's causing it without access to the installation, or a complete replica.

If you wish to attempt to locate the problem yourself, you will need Gnu Debugger and Valgrind.

This is beyond the scope of the support forum. If you wish to hire a coder to do this for you, report your thread and ask it to be moved to the Jobs forum.

 

[Ninja]

While it may not seem super obvious, you should take a look how your server are constructing Player userdata objects (and make sure it doesn't store them in tables that are never freed). Constructing one should under no circumstances be that slow.

 

[skysz]

Hi Ninja,

In fact, I'm working on the possibility of a bad written script that leads to these objects that aren't freed correctly when a player logs out.

Unfortunately, it goes beyond of my knowledge and I will do what @Lessaine suggested. I'll hire a coder.

Thank you guys!

 

[fabian766]

Looking at the fact that the most cpu is consumed by luaPlayerCreate and deque_iterator from SchedulerTask consume so much cpu it seems that some of your scripts have addevent leakage, it also explain the "memory leak" because addevent needs to be cached. Someone of your players know how to use this leak and uses it.

 

[Lessaire]

Someone of your players know how to use this leak and uses it.

Indeed, that is a strong possibility.

I would debug the problem on a replica installation and once you determine what the bug is leave the bug active your server. Insert new code expanding the logging from the responsible script, and determine which player is doing this. Do not reveal yourself. Stay ghost and watch what the player is up to. Resist the urge to confront them, you'll only tip them off that their item duping parade is over.

You want to determine the extent of what they've done, and how many players are involved. The best course of action once you have all the information is to perform a SQL query dropping all the duped items, during the downtime when you fix the bug. Then you can confront or start toying with the culprits if you're so inclined. Things like making fancy look-alikes of valuable items that explode when picked up. Or monsters that steal XP.

 

[gudan garam]

Indeed, that is a strong possibility.

I would debug the problem on a replica installation and once you determine what the bug is leave the bug active your server. Insert new code expanding the logging from the responsible script, and determine which player is doing this. Do not reveal yourself. Stay ghost and watch what the player is up to. Resist the urge to confront them, you'll only tip them off that their item duping parade is over.

You want to determine the extent of what they've done, and how many players are involved. The best course of action once you have all the information is to perform a SQL query dropping all the duped items, during the downtime when you fix the bug. Then you can confront or start toying with the culprits if you're so inclined. Things like making fancy look-alikes of valuable items that explode when picked up. Or monsters that steal XP.

Just, the imagination on this guy. woah