[suggestion] Different ways to load and unload players

[Znote]

The suggestion was originally posted on github in issue #1150 - Lag from excessive items in containers.

Lag from excessive items in containers [$10] · Issue #1150 · otland/forgottenserver

Sorry, the bug was caused by excessive items in the mail box ... if you can limit it would be great. The player was with some 70 thousand label in the mail box when the player entered the server wa...

github.com

This is an issue with 77 comments! And tons of suggested solutions and research. The current bounty is at 55$.

My suggestion is regarding the problems when a player logs in and forces the game to load his player items (which can cause lag when exceeding > 100k items).

Ofcourse the easy solution to massive player item containers is capacity limits (Which not everyone wants, and could cause problems with etc house item depot transfers). But even with limits, this should still aleviate uneccesary stress on the CPU when dealing with "container junkies", previous house owners, big players.

Playerloading memory overhead:

Minimal - Classic way, players are loaded into memory on login, and loaded out in logout.

Minimal Optimist - Classic way, but loads player out of memory 1 hour/xx minutes after logout (immediate shutdown/closeserver/server save). Prevents relog lag spam, but may cause one freeze per big player after the timeout.

Active - Loaded into memory on login, loaded out on shutdown.

PreActive - Players who has logged in past xx days are loaded in memory on startup. Loaded out on shutdown. (SQL inserts on save/close servers)

Semi Full / Semi PreActive - find heavy players and load them all on startup

Full - All players are loaded into memory on startup, loaded out on shutdown. (SQL inserts on save/close servers)

Unload = do the insert queries & free up the objects from memory etc...

With the big overhead (active) options, it might be a good idea to also add a "unload on upper RAM size cap". So if tfs.exe exceeds XXGB memory, trigger a Lua globalevent and unload the inactive players from memory to save up RAM.

That way, you can make do with a small box, but if you got 16+GB ram, you can just go all ham on it.

High RAM capable servers arent to rare these days, if your server can easily chug up enough RAM for the Minimal Optimist configuration, you may save yourself from players who otherwise might cause havoc on your server just to screw around with you.

 

[Delirium]

Where exactly is the delay caused? On the server or the database?

 

[Znote]

Where exactly is the delay caused? On the server or the database?

When you load a player: (Which happens at login)

https://github.com/otland/forgotten...b50c8069b25e85df1f2a/src/iologindata.cpp#L248

You do some operations like load player items:

https://github.com/otland/forgotten...b50c8069b25e85df1f2a/src/iologindata.cpp#L458

player depotitems:

https://github.com/otland/forgotten...b50c8069b25e85df1f2a/src/iologindata.cpp#L486

When these queries loads hundreds of thousands of rows, which is then iterated through and stored in an item map, it naturally becomes a big parsing job, for both the database and the server.

And when player is saved (which happens normally at logout):

https://github.com/otland/forgotten...b50c8069b25e85df1f2a/src/iologindata.cpp#L777

You delete the old copy of items from the database (player_items, player_depotitems etc), and inserts the new copy.

Now imagine having a million items in your house, loose that house so the million items gets transferred to your depot, and now you hate the server for whatever reason and all you do is login, logout, login, logout, login, logout.... All these items are being loaded, unloaded, loaded, unloaded from memory, causing alot of uneccesary stress on your cpu.

Now there might be optimisations that can be done here, but I don't neccesarily wants to explore it. This is just a sample of why it might be a bad idea to load and unload a player synonymous with login/logout.

My suggestion in OP (Opening thread post) is that we should let server owners choose which configuration fits them best, personally I would like to keep players loaded a while after they have logged out. And if its possible (and you have enough memory) to load everything at startup, why not do it?

This will potentially blow up your RAM usage, but if you got alot of RAM to blow, then I argue its worth it for a smoother in-game experience.

The "Playerloading memory overhead" options provided in OP is various compromises you are willing to do for your server, depending on how much available RAM you have at your disposal.

 

[Delirium]

Is updating the table with the differences only possible? I mean, if most items are the same, why update them? Also, indexing the table might save some processing time.

 

[Znote]

There might be optimisations that can be done here, but I don't neccesarily wants to explore it. This is just a sample of why it might be a bad idea to load and unload a player synonymous with login/logout process. Irrelevant of the direct performance optimisations that can be done during the loading and unloading of a player.

I talked to @Evil Hero about this a couple days ago, problem with at least some of my suggested configuration options is that we might need to introduce some duplicated code in the sources, to give the possibility to load and unload player at different times. I consider this something to look into deeper in TFS 2.0 though.

 

[Ochman]

Please keep in mind that I am willing to pay 50$ for the person who publicly shares the code that solves (or substantially postpones) this problem.

 

[oen432]

What if we load and unload players using separated threads? That should take care of bottleneck that causes said lag.

 

[Infernum]

What if we load and unload players using separated threads? That should take care of bottleneck that causes said lag.

We had this discussion internally, since the player can move and continue to do actions it can cause desync with the save and it wouldn't be saving the live state of the player of his/her inbox/depot items. That's why this thread was brought up, to create a solution to optimize save speed (especially on large servers) at the cost of memory.

 

[Ochman]

Just do not allow him to login again while he's being saved? So he cannot move and desync data, am I right? And block parcels etc (queue them).

 

[Infernum]

We're also talking about saving while ingame, not a global save where all players are kicked and the save occurs. The entire point of optimizing this is to reduce the freeze length. If you intentionally stop the player from doing actions while running a new thread to save player data, there's no point in having the thread at all because it will still take the same time to save and the player will be frozen for the exact same amount of time.

 

[gunz]

I suggest to talk about refactoring mailing system as well.

1. If you send mail and the receiver player is online that parcel is added directly to player’s mailbox - that part I would keep as is

2. If you send parcel to offline player (thats where the refactor should be done) - I would save the parcel in separate structure in Game class and add this parcel to player after he logs in. Also those saved parcels would be stored in separate sql table like ‘unreceived_mails’. Same will happen for house items.

This system would allow to do some more magic like async save/load players in separate thread (on login/logout).

 

[Tony32]

Do a Cipsoft "solution". Add a "feature" that adds a relog cooldown. lol.
And I do agree on the "update whats needed" idea. That would reduce db stress at least.

 

[Gesior.pl]

Problem reduced (binary save and player items RAM cache):
Server lags when login/logout with a lot of items - time to fix it! (https://otland.net/threads/server-lags-when-login-logout-with-a-lot-of-items-time-to-fix-it.266112/#post-2572676)

Could be reduced even more, if we decide to make it store player items in RAM, not clone items on load/save (as my cache).
To do this we need to handle all player logout item related events. Currently items 'problems' are handled by Player deconstructor that delete player 'inventory' (which remove items from 'decaying' list).