Server lags when login/logout with a lot of items - time to fix it!

[Gesior.pl]

There is 4 years old Issue on github about it:

Lag from excessive items in containers [$10] · Issue #1150 · otland/forgottenserver

Sorry, the bug was caused by excessive items in the mail box ... if you can limit it would be great. The player was with some 70 thousand label in the mail box when the player entered the server wa...

github.com

There are some conceptions, but zero code to fix this issue. Kondrah made code for it, but he sells it for 500$.
I'm bored of writing code for servers for money.

I decided to write fix for it and release it on GitHub.

My current conception (60% of code is ready and tested on 1.3)
Player items cache - load player items once per server restart, save them in asynchronous thread.
Player items binary format in database - save and load player items up to 10 times faster.

Items = all player items: 'on player', depots and inbox.

1. Keep items copies in RAM in objects PlayerCacheData*. [DONE]
2. On player load, check, if there are items in cache for given GUID. If there are, copy them into Player*. If not, load from database and put copy in PlayerCacheData*. [DONE]
3. On player save, copy Player* items into PlayerCacheData*. Add given PlayerCacheData* to 'items to save queue'. [DONE]
4. New thread (added to TFS) will save items from PlayerCacheData* to database. Class similar to DatabaseTasks, running on own DB connection. [DONE]
5. Add option to save items in database in binary format. [tested on Kasteria.pl, not implemented for TFS 1.3 yet]
6. Add LUA functions to manipulate cache:

• possibility to load all players items in 'onStartup' event
• possibility to load all players items to cache, change config.lua to make it use 'binary save' and save all cache to database = change database from relational to binary on running server

Looking for better ideas and comments after I release code (probably on 2019-09-06).

Anyone interested in sending me some donations for releasing high quality code on GitHub?

 

[gohamvsgoku]

Will be awesome if this solution work in all tfs 1.x+ not only 1.3, i think have many servers 1.x+ waiting for this fix ^^

 

[Evil Puncker]

Will be awesome if this solution work in all tfs 1.x+ not only 1.3, i think have many servers 1.x+ waiting for this fix ^^

its already time to use latest version already, there are lots of fixes and features that have been done since 1.x to 1.3

 

[Znote]

About storing items in binary format, I have a suggestion to store unique item references in the database, this might conflict with that?

[suggestion] Unique item reference database structure

Check link for suggested schema changes: https://gist.github.com/Znote/2508c1d50fcca318f80b3ea4bc6f10a3 CREATE TABLE IF NOT EXISTS `items` ( `item_uid` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `itemtype` smallint(6) NOT NULL, `count` smallint(5) NOT NULL, `attributes` blob NOT NULL...

otland.net

We already store items in tile_store as binary, and I kinda wanted to extrapolate that back into integer column values.

 

[LordCompi]

In my humble opinion, it should be read at the time when container is loaded. As safety option it shouldnt load all items at the time but first eg. 40's one ordered by index in container. It's very curious that things like decaying isnt fixed by years and we are stand still in these bugs. Think about your ideas in the way of as an exploiter, eg. how reopining locker or reloging may affect the whole process.

There are needs of refactoring and optimalizations of mechanics, since a lot of users can't start a project of hes new ots without fighting things, which lay in github issues for years

 

[Gesior.pl]

About storing items in binary format, I have a suggestion to store unique item references in the database, this might conflict with that?

[suggestion] Unique item reference database structure

Check link for suggested schema changes: https://gist.github.com/Znote/2508c1d50fcca318f80b3ea4bc6f10a3 CREATE TABLE IF NOT EXISTS `items` ( `item_uid` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `itemtype` smallint(6) NOT NULL, `count` smallint(5) NOT NULL, `attributes` blob NOT NULL...

otland.net

We already store items in tile_store as binary, and I kinda wanted to extrapolate that back into integer column values.

Simple binary items implementation requires it to be 'flat' data. It can represent sequence of rows, but in case of few tables and relations it will be much harded to implement.

I just figured out that binary save of items in useless in case when saving/loading items can be executed in asynchronous thread.
Who cares if it's faster or not, when we got no more login/logout lag issue.

I think that player-cache-manager implementation will make your idea much easier to implement. I moved all items saving to one function:

forgottenserver-fast-save/src/playercachemanager.cpp at master · gesior/forgottenserver-fast-save

TEMPORARY REPO - TFS with player cache and binary items save - gesior/forgottenserver-fast-save

github.com

When I finish this code. All player items - database manipulation will be in one place. Items, depots and inbox in one place.

In my humble opinion, it should be read at the time when container is loaded. As safety option it shouldnt load all items at the time but first eg. 40's one ordered by index in container. It's very curious that things like decaying isnt fixed by years and we are stand still in these bugs. Think about your ideas in the way of as an exploiter, eg. how reopining locker or reloging may affect the whole process.

There are needs of refactoring and optimalizations of mechanics, since a lot of users can't start a project of hes new ots without fighting things, which lay in github issues for years

Looong time ago there was idea to make TFS work in real-time-with-database. Then someone made some benchmarks and realized it's impossible, because server would freez with 50 players online even on fastest dedic.

About loading '40 items'. It cannot work like that. When you open NPC trade window/market, you see count of all your items. When you place some offer, it got to find that item in ANY backpack and remove it.
Remember that database operations are much slower then C++ memory operations.

 

[Gesior.pl]

If anyone is interested in testing, I released first version of clean code.
It's not tested yet. I will test it when I get back home.

LUA functions to manipulate items cache are not ready yet.

All changes:
otland/forgottenserver (https://github.com/otland/forgottenserver/compare/master...gesior:issue_1150)

I plan to post final version as post 2000

EDIT:
Added LUA functions to manipulate cache:

PlayerCacheManager.clearCache()
PlayerCacheManager.clearPlayerCache(guid)
PlayerCacheManager.loadPlayerCache(guid)

With them, you can load all players items cache in 'onStartup' event.

 

[Gesior.pl]

Commited fix for not saving backpacks:

refs #1150 cache player items v4, fixed containers save, tested · otland/forgottenserver@ed06e27

A free and open-source MMORPG server emulator written in C++ - refs #1150 cache player items v4, fixed containers save, tested · otland/forgottenserver@ed06e27

github.com

Tested whole code on local server.
Relog time (from onLogout to onLogin event) with 168k items (on player):
- with cache: 90 - 114 ms
- without cache: 4003 - 4858 ms

Relog time (from onLogout to onLogin event) with 336k items (168k on player + 168k in inbox):
- with cache: 131 - 178ms
- without cache: 6203 - 7151 ms

Compile target: Release, x64. Tested at i7 4x4.7GHz, SSD SATA Samsung Evo 860.

With cache relog is much faster, but sometimes (after 3-5 fast relogs) it freezes.
Saving items to database goes in separate thread and MySQL connection, but somehow it affects login/logout mysql queries.


I have to add binary items storage and some cache saving threshold (like 5-15 seconds) to block possibility to overload database by sending a lot of parcels to offline player.

EDIT:
Wrote binary items and tested.
Relog with 336K items (168k on player + 168k in inbox):
- binary items save: 252 - 278 ms
- normal items save: 4111 - 4150 ms

Times are lower than in previous tests. I optimized MySQL config.

 

[Gesior.pl]

Woohoo! Post 2000!
4373 days after registration

Added binary items save to my branch 1150:

refs #1150 save player items in binary format · otland/forgottenserver@4f7a532

A free and open-source MMORPG server emulator written in C++ - refs #1150 save player items in binary format · otland/forgottenserver@4f7a532

github.com

All changes from tfs 'master' to version with binary items save, player cache and save items threshold (to block people spamming mailbox):

Comparing otland:master...gesior:issue_1150 · otland/forgottenserver

A free and open-source MMORPG server emulator written in C++ - Comparing otland:master...gesior:issue_1150 · otland/forgottenserver

github.com

Created Pull Request:

Issue 1150 fix by gesior · Pull Request #2657 · otland/forgottenserver

3 changes to reduce problems described in #1150 Cache player items in RAM. Allow LUA script to load all players items into RAM at server start. Save items in asynchronous thread with own database ...

github.com

 

[HjugO]

https://patch-diff.githubusercontent.com/raw/otland/forgottenserver/pull/2657.patch

 

[Gesior.pl]

After comment from SaiyansKing ( Issue 1150 fix by gesior · Pull Request #2657 · otland/forgottenserver (https://github.com/otland/forgottenserver/pull/2657#issuecomment-530441011) ) I wrote update for binary save.
Now it works even more binary.

Tests of new version
Relog with 336K items (168k on player + 168k in inbox):

• binary items save: 252 - 278 ms
• NEW binary items save: 140 - 180 ms

I will prepare clean commit and pull request tommorow.

EDIT:
Relog on player with no items takes around 90ms, so change in items saving time is very big.

EDIT 2:
Created pull request with clean version of code:

Issue 1150 player items cache and binary items save v2 by gesior · Pull Request #2659 · otland/forgottenserver

Updated version of binary save and player items cache (v2). Optimized binary save/load to make it 2-4 times faster then before. 3 changes to reduce problems described in #1150 Cache player items i...

github.com

 

[HeberPcL]

@Gesior.pl, did that proposal die?

 

[pink_panther]

lol, how do you end up with 70k labels... intentional spam?

 

[Gesior.pl]

@Gesior.pl, did that proposal die?

As everything related to TFS. Nobody cares. Nobody test/review/accept PR.

 

[zbizu]

As everything related to TFS. Nobody cares. Nobody test/review/accept PR.

The people who are willing to test things are usually devs that don't own large servers. The people who own large servers are usually busy with keeping them up (making updates, fixing bugs, etc). The problem is that nobody really wants to keep an unstable server and nobody wants to play one either. Unstable server = losing players. Nobody wants to lose players when so few are still playing.

If we were to run some experimental server, it would have to have some thing to keep people interested. Pretty hard to do that nowadays.

 

[Nekiro]

No, it's not like nobody cares, this thing breaks backwards compatibility very badly (I see you made it optional though) + introduces more incoveniences like saving blobs to database (which we do in houses anyway) and being unable to see any item by pulling a query. These are the main problems with binary saving.

what cip is probably doing and what I would do is load all players to memory and save periodocally or at shutdown.

 

[Gesior.pl]

No, it's not like nobody cares, this thing breaks backwards compatibility very badly (I see you made it optional though) + introduces more incoveniences like saving blobs to database (which we do in houses anyway) and being unable to see any item by pulling a query. These are the main problems with binary saving.

what cip is probably doing and what I would do is load all players to memory and save periodocally or at shutdown.

I made 2 things in this PR:

• binary items save
• player items cache

Both are configurable from config.lua:

playerItemsCache = false
playerItemsCacheSaveThreshold = 5000
binaryPlayerItems = false

what cip is probably doing and what I would do is load all players to memory and save periodocally or at shutdown.

That's what playerItemsCache does. You can add globalevent 'startup', that executes PlayerCacheManager.loadPlayerCache(guid) from every guid in database.
Using playerItemsCache make server load players from RAM cache and save items in second thread (not dispatcher), so there is no lag on logout.

About looking for items in database - you need it only in some special cases (items cloned etc.). It's also possible with this system.
Cache all items from database to RAM. Change in config.lua 'binarySave = false'. Reload config. Save all items from RAM cache to database - they will be saved using old relational items structure.

EDIT:
I also made new 'decay system'. Which uses less CPU and has higher time precision of decaying. It does not break any compatibility. It's also not tested by anyone and ignore on github.

 

[Nekiro]

I made 2 things in this PR:

• binary items save
• player items cache

Both are configurable from config.lua:

playerItemsCache = false
playerItemsCacheSaveThreshold = 5000
binaryPlayerItems = false

That's what playerItemsCache does. You can add globalevent 'startup', that executes PlayerCacheManager.loadPlayerCache(guid) from every guid in database.
Using playerItemsCache make server load players from RAM cache and save items in second thread (not dispatcher), so there is no lag on logout.

About looking for items in database - you need it only in some special cases (items cloned etc.). It's also possible with this system.
Cache all items from database to RAM. Change in config.lua 'binarySave = false'. Reload config. Save all items from RAM cache to database - they will be saved using old relational items structure.

EDIT:
I also made new 'decay system'. Which uses less CPU and has higher time precision of decaying. It does not break any compatibility. It's also not tested by anyone and ignore on github.

There is no pr for new decay system, or I am missing something?

 

[HeberPcL]

I thought the code was not working well, after your tests. After seeing your comment.

 

[Gesior.pl]

I thought the code was not working well, after your tests. After seeing your comment.
View attachment 51543

Only version made for 8 years old engine.