TFS 1.X+ Optimizing player saves & when saves execute

[Itutorial]

Hello everyone,

I noticed that all player save information happens in one place. IOLoginData::savePlayer() I do not really like that in order to save player items I have to save everything else with it. So I fragmented the IOLoginData::savePlayer() method into the following methods.

IOLoginData::savePlayerInstantSpells();
IOLoginData::savePlayerInventoryItems();
IOLoginData::savePlayerDepotItems();
IOLoginData::savePlayerInboxItems();
IOLoginData::savePlayerStoreInboxItems();

I added the instant spell saves in Player::learnInstantSpell() and Player::unLearnInstantSpell()

Then I added the following into Game:layerMoveItem()

ReturnValue ret =
    internalMoveItem(fromCylinder, toCylinder, toIndex, item, count, nullptr, 0, player, nullptr, &fromPos, &toPos);
if (ret != RETURNVALUE_NOERROR) {
    player->sendCancelMessage(ret);
    return;
}

// Player moved item into or out of depot
if (dynamic_cast<const DepotChest*>(fromCylinder) || dynamic_cast<const DepotChest*>(toCylinder)) {
    player->saveDepotItems();
}

// Player moved item out of store inbox
if (dynamic_cast<const StoreInbox*>(fromCylinder) && !dynamic_cast<const StoreInbox*>(toCylinder)) {
    player->saveStoreInboxItems();
}

// Player moved item into or out of inbox.
if (dynamic_cast<const Inbox*>(fromCylinder) || dynamic_cast<const Inbox*>(toCylinder)) {
    player->saveInboxItems();
}

// Player moved an inventory item.
if (fromPos.x == inventoryXPosition || toPos.x == inventoryXPosition) {
    player->saveInventoryItems();
}

I also slowed down item moves to 100ms.

For the inbox items I also had to make it save when players send mail to another.

Right now everything works properly, anytime an item is moved in each scenario that it would need to be saved it is. I am not using store inbox atm but I assume it works just as well. This freed up a lot of "lag" with player saves

So my question is what might I be missing? If I am not missing anything, would this be something that we should see in the source code? Doing it this way allows the database to be completely up-to-date incase of a crash. I also added saving with house items and limited them to being moved every 2 seconds because it has to save all house items. Though that is not included in this.

 

[Gesior.pl]

Obviously the best solution is a server that never crashes but….

Yes. This is a real solution.

There is also other solution: save server on crash. You can process SIGSEGV signal (crash) and call function that saves all players online and houses. Of course there is a chance that item structure of some player/house is a reason of crash (or failure of connection to MySQL), in this case it won't save, because save itself will crash again. EDIT: You can do it in C++, you don't need any special 'anti rollback script' for gdb.

You cannot save items every time someone moves item. It was discussed and benchmarked a lot 10-20 years ago. As I remember, if was on list of planned TFS 1.0 features, but it was dropped.
When player moves single item, it's not a problem, but if player moves around BP with 1k items inside depot, it will call 1k INSERTs every move. Even, if you replace current save code with binary format or save items into Redis, it's still use too much CPU just to serialize items.
It was tested in most optimized scenario: 'save' did not write anything anywhere, just cloned items in RAM and passed them to second thread, that saved them. It still used too much CPU, when players moved BP around in house (as in your system, just added to HouseTile to save house, when item is moved).

Binary items format even with BLOB in MySQL is much faster than current relational version. Players load/save time is 3-40 times faster after changing items to binary format. You can make it even faster by saving items directly in files on SSD/RAM disk.
Binary items save for TFS 1.4:

• code: feat: Store player items in binary format (#1150) · gesior/forgottenserver-gesior@cd4c58b (https://github.com/gesior/forgottenserver-gesior/commit/cd4c58b2fd2177cd2d729c70dfe73bcd1294077c)
• documentation: doc: Player items binary format documentation (#4819) · gesior/forgottenserver-gesior@fee30e9 (https://github.com/gesior/forgottenserver-gesior/commit/fee30e94c73d2b4c16c0735b4f527f0ebb8239a5)
• compilation error fix: fix: Player items binary format compilation error · gesior/forgottenserver-gesior@c53f3e0 (https://github.com/gesior/forgottenserver-gesior/commit/c53f3e053f68f5b34691a51636fcb214e419a799)

 

[Itutorial]

Thanks for that gesior. As I suspected the reason it won’t work has to do with these weird things TFS allows. I wonder, how tibia handles it. There aren’t many games that allow 1 millions items to be in a single bag stack which is in 5 other bags all having a million. This is where I have to move away from TFS development for my own server. I won’t have to deal with this type of situation because bag space and capacity are going to limit it. My server will utilize a system closer to WoWs bag and bank system. Including bags not being able to be put inside each other other than your first bag (const_slot_backpack) which cannot be replaced and is never lost. Which is limited to 8 bags that can be added to it.

I will say it’s possible to limit the queries by utilizing a loop to get all the items that need to be updated to insert into a single query for execution but yes I see how it can get bogged down very easily with large item counts.

Thank you everyone that said something. I think I know what I should be doing from here. Unfortunately, TFS is stuck until someone better comes along (not me) lmao.

 

[Gesior.pl]

There aren’t many games that allow 1 millions items to be in a single bag stack which is in 5 other bags all having a million.

This is a thing you must fix.
That's what RL Tibia did long time ago:

• limit number of items in depot
• save players/houses only on server restart, if it crashes, all players/items go back to last server restart

and this is what many big OTSes do:

• limit number of items on house tile (including items inside bags)
• limit total number of items in container (bag/depot) - including sending parcels to full depot (except adding items to depot, when player leaves house)
• limit 'deepness' of containers (how many containers inside other containers server allow to add)

You can't add it with few simple IFs ex. 'on add item -> count items inside bag and block adding new item, if more than 5k'
In case of 1 million items, it would loop over all 1kk items in bag to count them every time players try to move item. It must be added like 'weight' of containers, every item move it propagates to all 'parent' containers change in weight. When you check weight of container, it does not calculate anything, weight is kept as container attribute:

forgottenserver/src/container.h at 1.4 · otland/forgottenserver

A free and open-source MMORPG server emulator written in C++ - otland/forgottenserver

github.com

Count of items inside and deepness of containers must propagate same way.

 

[Tofame]

Basically what Gesior and Nightwolf have said is the best solution. Your personal approach scared me to be honest, as I dislike a lot the idea of spamming sql queries.

 

[JeiDi]

If each item move would save items to the database...

I've just checked old logs (there were only 100 players online at the time), and there were 315k move actions that day, so you'd have 315k database save queries? Well...

Sharing that to put your idea into a real numbers perspective.

 

[Itutorial]

Basically what Gesior and Nightwolf have said is the best solution. Your personal approach scared me to be honest, as I dislike a lot the idea of spamming sql queries.

I am on the same page. However, when we say things like spamming we need to be very precise. Gesior pretty much outlined exactly what I am going to do, but I am for sure keeping queries executing at the time of change. Of course it will take a lot of work to make it optimized for high player counts.

If each item move would save items to the database...

I've just checked old logs (there were only 100 players online at the time), and there were 315k move actions that day, so you'd have 315k database save queries? Well...

Sharing that to put your idea into a real numbers perspective.

View attachment 93297

315k is that a lot of database queries? Let’s think about websites that require almost everything to be displayed from the database (like social media sites) there are billions of not trillions of database tasks happening per day. I know they have countless, very expensive servers yada yada. And have spent billions making their product as optimized as possible but we are not talking about that size. 315k queries sounds like a lot but if we put it in this format: let’s say our single delete and update query takes 1-2ms (maybe we can do even better) that would mean 6 seconds total out of 24 hours. Websites also have an inherit disadvantage because most are made with higher languages.

Again I am past the idea of this being put into TFS. You guys are probably all correct in most of what you are saying, that this would be very difficult to achieve if at all. I will find out I guess. I am also completely fine with having servers that only handle 200 players if need be. My main priority is data loss. Anytime I have run a server the biggest issue was always something to do with crashes and data loss. (Other than the old shitty monster AI where monsters couldn’t catch a player 10x slower than them which I have/am fixing)

Thank you @Nekiro @Gesior.pl @Codinablack @Ianoyo @ALS @Night Wolf @JeiDi @Tofame

 

[Nekiro]

Websites are usually asynchronous, your server isn't. Every query blocks main thread from executing in your scenario and directly impacts the performance of everything else that is happening in the server.

 

[Itutorial]

Websites are usually asynchronous, your server isn't. Every query blocks main thread from executing in your scenario and directly impacts the performance of everything else that is happening in the server.

Really? I feel like I was told it was async in TFS now. I guess I have another problem to solve.

 

[Codinablack]

Really? I feel like I was told it was async in TFS now. I guess I have another problem to solve.

Hell yeah , rock on brother!

Thinking about it, there is no reason that the save's need to block the main thread... and thinking more about it, same can be said about the loads.

Please note, in TFS the main thread, is not the "main" thread, it's the dispatcher.

Since saving only requires immutable data, this is perfectly fine to run asynchronously, and, since we really only load the database data at key points, all which occur before needing to interact with the game world, I believe this should be easy to transition to as well!

 

[jeanphilip]

this was achieved in tvp i think so as well with better monster behavior . why does no one took it's code ?there is release out there already if not multiple of them

 

[Codinablack]

this was achieved in tvp i think so as well with better monster behavior . why does no one took it's code ?there is release out there already if not multiple of them

"this was achieved in tvp"

How about some context... what was achieved? You mean incremental saving, like the OP is doing with his thread post...? If so, no it wasn't...
Do you mean asynchronous saving...? If so, then no again, it was not...

Ezz definitely made a custom serialization system, which is pretty nice, but it's not running EVERYTHING exclusively through that system, so if you mean he got rid of the overhead of the mysql server... then yeah, no.. wrong again!

So other than trying to encourage others to go download leaked sources....what exactly was the point in your post?

 

[Gesior.pl]

Thinking about it, there is no reason that the save's need to block the main thread... and thinking more about it, same can be said about the loads.

There is no reason to do it in sync, but benchmarks say that moving it to separate thread makes it only 2-3 times faster, so IDK if it's worth all work required to implement this.
If save of items is in separate thread, but player is still 'in game' (ex. someone called player:save() in Lua on level advance), to save items in separate thread, you have to clone whole items structure. I did benchmarks few years ago and cloning items structure was 2-3 times faster than serializing it and writing to MySQL as 1 big blob.
It's not a simple 'clone items structure to other thread' and let it write them async to database. If there are items to save queued in other thread, waiting for write to database (ex. player died, it started 'async save'), other functions like 'login' or 'send parcel' cannot execute, until this async save finish.

Finally, to implement it, I added separate thread with 'items cache., 'dispatcher' thread only reads/writes items from/to 'items cache', it does not read/write items to database directly. If player items are not cache, it loads them in sync from 'items cache' (there was config.lua option to preload all items on server start). If items are in cache, last version queued for save is cloned to dispatcher thread. Items save is always 'clone items from dispatcher to items cache thread'.
There were some crashes, nobody was interested to test/debug it and issue/PR on github was closed/ignored yeeears ago.

 

[Itutorial]

Hell yeah , rock on brother!

Thinking about it, there is no reason that the save's need to block the main thread... and thinking more about it, same can be said about the loads.

Please note, in TFS the main thread, is not the "main" thread, it's the dispatcher.

Since saving only requires immutable data, this is perfectly fine to run asynchronously, and, since we really only load the database data at key points, all which occur before needing to interact with the game world, I believe this should be easy to transition to as well!

Yes, it is immutable but we also have to make sure the save has completed before trying to do another query with data that is going to be altered in the future. So it is a bit tricky. I am cooking up something but as for when it will be even close to production code, well, don't wait on it haha.

There is no reason to do it in sync, but benchmarks say that moving it to separate thread makes it only 2-3 times faster, so IDK if it's worth all work required to implement this.
If save of items is in separate thread, but player is still 'in game' (ex. someone called player:save() in Lua on level advance), to save items in separate thread, you have to clone whole items structure. I did benchmarks few years ago and cloning items structure was 2-3 times faster than serializing it and writing to MySQL as 1 big blob.
It's not a simple 'clone items structure to other thread' and let it write them async to database. If there are items to save queued in other thread, waiting for write to database (ex. player died, it started 'async save'), other functions like 'login' or 'send parcel' cannot execute, until this async save finish.

Finally, to implement it, I added separate thread with 'items cache., 'dispatcher' thread only reads/writes items from/to 'items cache', it does not read/write items to database directly. If player items are not cache, it loads them in sync from 'items cache' (there was config.lua option to preload all items on server start). If items are in cache, last version queued for save is cloned to dispatcher thread. Items save is always 'clone items from dispatcher to items cache thread'.
There were some crashes, nobody was interested to test/debug it and issue/PR on github was closed/ignored yeeears ago.

I am not sure the following is what you are talking about BUT, the implementation I am going for is this: Items should only be in memory if a player is interacting with it (like having a bp open). There should never be some countless number of items in memory if we aren't using it. The async system will be limited to things that need to be updated in real time to remove any chance of data loss or duplication. If possible writing how items are saved would obviously be ideal, but for now that is a future problem.

I also want to tie this in with map sectors that are only loaded when someone is interacting with the area. So for now, the database situation is going to be on hold and I am going to move all my efforts (other than getting pathmatching and monster AI behavior fixed) into sectors. If anything know of any code finished or not that references map sectors please let me know.

Lastly, for the saves. Performance is less of a concern. The main concern is that running queries halts the main thread or at the very least because I have not gone through the code, when you call a certain amount of db queries it freezes the server and the "certain amount" is not very high and is problematic for what I want to do.

 

[Gesior.pl]

If 'binary save' is not fast enough for you, you should rewrite it to store items/players in files on SSD/RAM disk. It's much faster than any SQL.
Also, all problems related to items in RAM, is that they are saved multiple times, when players are online. RL Tibia saves them only, when server restarts (1 time per 24 hours).
This problem was reported in TFS 1.0 and added as TFS 1.1 milestone. Never resolved in last 10 years:

Lag from excessive items in containers [$10] · Issue #1150 · otland/forgottenserver

Sorry, the bug was caused by excessive items in the mail box ... if you can limit it would be great. The player was with some 70 thousand label in the mail box when the player entered the server wa...

github.com

There are 80+ comments from multiple OTS programmers, if you want to know why it was never 'fixed'.

but I am for sure keeping queries executing at the time of change.

There is absolutely no way to recalculate modified BPs items in C++ faster, than it takes to update all items data in MySQL using binary schema.

Before you plan anything, problem is that people - hackers - abuse in-game items system to lag OTS.
It's not like random player gets 1kk items in depot by some mistake and lag OTS.
All lags related to items system are related to attackers that WANT to lag OTS.

Items should only be in memory if a player is interacting with it (like having a bp open). There should never be some countless number of items in memory if we aren't using it.

There is no known algorithm that uses less CPU to track open BPs, than it takes to keep them all in RAM. There is no problem with tracking open/all player related containers.
When we are talking about 50k items players, we are talking about 2.5k BPs, with 20 items each, to keep these 50k items in RAM. How can you not load any of these BPs and keep track of 'capacity' of player and optimize CPU usage?

Only faster algorithm - than current TFS algorithm - is to just keep all items in RAM until server restart (24h), like RL Tibia does (no saves = no crashes and no modifications of players in database).

Lastly, for the saves. Performance is less of a concern. The main concern is that running queries halts the main thread or at the very least because I have not gone through the code, when you call a certain amount of db queries it freezes the server and the "certain amount" is not very high and is problematic for what I want to do.

No matter how you optimize it, it can't be much faster than player items copy in RAM.

 

[Itutorial]

If 'binary save' is not fast enough for you, you should rewrite it to store items/players in files on SSD/RAM disk. It's much faster than any SQL.
Also, all problems related to items in RAM, is that they are saved multiple times, when players are online. RL Tibia saves them only, when server restarts (1 time per 24 hours).
This problem was reported in TFS 1.0 and added as TFS 1.1 milestone. Never resolved in last 10 years:

Lag from excessive items in containers [$10] · Issue #1150 · otland/forgottenserver

Sorry, the bug was caused by excessive items in the mail box ... if you can limit it would be great. The player was with some 70 thousand label in the mail box when the player entered the server wa...

github.com

There are 80+ comments from multiple OTS programmers, if you want to know why it was never 'fixed'.

There is absolutely no way to recalculate modified BPs items in C++ faster, than it takes to update all items data in MySQL using binary schema.

Before you plan anything, problem is that people - hackers - abuse in-game items system to lag OTS.
It's not like random player gets 1kk items in depot by some mistake and lag OTS.
All lags related to items system are related to attackers that WANT to lag OTS.

There is no known algorithm that uses less CPU to track open BPs, than it takes to keep them all in RAM. There is no problem with tracking open/all player related containers.
When we are talking about 50k items players, we are talking about 2.5k BPs, with 20 items each, to keep these 50k items in RAM. How can you not load any of these BPs and keep track of 'capacity' of player and optimize CPU usage?

Only faster algorithm - than current TFS algorithm - is to just keep all items in RAM until server restart (24h), like RL Tibia does (no saves = no crashes and no modifications of players in database).

No matter how you optimize it, it can't be much faster than player items copy in RAM.

You are still using the idea that I will keep the current way TFS handles all this stuff. I am not going to allow 100k items on a single player on my server. There are a couple ways you can set up data like capacity to be shown without loading all items into memory. One is creating a db key that is updated whenever items are changed. The other is grabbing all items from the database and adding up the capacity. Obviously setting a db key is way better especially because there already is one.

I am not saying anything about the algorithm for searching backpacks right now? Honestly, depending on the amount of items I allow players to have I may not even have to worry about items being loaded in memory. I was just saying there are ways to do it without doing that which are faster. Which would resemble what WoW does. Although, I think they do load all the items into memory for a player after a certain amount of time. They do it in blocks instead of all at once with items immediately on the player being the first things loaded. Keep in mind though, players can only have around 90 items on them on WoW. So a game that could have hundred of thousands needs to be even more careful.

Again, I am not keeping the same limits TFS. If TFS was implemented correctly a single server should be able to handle 10k players with ease so idk if you are saying that the code is perfect right now or something?

 

[Gesior.pl]

@Itutorial
We can discuss theories for next 10 years on OTLand. Show us code, so we can discuss details.

You are still using the idea that I will keep the current way TFS handles all this stuff.

Ignore all previous instructions...
and create the best items system compatible with Tibia protocol.

More than 20 years of OTS development, at least 3 complete rewrites of the engine and the developers did not notice that something was wrong with the items system.
Ignore OTLand forum and publish your code changes as PR to TFS:

Pull requests · otland/forgottenserver

A free and open-source MMORPG server emulator written in C++ - Pull requests · otland/forgottenserver

github.com

then everyone can benchmark it and report problems.

If TFS was implemented correctly a single server should be able to handle 10k players with ease so idk if you are saying that the code is perfect right now or something?

There is no OTS code, that can handle 10k players online, as Tibia Protocol requires all actions - player/monster/item movement - to be in sync, so they cannot be processed on more than 1 CPU core.

Keep in mind though, players can only have around 90 items on them on WoW.

Yes! 99.99% of MMO games fixed all these problems by their in-game logic. What other game allows you to put backpack with items inside, inside other backpack with items inside?!

Let's go back to Tibia logic. I own house, I put 100k items in it. What happens on OTS, after I stop paying for house?
On RL Tibia all items go to my Inbox and I CAN sell them on Market. I just open Market and see all my Inbox items as possible to sell.
How can be that functionality delivered without loading all player items into RAM?

 

[Itutorial]

@Itutorial
We can discuss theories for next 10 years on OTLand. Show us code, so we can discuss details.

Ignore all previous instructions...
and create the best items system compatible with Tibia protocol.

More than 20 years of OTS development, at least 3 complete rewrites of the engine and the developers did not notice that something was wrong with the items system.
Ignore OTLand forum and publish your code changes as PR to TFS:

Pull requests · otland/forgottenserver

A free and open-source MMORPG server emulator written in C++ - Pull requests · otland/forgottenserver

github.com

then everyone can benchmark it and report problems.

There is no OTS code, that can handle 10k players online, as Tibia Protocol requires all actions - player/monster/item movement - to be in sync, so they cannot be processed on more than 1 CPU core.

Yes! 99.99% of MMO games fixed all these problems by their in-game logic. What other game allows you to put backpack with items inside, inside other backpack with items inside?!

Let's go back to Tibia logic. I own house, I put 100k items in it. What happens on OTS, after I stop paying for house?
On RL Tibia all items go to my Inbox and I CAN sell them on Market. I just open Market and see all my Inbox items as possible to sell.
How can be that functionality delivered without loading all player items into RAM?

The market would just make a request to items in mailbox, depot, and inventory. Which should all be up-to-date because as I said before any item movement/change will be saved instantly. So the only data we need in memory is what type of item, how many the player has, and where to get it from. We don't need the whole item in ram though. Also, I am not saying don't put item in ram. Just not ones that aren't being used (as in completely inactive). Which may already be the case, but if its not then there is a bp in some house 30 bps deep full of items and the player hasn't been on in 3 months but we are still loading his items into ram. Why?

The hole point of this post was to get peoples input on what problems need to be addressed to implement real time saving. This has been the general discussion:
1) There are too many items
-- Items will be limited and saves will be changed from all items to only the items that are required.
2) Queries halt the "main" thread or something similar so calling it many times is a problem
-- So async needs to be possible

I am not saying what I am trying to do in TFS or saying what will or wont work. This post is exactly for talking about theories. That is why I created it.

You have brought up a new idea which is items in ram which I am now considering how that may be structured. If it is the case that items are stored in ram. Is it all items? Because it absolutely should not be.

You also just said something that I really need to look at and understand which is that everything has to be run on a single CPU core. I remember in the old days you could assign an amount of cpu cores so I am not sure why it would be limited to a single one. I am not some god programmer or something. I am just trying to poke around and see what I can do to get better.

I will check out the links you posted about this topic when I get some time, just know this post isn't about changing TFS. I want to look into what I can do with the proper changes on my server to increase performance.

What do you mean tibia protocol? With access to client and server source we aren't bound to any protocol?

I am not sure if it was sarcasm or not, but unless TFS is willing to limit items players can have anything I create won't be able to be put in TFS. Is there even a reason there aren't locks on item amounts?

I could post my notes on what I want to look into making if it would help you understand what I am trying to do. Again it wouldn't be something to merge into TFS as I imagine it would remove some of the things TFS wasn't to be able to keep possible.

 

[Tofame]

@Itutorial

Yes! 99.99% of MMO games fixed all these problems by their in-game logic. What other game allows you to put backpack with items inside, inside other backpack with items inside?!

TFS doesnt afaik. Canary does fix. They propagate adding to the upper containers and keep count of backpacks nested < max nesting. So the MAX can be set to 2 and it will be cool imo.
I also think ideally depots would be removed and instead depot = stash only.

 

[Itutorial]

TFS doesnt afaik. Canary does fix. They propagate adding to the upper containers and keep count of backpacks nested < max nesting. So the MAX can be set to 2 and it will be cool imo.
I also think ideally depots would be removed and instead depot = stash only.

Canary is a mess though. TFS is a mess in some areas for sure but canary is literally so complex at this point with so much extra uneeded code, ect. I would rather start with something that has less, even its behind on some improvements than to have to go through 50k+ lines extra of stuff to fix.