TFS 1.X+ Optimizing player saves & when saves execute

[Itutorial]

Hello everyone,

I noticed that all player save information happens in one place. IOLoginData::savePlayer() I do not really like that in order to save player items I have to save everything else with it. So I fragmented the IOLoginData::savePlayer() method into the following methods.

IOLoginData::savePlayerInstantSpells();
IOLoginData::savePlayerInventoryItems();
IOLoginData::savePlayerDepotItems();
IOLoginData::savePlayerInboxItems();
IOLoginData::savePlayerStoreInboxItems();

I added the instant spell saves in Player::learnInstantSpell() and Player::unLearnInstantSpell()

Then I added the following into Game:layerMoveItem()

ReturnValue ret =
    internalMoveItem(fromCylinder, toCylinder, toIndex, item, count, nullptr, 0, player, nullptr, &fromPos, &toPos);
if (ret != RETURNVALUE_NOERROR) {
    player->sendCancelMessage(ret);
    return;
}

// Player moved item into or out of depot
if (dynamic_cast<const DepotChest*>(fromCylinder) || dynamic_cast<const DepotChest*>(toCylinder)) {
    player->saveDepotItems();
}

// Player moved item out of store inbox
if (dynamic_cast<const StoreInbox*>(fromCylinder) && !dynamic_cast<const StoreInbox*>(toCylinder)) {
    player->saveStoreInboxItems();
}

// Player moved item into or out of inbox.
if (dynamic_cast<const Inbox*>(fromCylinder) || dynamic_cast<const Inbox*>(toCylinder)) {
    player->saveInboxItems();
}

// Player moved an inventory item.
if (fromPos.x == inventoryXPosition || toPos.x == inventoryXPosition) {
    player->saveInventoryItems();
}

I also slowed down item moves to 100ms.

For the inbox items I also had to make it save when players send mail to another.

Right now everything works properly, anytime an item is moved in each scenario that it would need to be saved it is. I am not using store inbox atm but I assume it works just as well. This freed up a lot of "lag" with player saves

So my question is what might I be missing? If I am not missing anything, would this be something that we should see in the source code? Doing it this way allows the database to be completely up-to-date incase of a crash. I also added saving with house items and limited them to being moved every 2 seconds because it has to save all house items. Though that is not included in this.

 

[Gesior.pl]

Obviously the best solution is a server that never crashes but….

Yes. This is a real solution.

There is also other solution: save server on crash. You can process SIGSEGV signal (crash) and call function that saves all players online and houses. Of course there is a chance that item structure of some player/house is a reason of crash (or failure of connection to MySQL), in this case it won't save, because save itself will crash again. EDIT: You can do it in C++, you don't need any special 'anti rollback script' for gdb.

You cannot save items every time someone moves item. It was discussed and benchmarked a lot 10-20 years ago. As I remember, if was on list of planned TFS 1.0 features, but it was dropped.
When player moves single item, it's not a problem, but if player moves around BP with 1k items inside depot, it will call 1k INSERTs every move. Even, if you replace current save code with binary format or save items into Redis, it's still use too much CPU just to serialize items.
It was tested in most optimized scenario: 'save' did not write anything anywhere, just cloned items in RAM and passed them to second thread, that saved them. It still used too much CPU, when players moved BP around in house (as in your system, just added to HouseTile to save house, when item is moved).

Binary items format even with BLOB in MySQL is much faster than current relational version. Players load/save time is 3-40 times faster after changing items to binary format. You can make it even faster by saving items directly in files on SSD/RAM disk.
Binary items save for TFS 1.4:

• code: feat: Store player items in binary format (#1150) · gesior/forgottenserver-gesior@cd4c58b (https://github.com/gesior/forgottenserver-gesior/commit/cd4c58b2fd2177cd2d729c70dfe73bcd1294077c)
• documentation: doc: Player items binary format documentation (#4819) · gesior/forgottenserver-gesior@fee30e9 (https://github.com/gesior/forgottenserver-gesior/commit/fee30e94c73d2b4c16c0735b4f527f0ebb8239a5)
• compilation error fix: fix: Player items binary format compilation error · gesior/forgottenserver-gesior@c53f3e0 (https://github.com/gesior/forgottenserver-gesior/commit/c53f3e053f68f5b34691a51636fcb214e419a799)

 

[Itutorial]

Thanks for that gesior. As I suspected the reason it won’t work has to do with these weird things TFS allows. I wonder, how tibia handles it. There aren’t many games that allow 1 millions items to be in a single bag stack which is in 5 other bags all having a million. This is where I have to move away from TFS development for my own server. I won’t have to deal with this type of situation because bag space and capacity are going to limit it. My server will utilize a system closer to WoWs bag and bank system. Including bags not being able to be put inside each other other than your first bag (const_slot_backpack) which cannot be replaced and is never lost. Which is limited to 8 bags that can be added to it.

I will say it’s possible to limit the queries by utilizing a loop to get all the items that need to be updated to insert into a single query for execution but yes I see how it can get bogged down very easily with large item counts.

Thank you everyone that said something. I think I know what I should be doing from here. Unfortunately, TFS is stuck until someone better comes along (not me) lmao.

 

[Gesior.pl]

There aren’t many games that allow 1 millions items to be in a single bag stack which is in 5 other bags all having a million.

This is a thing you must fix.
That's what RL Tibia did long time ago:

• limit number of items in depot
• save players/houses only on server restart, if it crashes, all players/items go back to last server restart

and this is what many big OTSes do:

• limit number of items on house tile (including items inside bags)
• limit total number of items in container (bag/depot) - including sending parcels to full depot (except adding items to depot, when player leaves house)
• limit 'deepness' of containers (how many containers inside other containers server allow to add)

You can't add it with few simple IFs ex. 'on add item -> count items inside bag and block adding new item, if more than 5k'
In case of 1 million items, it would loop over all 1kk items in bag to count them every time players try to move item. It must be added like 'weight' of containers, every item move it propagates to all 'parent' containers change in weight. When you check weight of container, it does not calculate anything, weight is kept as container attribute:

forgottenserver/src/container.h at 1.4 · otland/forgottenserver

A free and open-source MMORPG server emulator written in C++ - otland/forgottenserver

github.com

Count of items inside and deepness of containers must propagate same way.

 

[Tofame]

Basically what Gesior and Nightwolf have said is the best solution. Your personal approach scared me to be honest, as I dislike a lot the idea of spamming sql queries.

 

[JeiDi]

If each item move would save items to the database...

I've just checked old logs (there were only 100 players online at the time), and there were 315k move actions that day, so you'd have 315k database save queries? Well...

Sharing that to put your idea into a real numbers perspective.

 

[Itutorial]

Basically what Gesior and Nightwolf have said is the best solution. Your personal approach scared me to be honest, as I dislike a lot the idea of spamming sql queries.

I am on the same page. However, when we say things like spamming we need to be very precise. Gesior pretty much outlined exactly what I am going to do, but I am for sure keeping queries executing at the time of change. Of course it will take a lot of work to make it optimized for high player counts.

If each item move would save items to the database...

I've just checked old logs (there were only 100 players online at the time), and there were 315k move actions that day, so you'd have 315k database save queries? Well...

Sharing that to put your idea into a real numbers perspective.

View attachment 93297

315k is that a lot of database queries? Let’s think about websites that require almost everything to be displayed from the database (like social media sites) there are billions of not trillions of database tasks happening per day. I know they have countless, very expensive servers yada yada. And have spent billions making their product as optimized as possible but we are not talking about that size. 315k queries sounds like a lot but if we put it in this format: let’s say our single delete and update query takes 1-2ms (maybe we can do even better) that would mean 6 seconds total out of 24 hours. Websites also have an inherit disadvantage because most are made with higher languages.

Again I am past the idea of this being put into TFS. You guys are probably all correct in most of what you are saying, that this would be very difficult to achieve if at all. I will find out I guess. I am also completely fine with having servers that only handle 200 players if need be. My main priority is data loss. Anytime I have run a server the biggest issue was always something to do with crashes and data loss. (Other than the old shitty monster AI where monsters couldn’t catch a player 10x slower than them which I have/am fixing)

Thank you @Nekiro @Gesior.pl @Codinablack @Ianoyo @ALS @Night Wolf @JeiDi @Tofame

 

[Nekiro]

Websites are usually asynchronous, your server isn't. Every query blocks main thread from executing in your scenario and directly impacts the performance of everything else that is happening in the server.

 

[Itutorial]

Websites are usually asynchronous, your server isn't. Every query blocks main thread from executing in your scenario and directly impacts the performance of everything else that is happening in the server.

Really? I feel like I was told it was async in TFS now. I guess I have another problem to solve.