• There is NO official Otland's Discord server and NO official Otland's server list. The Otland's Staff does not manage any Discord server or server list. Moderators or administrator of any Discord server or server lists have NO connection to the Otland's Staff. Do not get scammed!

Implementing Google Firebase into a server.... Is this a good idea?

nasu

Member
Joined
Jan 20, 2008
Messages
105
Reaction score
17
GitHub
KrwawyOrk
... I wrote quite an extended text and inadvertently somehow deleted it, drama, I will write in short:

This is a topic more for people who are interested in programming OpenTibia

Google has been providing some tool for a long time that can be used as a database for our game/server files or whatever. It's called Firebase. I use it for my simple web projects and don't need to keep databases on disk. It is a cloud-hosted database. The tool works for applications written in C++, C# (e.g. for games in Unity) and of course JavaScipt. The question is how much time will it take to rewrite the server code to implement firebase into OTserv? Actually would it make sense to do so? I know that for years OpenTibia uses mysql. Firebase bases the data structure in JSON.

I invite you to take a look:
 
Firebase is not for server side of game (OTServer). It can be used to update game client data: text, sprites etc.
It's main feature for smartphone app developers is it's automatic synchronization and build-in 'offline version'.
It is a cloud-hosted database. - cloud-hosted means $$$, a lot of $$$ :(

Cloud-hosted -> database is not on your server -> there is network delay between OTS and database.
Every milisecond delay in OTS-MySQL communiation gives milisecond lag for all players in game - that's how OTS works.
To fix that, you would need to rewrite big part of OTS engine. If you do so, you don't need to switch database, you can store data on anything in any format.

Real world scenario for Firebase:
  • you make smartphone app with information about your company products
  • your employees use it to search for products/prices
  • when they are at client location, they may not have access to internet.. app still works, as it keeps database copy on their phone
  • when employee get internet access again, it will download all changes (new products, updated prices) from company Firebase database automatically

Firebase site:
1652606910842.png
Realtime - MySQL is already realtime
Offline - useless for OTS
Accessible from Client Devices - useless for OTS
 
Last edited:
maybe firebase will be a good choice for simple turn-based games - RPG in the browser... I actually have some cool ideas

thanks for the information :p
 
Every milisecond delay in OTS-MySQL communiation gives milisecond lag for all players in game - that's how OTS works.

Wait, there's no async or off-main-thread database querying?
Are you telling me if MySQL randomly decides to take 10 seconds to complete some query, the entire server will freeze for those 10 seconds?
 
Wait, there's no async or off-main-thread database querying?
Are you telling me if MySQL randomly decides to take 10 seconds to complete some query, the entire server will freeze for those 10 seconds?
are you new GIF


Do you know how threads work in c++? That's not as simple as you would write await in javascript, c++ has no asynchronous interface, you gotta use mutexes (or callbacks) and shit to make simple query asynchronous.
Fire and forget is not hard, but awaiting result actually is.
 
are you new GIF


Do you know how threads work in c++? That's not as simple as you would write await in javascript, c++ has no asynchronous interface, you gotta use mutexes (or callbacks) and shit to make simple query asynchronous.
Fire and forget is not hard, but awaiting result actually is.
Thats why cipsoft back in 2006 at least used plain text files and not only 1 database, like TFS does.
 
are you new GIF


Do you know how threads work in c++? That's not as simple as you would write await in javascript, c++ has no asynchronous interface, you gotta use mutexes (or callbacks) and shit to make simple query asynchronous.
Fire and forget is not hard, but awaiting result actually is.

Not gonna pretend to know C++ threading, but in other languages it's pretty easy to spin up threads.
Might be worth looking into if there's a library somewhere that allows for C++ async behavior in a comprehensive way that's not inundated with mutexes, or at least not on the surface level.

Thats why cipsoft back in 2006 at least used plain text files and not only 1 database, like TFS does.

I was gonna say that would be hell on disk I/O, but then I remembered most computers nowadays use NVME, or at worst a SATA SSD.
Both of whom have pretty god tier random access, especially when the files are less than your average disk page.

Maybe it'd be worth putting XML account and character storage back in TFS. 😮
 
Maybe it'd be worth putting XML account and character storage back in TFS. 😮
Would you rather "SELECT player_id from player_storages where key = 1111" OR search through text files?
 
Wait, there's no async or off-main-thread database querying?
I made it 2 years ago. Cache players in RAM and save them in database in second thread. There was one bug, depot does not work after relog, some magic child-parent depot structure not initialized properly.
Nobody cared about it, nobody fixed it and it's still not in TFS.
 
Would you rather "SELECT player_id from player_storages where key = 1111" OR search through text files?

I would rather cache all players as data structures in ram on startup, that way you avoid disk I/O at the cost of maybe 10MB more ram usage.
And then you can just CharacterDB.GetCharacter(666).GetStorageValue(1111); rather than wait for MySQL to wake up.

And even if you didn't, I'm sure internally loading a kilobyte-size file off disk, parsing it into a data structure, and returning a storage value from the data structure would be exponentially faster than having to wait for a cross-process request.

I made it 2 years ago. Cache players in RAM and save them in database in second thread. There was one bug, depot does not work after relog, some magic child-parent depot structure not initialized properly.
Nobody cared about it, nobody fixed it and it's still not in TFS.

That's sad to hear. Sounds like it could have solved a lot of the stutter problems related to DB access. =/
 
Wait, there's no async or off-main-thread database querying?
Are you telling me if MySQL randomly decides to take 10 seconds to complete some query, the entire server will freeze for those 10 seconds?
Yes 😂 pack 100k items in your depot -> lag the entire server everytime you relog.

Thats why cipsoft back in 2006 at least used plain text files and not only 1 database, like TFS does.
Not the main point here though. Cipsoft made proper use of pthreads, so that queries to the database didn't interrupt the game loop either.
It was sometimes noticable upon exceeding the limit of unjustified kills that it could take seconds before the banishment was inserted (and game server notified), allowing you to take another kill meanwhile. That's because they kept their database elsewhere, one for all game worlds. So the delay was visible but unaffecting game dynamics. In OT you are forced to keep your database on the same machine and rely on its quickness, cause any delay, any bigger query, will stop the entire world.
Saving game server-specific data to files was another thing.

Answering the topic: no cloud-hosted or any other remote db will work for tfs.
 
Last edited:
Can't find it now, but there was a thread with some benchmark showing that tfs could lag for seconds upon each relog even with considerably low number of items (like 50k?). Ofc may depend on the machine and db, but those were the values +-. So yea, the shit is real. Executing all queries within the game loop is probably the biggest issue of tfs architecture and seemingly there's no will to take care of it as Gesior said.
 
Last edited:
Can't find it now, but there was a thread with some benchmark showing that tfs could lag for seconds upon each relog even with considerably low number of items (like 50k?). Ofc may depend on the machine and db, but those were the values +-. So yea, the shit is real. Executing all queries within the game loop is probably the biggest issue of tfs architecture and seemingly there's no will to take care of it as Gesior said.

And this is something every OT server out there is susceptible to?
 
Out of those open source distros that I know of, yes, cause they are all base on the same foundation.

So all you need is like 10 bots logging in and out of the game with characters that has a bunch of items in the depot and the server is essentially permanently DDOSed to death from the inside with no way to troubleshoot the cause. Yikes.
 
So all you need is like 10 bots logging in and out of the game with characters that has a bunch of items in the depot and the server is essentially permanently DDOSed to death from the inside with no way to troubleshoot the cause. Yikes.
No need for 10 bots, just one with a big stock of items. Found the thread: Server lags when login/logout with a lot of items - time to fix it! (https://otland.net/threads/server-lags-when-login-logout-with-a-lot-of-items-time-to-fix-it.266112/#post-2572247)
As you can see, 168k items caused the server to lag for 4-5 seconds. There was an attempt to optimize it by caching and using binary format, though prolly not finished and not implemented into tfs. And still wouldn't get rid of the core issue, cause even with 100 ms delay you can make the game unplayable by constantly logging in and out.
 
No need for 10 bots, just one with a big stock of items. Found the thread: Server lags when login/logout with a lot of items - time to fix it! (https://otland.net/threads/server-lags-when-login-logout-with-a-lot-of-items-time-to-fix-it.266112/#post-2572247)
As you can see, 168k items caused the server to lag for 4-5 seconds. There was an attempt to optimize it by caching and using binary format, though prolly not finished and not implemented into tfs. And still wouldn't get rid of the core issue, cause even with 100 ms delay you can make the game unplayable by constantly logging in and out.

Ouch. Yeah looks like bandaid solutions aren't going to work to fix things, you'd have to deal with the core issue of "main thread halts while waiting for MySQL to reply". You'd think 168k items is a lot but all you'd need is an account with some money, and just make a macro or bot place each gold piece in the depot as a separate item. Or at worst just run around collecting some blueberries and trash off the floor.
 
I would rather cache all players as data structures in ram on startup, that way you avoid disk I/O at the cost of maybe 10MB more ram usage.
And then you can just CharacterDB.GetCharacter(666).GetStorageValue(1111); rather than wait for MySQL to wake up.

And even if you didn't, I'm sure internally loading a kilobyte-size file off disk, parsing it into a data structure, and returning a storage value from the data structure would be exponentially faster than having to wait for a cross-process request.



That's sad to hear. Sounds like it could have solved a lot of the stutter problems related to DB access. =/
10mb? Interesting

Accessing ram is way faster, thats a fact, but I think it would take way more memory for big servers, unless you did some smart caching and avoid caching people that didn't log in for x days, but then you still gotta query something like database
 
10mb? Interesting

Accessing ram is way faster, thats a fact, but I think it would take way more memory for big servers, unless you did some smart caching and avoid caching people that didn't log in for x days, but then you still gotta query something like database

Yeah depends on the player count really, and I guess how many items they have in their inventory and depot. But the footprint of each player structure would just be kilobytes in size, so even something huge like 10,000 players each being 100kb worth of data, that'd still be just one gig of extra memory.

But you could definitely do something like only preload players who have logged in in the last 7 days or have gotten a certain level, if you're really memory constrained. And the only real overhead from that would be that you'd have to load and memcache a player on login, which can be done in a separate thread while the player waits to enter game.
 
Back
Top