• There is NO official Otland's Discord server and NO official Otland's server list. The Otland's Staff does not manage any Discord server or server list. Moderators or administrator of any Discord server or server lists have NO connection to the Otland's Staff. Do not get scammed!

Theorizing a Modern Engine Design

jo3bingham

Excellent OT User
Joined
Mar 3, 2008
Messages
1,103
Solutions
14
Reaction score
772
GitHub
jo3bingham
I was chatting with a buddy of mine, and we got on the subject of TFS and its strengths and weaknesses as it currently stands. As we all know, TFS is the engine to use when creating an Open-Tibia server. Sure, it itself is a fork of OTServ, but it lasted the test of time and I would be surprised if any server running today isn't using the TFS engine at its core (no matter how modified it is). And while TFS is battle-tested and, more-or-less, bug-free, it's still based on the original architecture of at least 10-year-old code. There's nothing wrong with old code ("If it ain't broke, don't fix it."), but there's also nothing wrong with looking at what's currently available and asking yourself how could it be better. I'm not knocking TFS (like I said, the engine is solid and his been around for quite some time), but a lot of it is still written using old coding practices (e.g., manually reference-counted pointers) and relying on custom, binary formats (OTB and OTBM), and the downside to this is that it can make it difficult to make major improvements without a lot of effort; to the point no one wants to actually make them. I know there's been a few attempts at different engines (PyOT and like three different C# engines come to mind), but I'm not aware of any public server that uses them.

To get to the point, we came up with some ideas for a more modern engine, and were curious what the community could come up with as well.

Ideas
  • Database: SQL is great, but what benefits could come from a NoSQL database? MySQL will be faster, but with how powerful modern CPUs are, you could use a NoSQL database and not notice any performance issues. For example, there's no need to stall the whole game thread when you can load players from the database asynchronously. If you use something like MongoDB, you get the benefit of more-easily readable data (BSON/JSON).
  • Items: Currently, items live in two separate files; items.xml and items.otb. I know the original intention for this, but, really, one file is enough, and while you're at it why not use a more modern format like JSON instead of XML or a custom binary format.
  • Map: OTBM is basically just XML in a binary format (nodes/attributes/etc.). The single-file format means maps, especially RL maps, are huge in size, and more than one mapper can't realistically work on the map at the same time. Another problem is there is no source-control; if you hire a mapper you have to trust that they didn't hide any secrets in the map. Sure, you can open it in RME and search for items by ID, but that's tedious. A better approach would be to split the map into sector files (256x256 tiles is what CipSoft uses) and in a readable format (again, JSON would be a great option). You could have a single file that holds all of the map files in it; this would allow you to group the map files into sub-directories (e.g., all of the map files for Rookgaard could be in their own folder) and easily say which parts of the whole map you want to load. Then you have the added benefit of using source-control (i.e., GitHub) to track changes to your map at a glance.
  • Language: C++ is great (it's in my top 3 languages that I use), but it can be quite difficult for beginners to grasp certain concepts and get their development environment setup properly. If I were building a more modern engine, I'd consider using a language such as Rust. Yes, Rust has a steep learning curve, but it can provide the same performance as C++ (sometimes better, sometime worse) and it's much easier/quicker to get setup to start working with a project. Plus you have the added benefit of it just being harder to write unsafe code, and you can more easily make your code modular (there would be no reason to have a single file with 16k+ lines of code...looking at you luascript.cpp).
This pretty much covers the ideas we threw around. What would you do if you were to build a new engine from scratch? Like I said before, I know other engines exist/existed, but they all held on to old ways of thinking. Drop OTBM for something better and just create a tool that will allow users to convert between it and your format. Nothing may come of this, but discussion is fun, and, you never know, someone could take the community's input and start the next era of Open-Tibia (like the OpenTibia-Unity client).
 
I'm all in for "reworking" all this old stuff, but we lack people that would help with that, but tbh I would keep c++ as the main language for the new engine. All these binary formats are deprecated especially otb. Maybe if someone or we could provide any funding for that, then we would get motivation of experienced devs, but right now these are all only discussions and this saddens me :(
 
I'm all in for "reworking" all this old stuff, but we lack people that would help with that, but tbh I would keep c++ as the main language for the new engine. All these binary formats are deprecated especially otb. Maybe if someone or we could provide any funding for that, then we would get motivation of experienced devs, but right now these are all only discussions and this saddens me :(
I agree with your points, but this is a theoretical discussion about what a new engine built from the ground up in a collaborative setting would look like. I don’t want this to turn into a discussion about how TFS doesn’t have enough contributors or how TFS could be better. Not to be rude, I just want this discussion to stay on topic.

This is a discussion that interests me, even if nothing comes out of it.
 
I agree with your points, but this is a theoretical discussion about what a new engine built from the ground up in a collaborative setting would look like. I don’t want this to turn into a discussion about how TFS doesn’t have enough contributors or how TFS could be better. Not to be rude, I just want this discussion to stay on topic.

This is a discussion that interests me, even if nothing comes out of it.
Wasn't talking about getting people to support tfs, but about getting people to make these brilliant ideas into reality.
 
Wasn't talking about getting people to support tfs, but about getting people to make these brilliant ideas into reality.
That’s fair. But ignoring that, I would be interested in hearing any ideas you may have. For example, I left out scripting because I think Lua is great, and probably the best option for this use-case.
 
Last edited:
A lot of people like python, I dont personally, but that could also be considered when it comes to scripting. Talking about all these json formats, wouldnt that be slower than raw binary format when loading? For example maps would be dynamically loaded if they were sectored.
 
A lot of people like python, I dont personally, but that could also be considered when it comes to scripting. Talking about all these json formats, wouldnt that be slower than raw binary format when loading? For example maps would be dynamically loaded if they were sectored.
Python is slower than Lua. JSON is slow when handling huge amount of data, especially if you don't save it in one line but formatted for better readability. Even then, not really recommended. BSON? Better/faster than JSON for sure. Frankly, we are talking about mere seconds on server startup. Map loading is what takes the most, I wouldn't care if items are loaded in 2 or 5 seconds.
 
@oen432 well if we would think about some type of "reloading" technique (e.g tfs /reload) then our loading items would start to matter. Didn't even know something like BSON exists. I will do some research in that matter.
 
I never found loading times a good argument. Who cares if server startup takes longer? Sure, if the times were substantial I could understand, but the difference between a binary format and something like JSON won’t be that much, especially with modern processing power. You could argue that longer startup times hinders development, but then that brings up the discussion of dynamic map loading.

Reloading is the same; are people actually reloading that often during production that the time to reload something like items is that important? Then that opens the discussion of why reload every item if you only modified one? Why not use arguments to specify which items to reload? /reload items all (reload all items), /reload items 3031 (only reload item with ID 3031), /reload items 3031 3032 ..., etc.
 
Why not use arguments to specify which items to reload? /reload items all (reload all items), /reload items 3031 (only reload item with ID 3031), /reload items 3031 3032 ..., etc.
Damn it, why is this not a thing yet? And that could work not only with items but monsters, NPCs, maybe even more, like actions or such, we would need to specify some identification for each event (name, id etc.). Maybe new command, /load, that will load NEW events/items from .xml by checking if given script/item id/event name is already in memory.
 
Engine thoughts:
When Roslyn was released a few years ago it had me thinking it would be cool to see a c# engine with c# scripts. Roslyn has scripting APIs that allow it to run C# script files. I have made use of the scripting APIs at work to make a program which allowed scripts to be run at anytime dynamically. This allowed the hardware guys to right small scripts to control the hardware the program was written against. I was able to write some more advanced scripts for them with multiple threads and events.

Map thoughts:
stian created a (.sec) sector map format for use with PyOT. I believe it was 256x256 chucks of map. One cool feature was the use of a 4th dimension in the map format. The idea was to be able to control which instance was loaded via scripts or even add new instances. This would allow a town to be mapped with different weather conditions. On a time or event bases the map chunk could dynamically be changed in the server. The use of the 4th dimension allowed multiple instances of the same map to load in the same location. I think the idea was to be something like WOW raids. (never played WOW myself to know for sure). Where each group of players is in the cave in their own instance. I'm not sold about json format for the map but I do like the improve version control abilities but really who is going to be able to diff that in text format besides minor edits. A visual diff tool would be required and could be added to a map editor.

Item thoughts:
PyOT put all items in the database. I can't remember much else about items for pyOT. I don't know have an idea for a better item format but a single file would be preferred. Ideally a file that is human readable without another program. I find keeping items.xml up to date and remembering which items were customized a real pain. A good way to solve that would be two "item files" of the same format. One with the cipbia item definitions and another one for custom items which overrides the defaults.
 
Last edited:
Great topic. I wish I were not on a phone or I’d share much more...

In my opinion, it would be beneficial - to OpenTibia as a whole - if the software were user friendly for Windows users as well. I know, Windows sucks... But it’s used by the majority and I don’t believe it should be a requirement for newcomers to know how to use Linux for them to be involved in OpenTibia and development. A lot of people I know are interested in this game but have no idea how to use a Linux OS, and quite frankly are not patient enough to learn how to do so. That’s how the new generation is. OpenTibia will be around for years and years, sure. How popular it will become is determined by how easily it is to enter. Minimal barriers to entry will be more inviting.
 
I'd love to see a C# .NET Core server. Yeah, its higher level and microsoft..
But yet, cross platform is easy now with Core.
 
if the software were user friendly for Windows users as well. I know, Windows sucks...
This has nothing to do with the software being unfriendly to Windows and everything to do with Windows being unfriendly to software. The developer toolchain is a 2nd class citizen on Windows. On Linux I can find an interesting syscall like this, find an example of it, and turn that example into a compiled binary in the PATH and usable by all users on the system is less than 2 minutes. Try doing that on Windows.


Oh god, I can't believe you seriously suggested MongoDB... please don't ever do that again. Mongo is never ok.

The rest of it... I like to ponder these things too. And do so often. But to be honest, just with the coders in this thread alone we could brew up new RFC-style specifications/standards for the whole stack, start a new server codebase from scratch, and get it up to passable usefulness in just under 4 months.

... and then we'd still be right where we are at now with the rest of the stack:

  • Look at the state of "open source spritepack". CipSoft may have no moral highground as they basically stole the entire skeleton of their game from Ultima 6, but this situation is not tenable forever.
  • Look at the state of the primary community client.
  • Look at the state of the map editor.
  • The object editor is built on dead tech, and a cursory glaces shows me the libs that underpin it are precompiled, which is a nuisance to porting.

But since some points were brought up that I have opinions on...

Java: Why? That just brings an air of souless corporate machinery. Switching to this gains you nothing. All the people who are excellent Java programmers, already code it all day at work, and do not want to look at more when they get home. The JVM is kicking ass, but the pure Java entry point is basically legacy and is floating on borrowed time. That's what Graal is all about. Now if you had suggested Scala and Akka I could understand but still, these are not things I would touch for "fun". C++, despite it's problems, isn't something that makes me sick to my stomach to look at after work.

ORM: This is how you royally tank your performance for almost nothing gained whatsoever. The translation between the type schema the engine uses internally and the persistence layer is already a solved problem. ORMs are invariably a pain in the ass layer that business uses because they have to and coders work on because they get paid to.

NoSQL: Like, it is rumored that every time you search "why MongoDB sucks" a computer in Google's datacenter catches fire, and that's why the results returned come as an integer and not infinite. The "No" in NoSQL is supposed to mean "Not only" but actually means '"No" good handling for typical SQL workloads.' in practice; which is why you will see the NoSQL engines that don't actually suck still paired with Postgres/MariaDB. However I can't really think of something OTs would need Elastisearch, Cassandra or Redis for, and that's the shortlist of ones that don't suck.

Rust: I like this idea a lot, if only because it mean using Javascript as gluescript is 100% feasible out of the box with this. Delicious heresy.

Item.xml + Item.otb being merged into a new format, and tooling the loader to allow overloading so projects can keep their custom items as a separated concern: This is precisely what I had already planned on doing myself. Aren't a significant amount of items.xml / item.otb the same across Tibia versions? It seems like if the item information packages were arranged like Master Set is some minimal version, like 8.0, and then overloaded by Set A is like 8.6, then overloaded by Set C and so on until you get to current Tibia version, it would be a lot easier to deal with. And if the client consumed the same or very similar file format? The community could claim some section of IDs of outside the range Cip is likely to get to soon and start collaborative item pack overload set. The server could also have tooling to make a "squashed" local copy so startup time isn't seriously hindered by doing the overloads on each startup.

The only thing I think the daemon is really missing is async event model and bindings for such in Lua. Some of it's there internally, but there have been many many times I thought to myself "God I miss EventEmitter" when doing something 20 LOC in Lua that would have been like 5 in Node. I've seen a few examples of this being pursued here on OTland, so I know it's in the pipeline but it's long overdue. I'm much more familiar with libuv than I am with Boost's ASIO, and if ASIO wasn't already what the other devs had pinned their hopes on I'd be working on integration of libuv right this second, because that would open up this for the gluescripters.

The Map Editor:

Anyway, as far as any map format ideas you guys have, I'm all ears, cuz replacing RME is a journey I've already dedicated myself to. Though, tbh, if I had to come up with my own format, I'd use 100% off the shelf: a tarball holding hjsons for spawns, houses, and metadata, and a zstd compressed protobuf for the map. A very programmer-centric format. Which is why I was probably just going to reuse OTBM and, as an optional thing, bump the version number add a "groupid" extension for items, since the group of users that just use multiple uniqueids in maps and ignore the errors that scrollby at daemon launch is rather large. So clearly uniqueid + actionid is not enough.

I really do like the idea of chunking up the map, and was already going to make my own tooling for it one way or another. I dunno if you guys remember, but before Menoxcide retired and transferred ownership of the FH dataset to me, he was betrayed by one of the other staff members because of lack of communications between the two of them, misunderstandings, and whatever else. Either way, that person leaked a copy of the dataset. However it had my work in it at that point, which I aggressively leveraged DMCA against everywhere. The cost in my time that little stunt sucked up I can never get back, and since then how to "componentize" the gameworld map is something I've thought about a lot, out of hard learned lessons on minimizing risk exposure to people protected from my wrath by oceans.

I think the real issue to be solved here is a workable scheme to abstract Position()s in the map chunks. Basically, the person working on the map chunk doesn't necessarily know where it will end up in the actual game world. Neither would the scripters making features for a particular map chunk. So I think a way for them to both collaborate and use waypoints, or some other similar abstraction layer, that basically become "Position symlinks" so that no matter where the map chunk gets deposited, scripts calling for positions in it still work.

Though I think duplicate unique ids in two different map chunks should 100% be a fatal error.

Once that's in place it's game on. Unfortunately, others have started to try to replace RME and it seems all gave up. I refuse that fate. Since Object Editor and Item OTB editor are some what in the same scope I may try to target them too. But I'm only one person and that means what really needs to be solved for the community now is the client, and I can't help there, because burn out is real, and I refuse that too.

Though, anyone who is working on such a thing, please please do yourself a huge favor and make it easy on me to help you when I am finally free to do so: imgui
 
I agree a lot with what you said jo3bingham. The bare server itself should be coded in Rust to ensure no crashes, everything else should be done in a well-known scripting language which is more modern like julialang or python, also formats should be either json, bson or just basicly anything else which is standardized. I once translated the otb format into Rust (https://crates.io/crates/ot - Shawak/otbmview (https://github.com/Shawak/otbmview)) but it's hell of a mess and all these custom formats prevent developers to build their own engine (ofcourse thats not the only reason).
 

I'm 100% in Nekiro's camp with No Python. The role of "gluescript" is mutually incompatible with languages featuring enforced whitespace. If I was gonna spend any amount of effort replacing Lua, I'd be helping the Chakra team, so I could use an ECMAScript JIT instead. (working with Google's V8 is unpleasant at best, so I really hope this is the direction Chakra takes now that it's been cast off since MS builds Edge atop Chromium. Seems like they are kinda planning on it.)

But if you guys are gonna target Rust, Rhai is just sitting there waiting for you.

Another thing to consider when trying to replace Lua with anything else is most of the pedestrian OT users that are gonna lay their hands on the gluescript are not real coders. And if you put in something that lacks (Ducky|Dynamic)-typing, they are gonna get frustrated. It's a trainwreck waiting to happen.
:trainwreck:
 
I'll leave here my 2 cents:

1.Items.xml/Items.otb
This should be merged in a new file format like JSON and used by server, client and map editor.

2. Get rid of .dat/.spr
With the above done we can get ride of .dat/.spr because we just need to store the sprites as png image files like Tibia does and use the same items file

3. Make a new tool or script to make easier to add new items/sprites
This one can be done really easy using any thing you want

4. Make a new tool or script to convert dat/spr to the new item file
 
I'll leave here my 2 cents:

1.Items.xml/Items.otb
This should be merged in a new file format like JSON and used by server, client and map editor.

2. Get rid of .dat/.spr
With the above done we can get ride of .dat/.spr because we just need to store the sprites as png image files like Tibia does and use the same items file

3. Make a new tool or script to make easier to add new items/sprites
This one can be done really easy using any thing you want

4. Make a new tool or script to convert dat/spr to the new item file
I don't think that dropping Cipsoft client support is going to happen.
 
Back
Top