• There is NO official Otland's Discord server and NO official Otland's server list. The Otland's Staff does not manage any Discord server or server list. Moderators or administrator of any Discord server or server lists have NO connection to the Otland's Staff. Do not get scammed!

Theorizing a Modern Engine Design

Instances would be cool, but the official Tibia client would need to be dropped in favor of an open-source client that could handle instances properly.
From what I remember pyot had support for instances without a custom client. But in a limited way that the sec map format supported instances. Then the game server decided which instance to serve and all the clients received that single instance. It was a nice way to allow changing environments and did so in a way that made it easier for someone to map. The server knew when to load and unload sectors and when to change instances. So that was a goal of the system if the current source code says otherwise.

Unfortunately, I think you're right. And I think the reason for this is that the official client is recognizable; users are used to the look/layout and know how to use it. They get the same experience, more-or-less, between Open-Tibia servers and official servers when it comes to the client. Plus, for server developers, it's one less thing they have to maintain.
+1 less work and less software for a OT creator and player. Custom clients are amazing technology wise. The fact that one or few developer can make such functional clients is amazing. However I don't want to download a damn custom client for every OT I log into. It was great when I had one client installed and an IP changer. It was possible to switch between cipbia and OTs easily. Most OTs at the time were focused on keeping up with the latest client version. Now the community is very splintered between all different client versions that each player enjoys the most. I like tibia and the custom takes people have on tibia not some total different mmorpg that might as well be zezenia. TFS refusing to support newer protocols is a real pain and makes the splintering worse.

There are several client 11 features I would like to see in more OTs. Features such as the sprites, cyclopedia (map, bestiary, charms), ingame shop, and market would be beneficial. Features of lesser importance might be prey system, news, friends, inspect, and house auction system. I could imagine playing deathzot and be able to see the custom monsters and items in the cyclopedia.


On topic about a flexible server engine for Tibia and not reinventing a client, map editor, or new game. This thread confirms that developers have preferred languages. Devs wants to use the language they like most or are interested in learning. Every language mentioned so far is capable of getting the job done no doubt. What would be your goal with a modern engine? You mention it could be tough for the community to change scripting languages. So you must have some thoughts in mind about what this modern engine should accomplish. The goal will affect the decision of technology chosen besides just the language used to code the server.
  • Just looking for a new code base which is easier to maintain?
  • Must it support existing lua scripts?
  • More players online at once? Not many servers surpass 1K players as it is. More performance is fun but for what gain.
  • Less system resources required to run the engine?
  • Lower bar of entry to run a server for greenhorn devs?
  • Drop all usage of XML because "John Doe" finds it annoying to work with?
  • Single engine to support multiple client versions at the same time?
  • A engine which can make better use of containers?
  • Does ease of migration from the old engine to the new engine matter?
  • Quickest to a usable engine?
It is interesting to read all the unique takes on a modern server. I'm not very surprised by so many different ideas in this community.

For me as a web developer tfs lack restful api. Imagine changing game state from website(e.g. voting for today's event). JWT for authorization should be just fine. With it community could create some new, fun stuff and admins could manage their server from admin panel. Just a concept I've been thinking on.
Another thing I was wondering, apart from tfs itself, is creating strictly a back-end service(without any template generation) for tfs, so there is a possibility for community to use any framework to implement their front-end solutions, but this is different topic.
See Asynchronous HTTP API with lua bindings by djarek · Pull Request #2010 · otland/forgottenserver (https://github.com/otland/forgottenserver/pull/2010) it might interest you.
 
Last edited:
My wishlist:

1. Rust internals
2. Lua scripting
3. Pure protobuf protocol (gRPC or similar) based of Tibia's latest protocol.
4. Entity Component System?
5. Map sectors

Reasons:

1. Rust is fast, parallel friendly, type & memory safe.
2. Lua(jit) is fast, small, memory friendly, and people know it here, compatibility libraries would be possible.
TypeScriptToLua (TypeScriptToLua (https://typescripttolua.github.io/)) or Teal (teal-language/tl (https://github.com/teal-language/tl)) could be used for a more strict language.
3. Would unify the protocols and both OTClient & UnityClient could support it.
4. ECS might fit perfectly for Tibia, possibly allowing horizontal scaling (multiple shards, one world)
5. Not loading the entire map into memory

A Redis database could be cool. You could interop the Lua scripting and pub/sub with the game engine.
 
Last edited:
Database: SQL is great, but what benefits could come from a NoSQL database? MySQL will be faster, but with how powerful modern CPUs are, you could use a NoSQL database and not notice any performance issues. For example, there's no need to stall the whole game thread when you can load players from the database asynchronously. If you use something like MongoDB, you get the benefit of more-easily readable data (BSON/JSON).

Database:
I worked a lot with NoSQL, and here's my feedback on them
Note: We have to think what do we want to accomplish before looking at different types of NoSQL, what do we want to be able to provide.

MongoDB is not great; it doesn't really bring anything good to the table over MySQL (except little bit better execution times for more results). MongoDB has also plenty of issues starting with its own design - replication or sharding, saying "Can't go wrongo with Mongo" is BS in real world of use when you are meaning to use its feature then its coming apart. (Not great at anything orientation, except marketting)

ElasticSearch is ok but does not fix the use as your typical database, it could be used as a map storage though if we ever want to have enormous size of map spanning couple GB's or TB's of data. ES is great for indices, and search feature - not something we can utilize correctly under small load of ots.
(Read oriented DB) - ok if we were to create some machine intelligence non-player controlled characters running and playing the game.

Cassandra its great if we ever wanted to have multiple world or one large world system, with game servers in different locations but needing to share same database/s quickly enough so that you can have same data in US, London and lets say Australia while being able to write to them simultaneously. Cassandra does that, but it doesn't handle indices well (newer versions actually completely removed them). Con - Hardware, needs a lot of separate nodes to work correctly.
(Its write oriented DB.)

Hbase on Hadoop
Great for reading millions of records at once. Great if you wanted to create some machine intelligence non-player controlled characters running and playing the game, then we would train the code with spark.
(read oriented)

Map:
Could run on something like ES or Cassandra allowing people to modify the map at the same time as its being played - this way if you dropped a backpack and game engine restarted everything is still there; and map editor could move things as game is being played; and multiple people can edit the map at the same time.


~ keep in mind that typical users of OTServers sometimes would even prefer to use sqlite db.
 
What about wider client version support?

We could enable/disable features based on client version we're running like otclient does and sort monsters by version like rme does with extensions.

In fact that would require plenty of extra lines of code, but having support for versions like 7.4, 7.6, 7.7, 7.92, 8.0, 8.1, 8.6, 10.98 would solve the problem with engine porting. This would also pave way for version 12+ being special case.

Most of the code lies in packets structure (handling the right version), items (drawing missing sprites as void/X token), looktypes (drawing incorrect as stealth looktype instead of sending one that is out of scope) and npcsystem (buy/sell count name instead of window). For rest of missing features, we could write some sick workarounds like trade instead of npc window (or classic buy/sell count name), green message split to chunks instead of FYI window, message -> talkaction instead of modal.

Solution for items/looktypes/sprites/connection would be creating a list of existing client ids and RSA keys for each version (probably xml).
 
Last edited:
I would suggest server-side javascript (nodejs) with typescript on top of it for good developer experience. It's a nice language, easy to learn, has a huge ecosystem (libraries), and a huge community. It can be used for both: engine* and for the scripting layer (I've been doing a lot of fullstack development and being able to reuse the same code for multiple parts of the app is really nice ex: frontend/backend or in our case: game engine, scripting layer, web/desktop clients(?)).

It would be nice if the new server wouldn't require daily server saves/restarts (it's 2020 and Tibia servers still get daily shut down for maintenance 🤪). I'd say the game state should be persisted to the DB more often. I don't think RDBMS's can handle the number of updates produced by the game server (MMORPG like tibia where a lot of items get moved all the time etc) in real-time but maybe we could come up with some "transactional" system with often periodical flushes to DB - to avoid rollbacks to the last server save and to disallow cloning items on crash (with transactions).

Another idea would be to record changes to the game state in chronological order and save them somewhere (not our normal DB) in real-time. This way we would be able to replay what happened to check what lead to the crash and optionally recover from the crash (apply all the updates from our log to the DB until crash/state corruption). Something like binlog for mysql/mariadb.

---

* Critical, CPU sensitive things like encryption, compression, and whatnot could, of course, be written rust/c++ (as a c++ addon or compiled to WASM?) for obvious performance gains.
 
I would suggest server-side javascript (nodejs) with typescript on top of it for good developer experience. It's a nice language, easy to learn, has a huge ecosystem (libraries), and a huge community. It can be used for both: engine* and for the scripting layer (I've been doing a lot of fullstack development and being able to reuse the same code for multiple parts of the app is really nice ex: frontend/backend or in our case: game engine, scripting layer, web/desktop clients(?)).

It would be nice if the new server wouldn't require daily server saves/restarts (it's 2020 and Tibia servers still get daily shut down for maintenance 🤪). I'd say the game state should be persisted to the DB more often. I don't think RDBMS's can handle the number of updates produced by the game server (MMORPG like tibia where a lot of items get moved all the time etc) in real-time but maybe we could come up with some "transactional" system with often periodical flushes to DB - to avoid rollbacks to the last server save and to disallow cloning items on crash (with transactions).

Another idea would be to record changes to the game state in chronological order and save them somewhere (not our normal DB) in real-time. This way we would be able to replay what happened to check what lead to the crash and optionally recover from the crash (apply all the updates from our log to the DB until crash/state corruption). Something like binlog for mysql/mariadb.

---

* Critical, CPU sensitive things like encryption, compression, and whatnot could, of course, be written rust/c++ (as a c++ addon or compiled to WASM?) for obvious performance gains.
NodeJS isn't worth using as a MMO game server. Servers at least few hundreds of players would suffer a lot.
 
For me as a web developer tfs lack restful api. Imagine changing game state from website(e.g. voting for today's event). JWT for authorization should be just fine. With it community could create some new, fun stuff and admins could manage their server from admin panel. Just a concept I've been thinking on.
Another thing I was wondering, apart from tfs itself, is creating strictly a back-end service(without any template generation) for tfs, so there is a possibility for community to use any framework to implement their front-end solutions, but this is different topic.

The TFS devs currenting are targetting some sort of Boost ASIO + BEAST builtin-in httpd. While I think it would be terrible for daemon to try and host an actual website itself, I do believe a simple JSON API so the AACs no longer (have to) touch the database themselves is the actual itch they are trying to scratch.
Post automatically merged:

NodeJS isn't worth using as a MMO game server.

As a NodeJS expert, I have to agree. Node could be made to do this role in a performant fashion, but you'd be doing it with a lot of C-bindings which kinda defeats the purpose. Node does wonders on distributed workloads. The fact that I can write production ready software that can bring an AMD 7742 to it's knees in just under two weeks is a testament to how amazing Node is, but it's the wrong tool for this job.
 
From what I remember pyot had support for instances without a custom client. But in a limited way that the sec map format supported instances. Then the game server decided which instance to serve and all the clients received that single instance. It was a nice way to allow changing environments and did so in a way that made it easier for someone to map. The server knew when to load and unload sectors and when to change instances. So that was a goal of the system if the current source code says otherwise.


+1 less work and less software for a OT creator and player. Custom clients are amazing technology wise. The fact that one or few developer can make such functional clients is amazing. However I don't want to download a damn custom client for every OT I log into. It was great when I had one client installed and an IP changer. It was possible to switch between cipbia and OTs easily. Most OTs at the time were focused on keeping up with the latest client version. Now the community is very splintered between all different client versions that each player enjoys the most. I like tibia and the custom takes people have on tibia not some total different mmorpg that might as well be zezenia. TFS refusing to support newer protocols is a real pain and makes the splintering worse.

There are several client 11 features I would like to see in more OTs. Features such as the sprites, cyclopedia (map, bestiary, charms), ingame shop, and market would be beneficial. Features of lesser importance might be prey system, news, friends, inspect, and house auction system. I could imagine playing deathzot and be able to see the custom monsters and items in the cyclopedia.


On topic about a flexible server engine for Tibia and not reinventing a client, map editor, or new game. This thread confirms that developers have preferred languages. Devs wants to use the language they like most or are interested in learning. Every language mentioned so far is capable of getting the job done no doubt. What would be your goal with a modern engine? You mention it could be tough for the community to change scripting languages. So you must have some thoughts in mind about what this modern engine should accomplish. The goal will affect the decision of technology chosen besides just the language used to code the server.
  • Just looking for a new code base which is easier to maintain?
  • Must it support existing lua scripts?
  • More players online at once? Not many servers surpass 1K players as it is. More performance is fun but for what gain.
  • Less system resources required to run the engine?
  • Lower bar of entry to run a server for greenhorn devs?
  • Drop all usage of XML because "John Doe" finds it annoying to work with?
  • Single engine to support multiple client versions at the same time?
  • A engine which can make better use of containers?
  • Does ease of migration from the old engine to the new engine matter?
  • Quickest to a usable engine?
It is interesting to read all the unique takes on a modern server. I'm not very surprised by so many different ideas in this community.


See Asynchronous HTTP API with lua bindings by djarek · Pull Request #2010 · otland/forgottenserver (https://github.com/otland/forgottenserver/pull/2010) it might interest you.
That sounds less like instances and more like dynamic maps, but I'm basing my definition of instances on WoW.

Yeah, it's cool to see everyone's ideas and suggestions. Obviously, not everyone is going to agree on the same language, database, scripting language, etc.

I believe I've already answered what I think it should accomplish; maintainability. The monolithic design of TFS can be frustrating to work with and difficult to progress. A more modular system would possibly solve those issues.

Perhaps it could support scripts that are compatible with the current TFS release, but there are so many custom scripts it would be nearly impossible to support them all.

As a developer, performance gains are just a fun achievement whether you have 1 player online or 1000.
Database:
I worked a lot with NoSQL, and here's my feedback on them
Note: We have to think what do we want to accomplish before looking at different types of NoSQL, what do we want to be able to provide.

MongoDB is not great; it doesn't really bring anything good to the table over MySQL (except little bit better execution times for more results). MongoDB has also plenty of issues starting with its own design - replication or sharding, saying "Can't go wrongo with Mongo" is BS in real world of use when you are meaning to use its feature then its coming apart. (Not great at anything orientation, except marketting)

ElasticSearch is ok but does not fix the use as your typical database, it could be used as a map storage though if we ever want to have enormous size of map spanning couple GB's or TB's of data. ES is great for indices, and search feature - not something we can utilize correctly under small load of ots.
(Read oriented DB) - ok if we were to create some machine intelligence non-player controlled characters running and playing the game.

Cassandra its great if we ever wanted to have multiple world or one large world system, with game servers in different locations but needing to share same database/s quickly enough so that you can have same data in US, London and lets say Australia while being able to write to them simultaneously. Cassandra does that, but it doesn't handle indices well (newer versions actually completely removed them). Con - Hardware, needs a lot of separate nodes to work correctly.
(Its write oriented DB.)

Hbase on Hadoop
Great for reading millions of records at once. Great if you wanted to create some machine intelligence non-player controlled characters running and playing the game, then we would train the code with spark.
(read oriented)

Map:
Could run on something like ES or Cassandra allowing people to modify the map at the same time as its being played - this way if you dropped a backpack and game engine restarted everything is still there; and map editor could move things as game is being played; and multiple people can edit the map at the same time.


~ keep in mind that typical users of OTServers sometimes would even prefer to use sqlite db.
Like I mentioned in a previous post, perhaps a NoSQL database (or Mongo in particular) is the wrong choice for this environment. Your mention of Cassandra is interesting, though. Server owners have to make a decision where their server will be hosted (EU, NA, SA, ?), but being able to host the same world in multiple regions would be cool.

I like the idea of saving the game state as events transpire. If the engine crashed there would be no need for rollbacks, and there would be no need to freeze the server to do a global save ever so often.
I would suggest server-side javascript (nodejs) with typescript on top of it for good developer experience. It's a nice language, easy to learn, has a huge ecosystem (libraries), and a huge community. It can be used for both: engine* and for the scripting layer (I've been doing a lot of fullstack development and being able to reuse the same code for multiple parts of the app is really nice ex: frontend/backend or in our case: game engine, scripting layer, web/desktop clients(?)).

It would be nice if the new server wouldn't require daily server saves/restarts (it's 2020 and Tibia servers still get daily shut down for maintenance 🤪). I'd say the game state should be persisted to the DB more often. I don't think RDBMS's can handle the number of updates produced by the game server (MMORPG like tibia where a lot of items get moved all the time etc) in real-time but maybe we could come up with some "transactional" system with often periodical flushes to DB - to avoid rollbacks to the last server save and to disallow cloning items on crash (with transactions).

Another idea would be to record changes to the game state in chronological order and save them somewhere (not our normal DB) in real-time. This way we would be able to replay what happened to check what lead to the crash and optionally recover from the crash (apply all the updates from our log to the DB until crash/state corruption). Something like binlog for mysql/mariadb.

---

* Critical, CPU sensitive things like encryption, compression, and whatnot could, of course, be written rust/c++ (as a c++ addon or compiled to WASM?) for obvious performance gains.
I wouldn't be opposed to TS as a scripting language, but I can't agree to Node.js for the engine. You're just not going to get the same performance as you would with a low-level language, even if you interop with them. Maybe it can handle 1000 players online just fine, but maybe it can't? It would be frustrating to put all that time into something that ends up being limited.
 
Ideally we should have the heavy lifting done by C++ while scripting everything else in lua(or python or javascript). I don't know how far it can be taken, but it would be nice if almost all the game engine core was in lua.
 
As a TypeScript developer I don't think TypeScript/JS should be the core/internals due to the GC/perf issues.
 
due to the GC/perf issues.
Are you expecting users to do Mandelbrots in the gluescript?

it would be nice if almost all the game engine core was in lua.
If you guys really want to take it that far then you better fork LuaJIT and start baking in some creature comforts, because Lua's syntax oddities and minimalism-to-the-point-of-religious-asceticism corelib make it inhospitable for "living in" to such a level.

At that point, maybe the API should be done in Lua? That still brings us right back to basically remaking Luvit without the benefit of luv and underpinnings since TFS already has that role filled.
 
...but it would be nice if almost all the game engine core was in lua.

Why? I see so many people saying this, I never understood the reason behind it.

Also, while this discussion is really nice to have, I don't see any reason to do something like this when there are 3 year old issues still not resolved.

Its like you are expanding the wheel while the wheel is moving but the wheel is not really a wheel, its more like a square. First get the wheel to be round, then do all the revamp.

And another thing, getting break changes on TFS 2.0 would be one thing. Another completely different would be to drop lua support. While the people on this topic would appreciate and contribute, I don't believe live servers would be cool with migrating all their datapack to javascript or something like that. And that would bring the big servers contribution to a even lower number. And the new version's adopt rate.
 
Last edited:
Are you expecting users to do Mandelbrots in the gluescript?


If you guys really want to take it that far then you better fork LuaJIT and start baking in some creature comforts, because Lua's syntax oddities and minimalism-to-the-point-of-religious-asceticism corelib make it inhospitable for "living in" to such a level.

At that point, maybe the API should be done in Lua? That still brings us right back to basically remaking Luvit without the benefit of luv and underpinnings since TFS already has that role filled.
My simple idea for now was to implement the networking code in C++ and then pass the messages(packets) to lua using a lock free queue that lua would call a function to get the items from. It would be lua game loop(different from tfs which is task based) which is much like the design that cipsoft chose which i think gives us more control, prediction and maybe performance. Maybe lua gc could be controlled to only work in the end of the loop.

As someone that has been studying and optimizing tfs, the bottlenecks are: memory allocation/deallocation, Map::getTile function, Game::getCreatureById, item attributes, path finding and get spectators. I think map/getspectators/pathfinding would be needed to be implemented in the C++ side. Everything else could be done in lua. [Feature Request] parsePacket to Lua! · Issue #1786 · otland/forgottenserver (https://github.com/otland/forgottenserver/issues/1786#issuecomment-218434697) Mark opinion this.

One note on the luajit subject: I believe in tfs we only use the luajit interpreter. There is few reason that luajit won't jit the lua code and will fallback to interpreter mode and few years ago i made a test and as far i remember it was kinda impossible to fix it in tfs(i think it was due the lua c binding? idk). Its explained here: Not Yet Implemented (http://wiki.luajit.org/NYI)

Why? I see so many people saying this, I never understood the reason behind it.

Because there is a layer between the C++ and lua that i don't like. Btw since it uses the old lua c api its slow(as far i have been reading in the internet, ffi is the way for performance). I want freedom, flexibility, speed of development. I also want my server to not crash when there is a small mistake.

And another thing, getting break changes on TFS 2.0 would be one thing. Another completely different would be to drop lua support. While the people on this topic would appreciate and contribute, I don't believe live servers would be cool with migrating all their datapack to javascript or something like that. And that would bring the big servers contribution to a even lower number. And the new version's adopt rate.
Live servers got the money to fix or adapt anything they want. They already have their own custom tfs which most of time does not get updates from the main repo. "And that would bring the big servers contribution to a even lower number. " Name the big contributions they have made.
 
Last edited:
Name the big contributions they have made.

🤔 They lined the pockets of someone who fixed all the bugs in the client and then didn't send a single patch back upstream? 🤐

memory allocation/deallocation
The eternal problem. Yair Sade, Mooly Sagiv, and Ran Shaham wrote an interesting paper a few years back about thread local storage.

path finding
And this is where you enter the serious CS zone and things go to full esoteric graph theory. Does the engine already leverage a separate connectivity checking algorithm (UF) where it makes sense to? Since lets you return early in your pathfinder for the small price of a O(1) function call?
 
Why? I see so many people saying this, I never understood the reason behind it.

Also, while this discussion is really nice to have, I don't see any reason to do something like this when there are 3 year old issues still not resolved.

Its like you are expanding the wheel while the wheel is moving but the wheel is not really a wheel, its more like a square. First get the wheel to be round, then do all the revamp.

And another thing, getting break changes on TFS 2.0 would be one thing. Another completely different would be to drop lua support. While the people on this topic would appreciate and contribute, I don't believe live servers would be cool with migrating all their datapack to javascript or something like that. And that would bring the big servers contribution to a even lower number. And the new version's adopt rate.
To expand on what @Yamaken said, the reason I've seen is that by having the majority of the core of the game in Lua means there's less need to modify the low-level language. Think of it like, a beginner could download a release and customize it greatly without ever having to touch the C++, or even compile it. But that's a lot easier said than done.

Don't get it twisted, I have no intentions of doing anything in this thread. Its only purpose is a theoretical discussion, which, so far, I've found interesting. And you never know, maybe this discussion spurs a new wave of investment in TFS.
 
Also for better performance i thought about a server engine that mix single thread and multithread using fibers/job system. How it would work? Main thread runs the game engine loop(only main thread is allowed to produce global state changes), while inside the game engine loop you use tasks to split the trouble between threads and the main thread will wait while the tasks are executed(even the main thread could be used to execute the tasks). With this, we could avoid locking and problems with multi threading since the tasks we are going to execute won't have colateral effects(global state change). This could be used for packet processing, path finding, some type of async spectator code, idk.
Post automatically merged:

Don't get it twisted, I have no intentions of doing anything in this thread.
I think some of us have.
Post automatically merged:

🤔 They lined the pockets of someone who fixed all the bugs in the client and then didn't send a single patch back upstream? 🤐


The eternal problem. Yair Sade, Mooly Sagiv, and Ran Shaham wrote an interesting paper a few years back about thread local storage.


And this is where you enter the serious CS zone and things go to full esoteric graph theory. Does the engine already leverage a separate connectivity checking algorithm (UF) where it makes sense to? Since lets you return early in your pathfinder for the small price of a O(1) function call?
I don't know how the pathfinding in tfs works honestly.
About memory allocation, if you have a game loop and your engine is written in a language like C++ you can have a huge memory pool that can be used to allocate temp objects which won't exist at the end of the loop, in the end of the loop you just reset the pool and you reuse the same memory for ever without the trouble of calling slow allocation methods at all. Not doable with lua :(
 
Last edited:
There are several client 11 features I would like to see in more OTs. Features such as the sprites, cyclopedia (map, bestiary, charms), ingame shop, and market would be beneficial. Features of lesser importance might be prey system, news, friends, inspect, and house auction system. I could imagine playing deathzot and be able to see the custom monsters and items in the cyclopedia.
Solution is easy, develop otclient. I'm sure that you can make almost everything you want. As you can see we did something similar that you talked about on our server:
uHCuz6J.png
 
Whichever direction this community decides to go, I want to be involved in both learning and becoming an active presence/developer for TFS and possibly an OTClient project.
 
Back
Top