• There is NO official Otland's Discord server and NO official Otland's server list. The Otland's Staff does not manage any Discord server or server list. Moderators or administrator of any Discord server or server lists have NO connection to the Otland's Staff. Do not get scammed!

Proof of concept of a new game engine

bpawel10

Premium User
Premium User
Joined
Nov 26, 2016
Messages
52
Solutions
1
Reaction score
85
TL;DR
I started creating a new game engine and now I’m sharing a proof of concept with you to get feedback. If my idea turns out to make sense I’ll continue working on it and hopefully after a few years we will finally eradicate that crap called TFS.

Introduction
First of all, I want to thank you guys for creating and contributing to this thread, thanks to it I discovered Rust and started learning it, and after reading a few chapters of this great book I just fell in love with it. No wonder it's the most loved language for 6 years now, it’s simply the best programming language on the planet.

So, inspired by the thread I mentioned above, I now finally started creating this modern engine. And here are the goals I want to achieve:
  • modern technologies
  • modular architecture
  • easily customizable
  • mechanics as close to Cipsoft as possible (as default, but easily customizable, as everything)
  • support for as many protocol versions as possible
  • no unsafe code
  • unit/integration/e2e tests
Engine architecture
Now, let me explain the structure of the game world: it’s a hash map of tiles. Every tile contains a vector of entities, and an entity can be anything: player, monster, item, NPC, and so on. There’s no inheritance here, but composition instead - every entity can have attributes. So for example, you can use entity! macro to easily create an item with action id:
Code:
entity![Item(1), Action(1)]
as long as Item and Action attributes exist, of course.

System
The main concept of the engine architecture is to keep its core as minimal as possible, and put all business logic in systems instead. To create a system, create a new Rust file anywhere inside systems directory and write:
Code:
use crate::core::prelude::*;

system! {}
and now, you can put other things inside these curly brackets. Here’s a list of "building blocks" that systems can contain:

Attribute
Attribute is a struct that implements Attribute trait. But all this implementation code will be added automatically by the #[attribute] macro, so all you need is:
Code:
#[attribute]
pub struct Description(String);
Attributes are used to insert some data to entities. You can also create an attribute without any fields inside and use it as a flag. #[attribute] macro will under the hood create new getter method on Entity struct (which returns Option type). For example in this case it would create .description() method that allows you to easily get access to the attribute. You can create attributes with any data you want, and then add them to entities, without modifying engine core. There are no methods for modifying attributes, though, because only commands should be able to change game state.

Event
Event is a struct that implements Event trait. Again, it’s easy:
Code:
#[event]
pub struct UseEvent {
    source: Option<Entity>,
    target: Entity,
}
Events are obviously used to handle something that happened. You can react to events using effects, or emit them using tasks. The engine can also emit some basic events, for example when a tile or an entity was updated.

Command
Implements Command trait.
Code:
#[command]
pub struct EmitEventCommand(pub EventType);
Commands are (or at least should be) the only way to change game state. You can return them from effects and the engine will process them in loop.

Task
Task is a block of asynchronous code that is actually a stream.
Code:
task! {
    // do something
}
Task should be used if your system needs to run some asynchronous operations. You can „return” multiple values from it and await some things in the meantime. To „return” a value, use yield keyword. Notice that you can only yield Option<Event>. Yielding None can be useful if you have a task that only awaits something and there’s no need to return any value.

There’s one tricky thing about tasks - the syntax above automatically adds it to tasker under the hood, so you can’t use it inside effects. I’ll think about how to solve it in a better way, but for now you can create a task inside effect manually, for example:
Code:
let task = Box::pin(stream! {
    // do something
 }) as TaskType;
and then you can return task variable from effect.
There are also other ways to create a stream, for example into_stream or unfold.

Effect
Effect is a function that accepts an event and returns vectors of commands and tasks.
Code:
#[effect(TickEvent)]
fn handle_tick(event: EventType, attributes: GameAttributesType, world: WorldType) -> EffectResultType {
    // do something
    Some((Vec::new(), Vec::new()))
}
There is a big flaw here - both attributes and world arguments passed to effect are of type Arc<Mutex<T>>, so it’s actually possible to directly change game state here. This is something that definitely has to be solved in the future.

And how all these blocks work together? It’s basically a mix of: CQRS, event sourcing and ECS. If you want more, check out this great video that inspired me.

First of all, engine core is multithreaded. One core is used for game simulation, and all other cores are managed by tokio (it can move spawned tasks between threads if needed). First, there is a tasker (which runs in tokio task) that waits for new tasks and processes them asynchronously, and when a task yields an event, it then wraps it in EmitEventCommand and sends to the engine. The engine waits in loop for new commands and processes them, usually changing game state. Then it can also emit events to notify everyone that the game state has changed. There’s a special command I mentioned above, EmitEventCommand, which will call all listeners for specific event that are registered. And listeners are just these functions with #[effect(…)] attribute, that are registered at server startup.

So for example, what will happen if you use a lever in the client?
  1. Task that runs tcp server will get and decode the packet and emit GamePayloadEvent containing decoded payload
  2. Appropriate effect will get this event and match it with Payload::UseItem variant, and then emit UseItemPayloadEvent containing position and item id
  3. Another effect will get this event and return EmitEventCommand containing UseEvent
  4. Effect in lever script will get this event, check that its action matches Actions::Lever and return SetEntityAttributeCommand to switch the lever (change its id)
  5. Engine will process this command and emit ChangedEntityEvent containing position and attribute name
  6. Another effect will get this event, check that the attribute that has changed was item, and then send a packet to the client to notify it about the item update

Proof of concept
I created a proof of concept that allows you to login, walk, switch a stone switch and use a lever. It has a few things hardcoded (map loader, login/game login packets, registering systems), and some parts of the code could be better, for example there’s quite a lot of cloning which I’ll have to solve somehow, and there are multiple compiler warnings, mostly about unhandled results, but who cares - for now it’s enough to demonstrate the idea. And here's how it looks:

View attachment skyless_poc.mov

If you want to test it yourself, here's the code.



Besides these two scripts for handling switch and lever that I mentioned above, there’s also a tick system. It’s disabled by default, I included it just to show you an example of how we can use tasks to yield multiple events. In this case it will yield TickEvent every second, which could later be used to implement monsters or NPC „thinking”. And of course the whole network code was implemented as systems, so you can also check it.

So now it's time for your feedback. What do you think about this architecture? Maybe you can see some other use cases that will be hard or even impossible to implement using systems in their current shape? Let me know, so I can tweak it before it grows. And last, what do you think about the name?
 

trou

ot revolution is comming
Senator
Joined
Aug 12, 2008
Messages
1,351
Reaction score
34
I'm speechless. Great job! I recently started using Rust and compiling it to WASM and also feel in love with it.

Skyless sounds good. "Sky is the limit" but since there's no sky, there are no limits. I'm not sure though if it will be possible to implement lua script system alongside systems, or I'm wrong?
 

jo3bingham

Excellent OT User
Joined
Mar 3, 2008
Messages
1,085
Solutions
14
Reaction score
708
GitHub
jo3bingham
I've also been working on a POC engine in Rust (login/game server, loading map from disk, logging in, and walking around is as far as I've gotten), but I may scrap that and contribute to yours. When I get a chance to sit down at my computer I'll have a look through the code, but the way you've explained it in your post sounds exciting. Would you be willing to enable Discussions on the repo? I find that to be a better place for questions and conversations than Issues.
 

oen432

Legendary OT User
Joined
Oct 3, 2014
Messages
1,457
Solutions
49
Reaction score
1,550
Location
Poland
GitHub
Oen44
Last edited:

jo3bingham

Excellent OT User
Joined
Mar 3, 2008
Messages
1,085
Solutions
14
Reaction score
708
GitHub
jo3bingham

oen432

Legendary OT User
Joined
Oct 3, 2014
Messages
1,457
Solutions
49
Reaction score
1,550
Location
Poland
GitHub
Oen44
Also... this is Game Server you are talking about, not Game Engine. You can't play without separate client.
 
OP
OP
B

bpawel10

Premium User
Premium User
Joined
Nov 26, 2016
Messages
52
Solutions
1
Reaction score
85
I'm not sure though if it will be possible to implement lua script system alongside systems, or I'm wrong?
If you're asking if it's possible to implement it with this systems architecture, it is. And I think I'll have to do it eventually, because the community demands it. But thanks to this architecture if someone hates Lua (like me) he can just create a system for scripts in any other language he likes and just replace it. I was thinking about TypeScript (or at least JavaScript), but the problem is they're much harder to add as scripting languages, so maybe I'll just stick to Rust.

Would you be willing to enable Discussions on the repo?
enabled
 

Shawak

Intermediate OT User
Joined
Sep 11, 2008
Messages
1,980
Solutions
2
Reaction score
114
Location
Germany
GitHub
Shawak
Cool! I've also written a parser in Rust for .dat, .otb, .otbm and .spr files in case you are interested, although it's not 100% finished
 

jo3bingham

Excellent OT User
Joined
Mar 3, 2008
Messages
1,085
Solutions
14
Reaction score
708
GitHub
jo3bingham
Now that I've had time to sit down and look at the code fully, here's my thoughts. I like the system, I like the modularity. I like that Rust basically forces you to use anything other than OOP. My main concern is complexity. I was able to understand it fairly quickly, but it may be intimidating to others; especially those who have never worked with Rust. If you can build it in a way that very little modification needs to be done to the Rust source code, or supply detailed documentation, then that may not be a problem.

If you're serious about taking over TFS then you will have to include Lua scripting, and, not only that, the Rust engine needs to do as little as possible while the scripting needs to handle as much as possible (which I realize is easier said than done). This will allow those who have issues with Rust (and C++ for that matter) to be more likely to pick up your system.

The last thing I want to point out is that you will need to accept the fact that you'll probably end up being the only contributor to your project. It's unfortunate, but it's true. I won't get too much into it here, but I will say that I'm willing to contribute as much as I can (I'm a full-time software engineer, I'm married, and I'm getting old, so I can't contribute as much as I'd like), but the odds that commits to your repo will only be from you for some time is very high.

On a separate note, I am curious, if you do decide to continue with the project, what your next goals are. It would be interesting to see a roadmap of what needs to be done to reach a MVP (i.e., what point would you be happy to publish an alpha/beta release).
 

slayzer

Klaffen
Joined
Dec 28, 2008
Messages
272
Solutions
1
Reaction score
22
Location
Norway
Projects like this makes me wanna learn Rust :D
I do hope this project lives long enough for me to learn it and find a way to contribute.
 
OP
OP
B

bpawel10

Premium User
Premium User
Joined
Nov 26, 2016
Messages
52
Solutions
1
Reaction score
85
If you can build it in a way that very little modification needs to be done to the Rust source code
I think I explained it in the first post, the goal is to allow users to implement anything without modifying the core, but it doesn't mean you don't have to know Rust. But it also doesn't mean everyone using it has to know Rust, they can use systems created by others.

Let's take this example: you want to create a cast system.

To do that in TFS, you not only have to know C++ very well, but also change a lot of existing files. And then if you share it with others, they will have to either use your code with cast system as a whole, or in case they already have their own TFS version with some customizations, take a diff of your cast system and manually change all these things in their code and prey that it worked.

To do that in this engine, you have to know Rust very well, but you can implement everything by creating new systems. If you need to store a list of casts, you can create a new attribute inside a new system and add it to the game. If you need to store some additional cast data in player entity, you can create an attribute for that as well. If you need a new tcp listener for cast spectators, you can do it in system, inside a task. And then people can just copy paste these files, enable them in config and it'll work.

Actually, now when I'm thinking about this example, I can see some issues if you try to implement it using systems in their current shape. For example, it would require some kind of overriding mechanism, to allow you to create a new login handler that would return casts when login and password are empty. Or maybe it would be enough to tell people they have to disable default login system manually in config? Now you can see why I wanted to share it with you when it's small and discuss use cases like this one.

If you're serious about taking over TFS then you will have to include Lua scripting
I was thinking about scripting language today, and kinda changed my mind. First, regarding multiple languages support, it should be quite easy to achieve with systems, but I think the best approach would be to choose one language that will be used in the repo and officially supported. And then I or community can write systems for other languages in separate repos. Regarding Lua, I don't think it should be this supported language just because it's used in TFS. This is a new engine, and I shouldn't look at what TFS chose but what's the best. What's the point of creating a new engine that will imitate another?

So I did more research today and I'm thinking about rhai. There's one downside, though - I couldn't find anything about writing unit tests. But it's safe at least, while Lua is written in C.

Another crazy idea would be to use Wasm. This way scripts could be written in any language that can be compiled to Wasm, for example AssemblyScript. Unfortunately, I couldn't find any mature enough Wasm compiler for Lua. But this is probably overkill anyway.

Rust engine needs to do as little as possible while the scripting needs to handle as much as possible
So where do you think should be the limit? I don't want it to turn into a Lua engine. Also I don't understand why you're so afraid of Rust. If the engine becomes popular people will start writing systems and giving/selling them to others. I'll also think about adding a possibility to register systems with arguments, and then every system could either have its own config file, or a separate section in main config.

If we're talking about systems, here's another thing - if you look at this code (ignore commented ping system) you can see that these 2 network handlers have to be registered separately, although they're in fact a part of game server. So we definitely need some composition here, either allowing one system to include others, or combining multiple systems into plugins.

you'll probably end up being the only contributor to your project
This is fine, I didn't expect anyone to write any code, I expected to discuss the architecture. In the end, this community is famous for talking rather than doing.

It would be interesting to see a roadmap
Ok, I'll try to write one.
 

jo3bingham

Excellent OT User
Joined
Mar 3, 2008
Messages
1,085
Solutions
14
Reaction score
708
GitHub
jo3bingham
Let's take this example: you want to create a cast system.
Ah, that’s a very fair point.
I think the best approach would be to choose one language that will be used in the repo and officially supported
Agreed. While I understand your dislike of Lua, the reason I mention it is because that’s what people in this community are used to. There’s a better chance of getting scripters to contribute if it’s a language they’re used to.
Also I don't understand why you're so afraid of Rust.
I can promise you I’m not afraid of Rust. I love Rust. If it were up to me, the engine and scripting system would all be in Rust. But I don’t think that’s realistic, at least not for community adoption.
Ok, I'll try to write one.
Awesome! Even if it’s a minimal roadmap, it would at least help me, and others, understand what could possibly be contributed now.
 
OP
OP
B

bpawel10

Premium User
Premium User
Joined
Nov 26, 2016
Messages
52
Solutions
1
Reaction score
85
There’s a better chance of getting scripters to contribute if it’s a language they’re used to.
Scripting languages are easy, I think if someone has learned Lua now he just needs to learn new syntax.

Also, I'm not sure if you realize that even if we choose Lua, it won't be that easy to move from TFS. Functions arguments will vary, and more importantly, the way it works - you'll have to return an array of commands or something like that from Lua script, similarly to the way it works now in effects. You can't directly change player or item data inside Lua script, it's not how the engine should work. So regardless of the language if someone already created a big TFS project and don't want to spent a lot of time migrating he'll still use TFS.
 

oen432

Legendary OT User
Joined
Oct 3, 2014
Messages
1,457
Solutions
49
Reaction score
1,550
Location
Poland
GitHub
Oen44
You can't directly change player or item data inside Lua script, it's not how the engine should work.
You can't do that using TFS either. You have to use bindings and call C++ functions from Lua, that's where data is modified, not strictly inside Lua.
 

jo3bingham

Excellent OT User
Joined
Mar 3, 2008
Messages
1,085
Solutions
14
Reaction score
708
GitHub
jo3bingham
That’s a good point.
You can't directly change player or item data inside Lua script, it's not how the engine should work.
Agreed. One thing I hate about TFS (other than being a monolithic monstrosity) is how heavily it leans on OOP and causes the codebase to be this spaghetti-web of indirection.

In that case, I second using rhai for the official scripting language of the engine.
 

Yamaken

Pro OpenTibia Developer
Premium User
Joined
Jul 27, 2013
Messages
521
Solutions
4
Reaction score
407
That’s a good point.

Agreed. One thing I hate about TFS (other than being a monolithic monstrosity) is how heavily it leans on OOP and causes the codebase to be this spaghetti-web of indirection.

In that case, I second using rhai for the official scripting language of the engine.
You mean how it does not rely on components like an entity system?

When you want to add game content scripting is the only sane way.
 
OP
OP
B

bpawel10

Premium User
Premium User
Joined
Nov 26, 2016
Messages
52
Solutions
1
Reaction score
85
Ok, here's the roadmap. In general, the plan is to recreate the behavior of the leaked Cipsoft engine, but using modern technologies (it’s mainly about file formats). It’s worth noting that even Cipsoft now believe that creating their own formats wasn’t a good idea.

Config and systems composition/plugins
I explained it a bit a few posts above, basically we need to allow one system to be split into multiple files or allow one system to include others or add something like plugins that would consist of systems. And every system could be configurable either in its own section in the main config file or in its own config file that would be then imported. And I also want to allow importing predefined configs. This could be used to create separate config files for each protocol version. For example configs for versions <= 8.2 would add game server system with Adler-32 checksum feature disabled, while configs for 8.3+ would enable it, and the same for all the other differences. This way we can have only one codebase and if you want to change protocol version you just change one line in the main config file.

There’s one quite promising config format that supports imports, but there’s no way to override imported values, and these imports must be added at the beginning of the file, so maybe we should find something better. Overriding imported values would be very useful if someone wanted to customize it, for example create a 7.4 server without uh trap.

Map
I described a new map format some time ago and I think we should use something like that in this engine. But for that we also need a map editor. I actually had an idea to create an app that would have all the tools put together (map editor, object builder, maybe some tools to generate things like NPCs or monsters), but for now I could create just a map editor that could be expanded in the future. It would delay the engine, but I never said it will be ready anytime soon, did I?

Items
This one is quite simple, instead of XML and OTB files we need one text file for everything. TOML seems to be suitable for this purpose.

Database and query manager
MySQL is unlikely to work because there are no schemas, you can define any custom attribute that later has to be saved in the db somehow. PostgreSQL has this JSONB column type that maybe could work, but I think NoSQL database would still be better. I have been using MongoDB at work for several years, though, and it's not that good in my opinion. When the application starts growing you suddenly have to write very complex queries using aggregate and it just gets worse and worse. Fortunately, there are also graph databases. But I never used any of them, so I will have to do some research.

Also regarding things like login server or query manager, I think that they should be included as part of the game server as default, because it’s just much simpler to deploy and it’ll be enough for most users, but I also want to allow more advanced users to move them into separate projects, for example if they want to have multiple game worlds or reuse query manager in website backend.

Protocol
This one’s simple as well, basically we have to write all the missing code for decoding/encoding packets.

I was also thinking about refactoring it a little, because now all this parsing is done in one place using match, but it would be better to split it into separate files imo. And to do that we would need to change payload enum a little, because you can’t implement anything only for specific enum variant.

So for example, instead of:
Code:
enum A {
    B { b: u8 },
    C { c: u8 },
}
we can have:
Code:
struct B { b: u8 }
struct C { c: u8 }

enum A {
    B(B),
    C(C),
}
and now it can be split easily. Or maybe it would be even better to use trait objects here as well?

Scripting
So we have to choose a language and implement it. I asked on Rhai discord about unit testing and got a reply that it’s a good question and someone should create a framework for testing, but who knows when it will happen? So maybe we should wait with it for some time.

Mechanics
Basically we have to decompile Cipsoft’s leaked binary and implement all mechanics based on that. It’s actually not that hard, I played with it yesterday, first you need to decompile it to C pseudocode using IDA Pro with HexRays and then to make it readable you just have to convert some variables to proper structs and rename them. But it's for sure a lot of manual work.


Let me know if I missed something.
 
Top