• There is NO official Otland's Discord server and NO official Otland's server list. The Otland's Staff does not manage any Discord server or server list. Moderators or administrator of any Discord server or server lists have NO connection to the Otland's Staff. Do not get scammed!

.Net Core OT Producer/Consumer model

frozenrose

New Member
Joined
Dec 15, 2019
Messages
5
Best answers
0
Reaction score
3
I am developing my own engine in C# . My goal is to create an engine that work with Parallel tasks that all take care of a part of the system in a non blocking way, instead of a Scheduler to reduce the amount of context changes.

I am designing it for my own project so the goal is not to create a replica of Tibia systems and formulas, but instead to make it very easy and straightforward to add new systems like highly customizable items, unique monsters with levels/custom behavioral system and such.

I wanted to know if it could interest anyone. The source code is fairly messy for now, but I should be starting to clean it shortly (in the upcoming weeks) :)

Liza
 

Znote

<?php echo $title; ?>
Staff member
Global Moderator
Premium User
Joined
Feb 14, 2008
Messages
6,802
Best answers
236
Reaction score
1,534
Location
Norway
So many C# projects are spawning and dying in the OT scene.

The current one that is active appears to be this: jlnunez89/fibula-mmo (https://github.com/jlnunez89/fibula-mmo)
I recommend you to latch onto those guys, and try to make it survive. Perhaps implement your vision there. It would be cool to see something go beyond the prototype stage everyone gets stuck in.

They got a discord group thats fairly active these days. Not a bad day for a C# developer.
 

Ochman

Advanced OT User
Joined
Feb 27, 2016
Messages
302
Best answers
4
Reaction score
193
I - honestly - always respected someone else's projects, no matter how different it would be from my taste. The amount of time, skill and effort put into ANY project, game, graphic, youtube channel, painting or whatever deserves at least minimum respect.

That being said, I am really worried about the future of solo engine developing. I am not doubting your skill, I am doubting effort that one person has to put in. We are not talking about one week, one month or even one year. We are talking about years of development and that regardless of what will happen in your life, you have to put every single free minute in this project to succeed.

Ironically, I am not saying it to discourage you, but rather to motivate. Good luck on that. I'll watch the project; but sadly - I won't contribute in any way, since I am bad at C#.
 

Delirium

OTLand veteran
Staff member
Global Moderator
Joined
May 28, 2007
Messages
3,348
Best answers
1
Reaction score
231
Location
Athens, Greece
So many C# projects are spawning and dying in the OT scene.

The current one that is active appears to be this: jlnunez89/fibula-mmo (https://github.com/jlnunez89/fibula-mmo)
I recommend you to latch onto those guys, and try to make it survive. Perhaps implement your vision there. It would be cool to see something go beyond the prototype stage everyone gets stuck in.

They got a discord group thats fairly active these days. Not a bad day for a C# developer.
As a C# developer, I can also vouch for that source code. It looks professional, clean and well-designed.
 
OP
F

frozenrose

New Member
Joined
Dec 15, 2019
Messages
5
Best answers
0
Reaction score
3
This fibula project does look very clean, yes. It is not my type of coding though, so I won't join on the effort. It uses a lot of external ressources which creates a bunch of overhead in their network communication and such. They also use an approach that put enormous strain on the garbage collector, which become a real issue in project like a game server that runs for extended periods of time.

Thank you all for the encouragements. I am not planning on implementing all of tibia's original features, but instead creating a backbone that will let people build their own system on top of it. All the features like the market and such will be implemented to interact with the client, but the monster system will be very different. Same for the item system and such.

I don't necessarily plan on needing help on the c# code itself, at best I might need people to try and make it crash at some point so I can fix the issues :)

The goal is to give a fully linear approach so people can run their server very efficiently on a single core machine (or single vcore cloud service) and automatically scale to more layers of "producer/consumer" as more cores are available. I am trying to avoid as much context change and exception catching as possible to not reduce the performance without reasons. In micro tests it seems to be doing very well until now, will see once everything is connected together. I am going to implement 8.6 to start with and then add the other version as "addition" so people can use the same source to run any version they desire.

I think that with project like the client on unity that is written in c#, it will give a good chance to people to end up creating their own little game :)

And yes, such a project is very time consuming so it won't be production ready for a while. I have hopes though :)

Liza
 

Ezzz

Developer
Joined
Feb 26, 2010
Messages
1,738
Best answers
2
Reaction score
470
Location
Caracas, Venezuela
I was trying to make one, but due to lack of time I stopped working on it, also because the RSA encryption system always failed for some reason under C#.

 
OP
F

frozenrose

New Member
Joined
Dec 15, 2019
Messages
5
Best answers
0
Reaction score
3
Yeah, the RSA part was a pain to implement. There was a lot of testing to get it working.

If you look at the SharpOT project on github you will see how it got implemented in this project, which works. It isn't the cleanest implementation, but starting with it you should be able to set up a RSACryptoProvider easily :)
 

Ezzz

Developer
Joined
Feb 26, 2010
Messages
1,738
Best answers
2
Reaction score
470
Location
Caracas, Venezuela
Yeah, the RSA part was a pain to implement. There was a lot of testing to get it working.

If you look at the SharpOT project on github you will see how it got implemented in this project, which works. It isn't the cleanest implementation, but starting with it you should be able to set up a RSACryptoProvider easily :)
That implementation fails every 2 logins, fails to decrypt the packet sometimes.
 

taght

New Member
Joined
Jul 2, 2007
Messages
54
Best answers
0
Reaction score
1
I believe it's a good idea, but you are not first on this heroic path. Like others say, that is a lot effort to be made to achieve any reasonably state, I hope you will be the one with enough determination.
 

Delirium

OTLand veteran
Staff member
Global Moderator
Joined
May 28, 2007
Messages
3,348
Best answers
1
Reaction score
231
Location
Athens, Greece
May I ask, out of curiosity, how is the producer/consumer model going to be implemented in this server? What benefits would that have?

The main reason I am asking is because the producer and the consumer are usually linked by a queue of data and in this case, I fail to see what sort of data a Tibia server needs to queue.
 
OP
F

frozenrose

New Member
Joined
Dec 15, 2019
Messages
5
Best answers
0
Reaction score
3
That implementation fails every 2 logins, fails to decrypt the packet sometimes.
I am not using this implementation anymore, but I never had this issue over numerous tests when I did. Are you sure that your buffer is filled correctly everytime?
Post automatically merged:

May I ask, out of curiosity, how is the producer/consumer model going to be implemented in this server? What benefits would that have?

The main reason I am asking is because the producer and the consumer are usually linked by a queue of data and in this case, I fail to see what sort of data a Tibia server needs to queue.
On single core, everything will be happening in a single loop with priority based on timing.

On multiple cores one loop will be processing login/messages, then adding the action that affect the player to its "behavior to process" list/queue.

Another loop will be handling most of the game logic and the Players are treated like any other creatures, except that they don't "go to sleep" when they don't have spectators(obviously). Every creatures that is awake then get their behavior processed. (other behavior like regeneration and such get processed when the creature "awaken" calculating the time it was asleep)

This way I don't need to make anything threadsafe and I avoid costly context changes (I will see how well it performs when I get a first prototype up and running, some parts of the design might change then)

I got most of the systems working by themselves, isolated. Now I just need to connect everything together and make it works with the client. I am "losing" some time on the items.otb and items.xml right now. Making a parser to convert them to a json format that will reflect my needs better (and make it easier to add custom items)
 
Last edited:

Yamaken

Pro OpenTibia Developer
Joined
Jul 27, 2013
Messages
504
Best answers
3
Reaction score
374
I am not using this implementation anymore, but I never had this issue over numerous tests when I did. Are you sure that your buffer is filled correctly everytime?
Post automatically merged:



On single core, everything will be happening in a single loop with priority based on timing.

On multiple cores one loop will be processing login/messages, then adding the action that affect the player to its "behavior to process" list/queue.

Another loop will be handling most of the game logic and the Players are treated like any other creatures, except that they don't "go to sleep" when they don't have spectators(obviously). Every creatures that is awake then get their behavior processed. (other behavior like regeneration and such get processed when the creature "awaken" calculating the time it was asleep)

This way I don't need to make anything threadsafe and I avoid costly context changes (I will see how well it performs when I get a first prototype up and running, some parts of the design might change then)

I got most of the systems working by themselves, isolated. Now I just need to connect everything together and make it works with the client. I am "losing" some time on the items.otb and items.xml right now. Making a parser to convert them to a json format that will reflect my needs better (and make it easier to add custom items)
I think its still looks like tfs since tfs has the asio thread/loop(producer) and the dispatcher loop(consumer).
You still need thread safety in the queues/thread communication.
 
Last edited:

Delirium

OTLand veteran
Staff member
Global Moderator
Joined
May 28, 2007
Messages
3,348
Best answers
1
Reaction score
231
Location
Athens, Greece
I am not using this implementation anymore, but I never had this issue over numerous tests when I did. Are you sure that your buffer is filled correctly everytime?
Post automatically merged:



On single core, everything will be happening in a single loop with priority based on timing.

On multiple cores one loop will be processing login/messages, then adding the action that affect the player to its "behavior to process" list/queue.

Another loop will be handling most of the game logic and the Players are treated like any other creatures, except that they don't "go to sleep" when they don't have spectators(obviously). Every creatures that is awake then get their behavior processed. (other behavior like regeneration and such get processed when the creature "awaken" calculating the time it was asleep)

This way I don't need to make anything threadsafe and I avoid costly context changes (I will see how well it performs when I get a first prototype up and running, some parts of the design might change then)

I got most of the systems working by themselves, isolated. Now I just need to connect everything together and make it works with the client. I am "losing" some time on the items.otb and items.xml right now. Making a parser to convert them to a json format that will reflect my needs better (and make it easier to add custom items)
It does sound like you've thought of the architecture so far which is good. I, personally, like seeing programmers that know what they're about to write and have planned ahead.

I still don't see how the producer/consumer model is going to be implemented in this. It sounds more like a multi-threaded approach rather than a stricly producer/consumer approach. I have another question as well. How can you ensure that all of your threads will not be accessing the same resource simultaneously at some point and thus stepping into, let's say, a race condition?
 
OP
F

frozenrose

New Member
Joined
Dec 15, 2019
Messages
5
Best answers
0
Reaction score
3
I think its still looks like tfs since tfs has the asio thread/loop(producer) and the dispatcher loop(consumer).
You still need thread safety in the queues/thread communication.
Asio from boost isn't a single loop, it starts a crazy amount of thread to manage the connections. Similar for the scheduler used in the game logic. It reminds me of some projects that I had to fix at work where people were going overboard in async coding where it wasn't needed. Tfs isn't as bad as those, but I still want to try a different logic and see how it performs

I might go with a simple queue and a spinwait that shouldn't have to wait more than 1 frame. But I will try a channel implementation first as they are usually safe for 1 writer and 1 reader at the same time.
Post automatically merged:

It does sound like you've thought of the architecture so far which is good. I, personally, like seeing programmers that know what they're about to write and have planned ahead.

I still don't see how the producer/consumer model is going to be implemented in this. It sounds more like a multi-threaded approach rather than a stricly producer/consumer approach. I have another question as well. How can you ensure that all of your threads will not be accessing the same resource simultaneously at some point and thus stepping into, let's say, a race condition?
Yes, I played around with TFS a bit and it is a great project. I just like thinking out of the box and trying new approaches. I had it on the back of my head for a bit, my subconscious came up with a lot of different ways of implementing it.

Well this is what the Producer/Consumer model is, either 2 processes or 2 threads, 1 that produce data and the other one that consume. My first thread listening to sockets, parsing and adding the data to a "queue" that will be consumed and processed by the other thread. (Or at least this is what we use it for in software engineering)
Post automatically merged:

I am very curious about how the client handle "transaction messages". If someone played around with it, I'd be interested to get some information. Is it only handling specific packets in transaction or is it handling any packets following each others ? I will eventually get to the point where I can test that, but intels are always appreciated :)
 
Last edited:

Delirium

OTLand veteran
Staff member
Global Moderator
Joined
May 28, 2007
Messages
3,348
Best answers
1
Reaction score
231
Location
Athens, Greece
Asio from boost isn't a single loop, it starts a crazy amount of thread to manage the connections. Similar for the scheduler used in the game logic. It reminds me of some projects that I had to fix at work where people were going overboard in async coding where it wasn't needed. Tfs isn't as bad as those, but I still want to try a different logic and see how it performs

I might go with a simple queue and a spinwait that shouldn't have to wait more than 1 frame. But I will try a channel implementation first as they are usually safe for 1 writer and 1 reader at the same time.
Post automatically merged:



Yes, I played around with TFS a bit and it is a great project. I just like thinking out of the box and trying new approaches. I had it on the back of my head for a bit, my subconscious came up with a lot of different ways of implementing it.

Well this is what the Producer/Consumer model is, either 2 processes or 2 threads, 1 that produce data and the other one that consume. My first thread listening to sockets, parsing and adding the data to a "queue" that will be consumed and processed by the other thread. (Or at least this is what we use it for in software engineering)
Post automatically merged:

I am very curious about how the client handle "transaction messages". If someone played around with it, I'd be interested to get some information. Is it only handling specific packets in transaction or is it handling any packets following each others ? I will eventually get to the point where I can test that, but intels are always appreciated :)
In that case you should take a look at the Dataflow (Task Parallel Library or TPL) library of Microsoft's. It provides the necessary tools to build producers, transformers, broadcasters, consumers etc. Should be really helpful in your case.
 

jlnunez89

a.k.a BurnMc
Premium User
Joined
Jul 5, 2007
Messages
155
Best answers
0
Reaction score
51
Location
Seattle, WA
Whuut, I didn’t realize my project was featured in this thread until now.

(On mobile so shitty format)

This fibula project does look very clean, yes. It is not my type of coding though, so I won't join on the effort. It uses a lot of external ressources which creates a bunch of overhead in their network communication and such. They also use an approach that put enormous strain on the garbage collector, which become a real issue in project like a game server that runs for extended periods of time.

Not sure what you mean by “a lot of external resources” in the network communication, but it sounds like you didn’t spend too much time exploring the code, as the only external piece was the encryption part (which now I have given a second pass and added some industry accepted approach).

On you second point tho, not sure what the problem is with using the garbage collector the way it was designed to be used...

It sounds like you are scared to use the GC and/or you plan to be closely managing the memory, to which I would say you probably don’t want to be using a managed language in the first place.

There seems to be a common misconception where only because someone uses the garbage collector the performance must drop to the floor... in reality you can create as as many gen1 items as you want as long as they are able to be collected freely or they turn gen2 and (ew) gen3, which actually makes you take a performance hit.

I’ve load tested my service and I’m pretty happy with the virtually no memory it uses and less than 1% cpu it consumes, but granted this is all relativeto what hardware you run stuff on. Still I’m pretty sure it won’t be bad at all even after running for “extended periods of time”, since I’ve done similar work on a crypto arbitrage trading bot that creates thousands of objects and collects them really fast, and can run for maybe a month (I think I’ve left it unattended for almost 3 weeks without problems).

Im curious to see what you came up with as your producer/consumer design goes. My design is actually pretty much that: a central priority queue that takes operations that happen in the game, fired as time elapses by a single consumer, and produced by either incoming requests (i. e. the player’s client) or other game components.

With that I can get away without a central “creature thinking” or similar loop, and rather have mini state machines (eg. per creature), with operations creating/expediting/cancelling more operations; think a two method recursion with delays, semaphored by the single consumer of the queue.

One last note is that I didn’t go with Microsoft’s TPL mainly because I needed to control task expedition and task cancellation. I know TPL supports cancellation but then you actually have to keep track of the token that each task observes and cancel the source of it... which I didn’t feel like doing. Also there is some real overhead that the TPL framework has in terms of scheduling and task running (and it must, to support such broad and generic scenarios it was designed for).

ninja edit: typos
 

averatec

Intermediate OT User
Joined
Jun 1, 2007
Messages
2,231
Best answers
3
Reaction score
147
Location
Poland
@jlnunez89

One question, does your project support LUA? Sorry, I didn't check your github yet.
 

jlnunez89

a.k.a BurnMc
Premium User
Joined
Jul 5, 2007
Messages
155
Best answers
0
Reaction score
51
Location
Seattle, WA
Just wanted to reiterate my point on not fearing using the GC as intended: spawned about 50 mino archers and some cyclops (all path-finding and attacking), then ran around...

ninja edit: seems imgur loads nicely embedded, left the longer gif.
 

Attachments

Ezzz

Developer
Joined
Feb 26, 2010
Messages
1,738
Best answers
2
Reaction score
470
Location
Caracas, Venezuela
Just wanted to reiterate my point on not fearing using the GC as intended: spawned about 50 mino archers and some cyclops (all path-finding and attacking), then ran around...

ninja edit: seems imgur loads nicely embedded, left the longer gif.
0.5% average CPU usage for 1 player? It's still too much as to what we can do with C++ on the same circumstances, imagine 100 players and monsters also retargeting many times and finding paths for different players at a time and over 300 players doing stuff? That's overkill on the CPU.

Imagine 300 and plus average players, and about 300 or more active monsters at a time, including player actions from moving objects, removing objects, etc ... How would the performance be? Also, isn't an object pool a better approach in C#? CipSoft uses object pools for everything but creatures.
 
Top