• There is NO official Otland's Discord server and NO official Otland's server list. The Otland's Staff does not manage any Discord server or server list. Moderators or administrator of any Discord server or server lists have NO connection to the Otland's Staff. Do not get scammed!

Source's custom OT game engine (TypeScriptFTW)

just courious did you performed any load tests against your server? If yes which tools and techniques did You used?
Hey. Nah, not really, except after I made PyOTC I logged into my server with 3000 players. This almost maxed out a single CPU core for the server.
Keep in mind these players don't do anything though, they just turn every 30 seconds to not get disconnected, compared to actually alive players, so my server is probably not more optimized than TFS just because it can handle 3000 players - this is 3000 idle only players.

Some good tools to use is Valgrind + KCachegrind to check server performance + check for memory leaks, causes of segfaults, and probably more.
Beyond that I also check the performance of code inside the server code (internal tests) by using timers that tells me how many µs are used for each function call (removed after it's been checked).
This means I know exactly how much actually or potentially expensive operations cost, and so there's no need for load testing.
Cheers.
 
Update: I'm planning on maybe starting to work on this project again by new year.

For now I've implemented multiprocess epoll in its own library which this server (TaoEngine) should be able to use.
If successfully implemented it will be able to handle more concurrent connections than TFS, especially if you disable XTEA encryption (at least for anything other than say chat messages) you will see a huge boost with this server compared to TFS in the ability of handling large amounts of connections (XTEA is mildly/moderately "expensive" so without it you will see a larger boost than with it, but you'll see a big boost either way).

In fact I wrote a web server recently using this module and it handles more than 200% the number of connections than Nginx which is known to be one of the fastest public web servers available, when using a simple naive static page implementation, and still around 50% more after full basic web server implementation (reads and caches HTML files, indexing, realpath(...), status codes 404, 400, 200, etc).

Merry Xmas and a happy new year everyone.
 
Update: I'm planning on maybe starting to work on this project again by new year.

For now I've implemented multiprocess epoll in its own library which this server (TaoEngine) should be able to use.
If successfully implemented it will be able to handle more concurrent connections than TFS, especially if you disable XTEA encryption (at least for anything other than say chat messages) you will see a huge boost with this server compared to TFS in the ability of handling large amounts of connections (XTEA is mildly/moderately "expensive" so without it you will see a larger boost than with it, but you'll see a big boost either way).

In fact I wrote a web server recently using this module and it handles more than 200% the number of connections than Nginx which is known to be one of the fastest public web servers available, when using a simple naive static page implementation, and still around 50% more after full basic web server implementation (reads and caches HTML files, indexing, realpath(...), status codes 404, 400, 200, etc).

Merry Xmas and a happy new year everyone.
Glad to see you back m8
 
Been working 'hard' on TaoEngine again since Jan the 2nd.

Key features / fixes:
  • Added automatic reloading of real tibia cylinders like in Realots every minute, going from order Z to X to Y, only refreshing Refresh tiles, does not refresh when players nearby, and with many optimizations to reduce CPU usage, for example skipping sectors when they do not contain a Refresh tile, but still counting it such that the pace is still correct (4H48M per full map cycle), note: no visual lag at all running at speed 65535 on localhost even while the largest chunks of tiles are being updated (15000+ at once) ;-)
  • Support for running anti-cheat LSTM (AI neural net) on CPU (multicore)
  • Characters are automatically saved upon logout now, including death -> lose inventory, skills, exp/level, magic level, and gets sent to temple upon login
  • Server now prints out the correct number of MoveUse scripts used in each category
  • Many bug fixes and improvements in C++ and otherwise

Changelogs:
  • Fixed alevo bug (insta-deletes from ground and such)
  • Fixed "can't add item to container unless secondary"
  • Support for running LSTM on CPU (determined automatically like before, but no runtime error)
  • Enable extended opcodes for proto != 770 regardless of OS
  • Fixed elemental fields landing on stairs, holes, etc + simplified by adding cr.damage to addField instead
  • Fixed "wolf rarely drops anything" (general monster loot) bug
  • Added a bit more info when rare "creature not found on tile" fatalError occurs
  • Fixed updateConditions (fire etc conditions for monsters) by iterating over all creatures np (CPU remains at 1% usage of a single thread for some reason now) to update all creature conditions!! :O
  • Fixed addItemPy attributes lingering when e.g. fishing
  • Added bool worlditem and bool !() worlditem
  • Simplified moveItem inventory restrictions and added actor
  • Started messing with trying to fix merge items in inventory
  • Fixed merge items on ground (should not merge if count > 100) and in inventories
  • Added poofs back when attacked (not armor hit only)
  • Added health/mana up the same amount as max health/mana when leveling up
  • Alevo container
  • world2.moveItem(...) improvement
  • world::deleteItem and stuff, moveItem is still work in progress
  • New world::deleteItem(...) function, both world2 and world setItem now uses deleteItem + addItem
  • Bug: dead bodies gets added to stackpos 0 for some reason
  • Easy to fix monster corpse added to stackpos 0 (stackpos was set to 0 lol)
  • Fixed so GMs can cast all spells
  • Fixed trap (or misc. functions related to it) for now, now need to add parsing of !IsPeaceful, then add Reverse condition for all ME conditions
  • Fixed armor formula slightly (armor 1 should count)
  • Fixed trap completely (by fixing MoveUse parsing stuff)
  • Fixed eating apples XD (bunch of internal changes)
  • Fixed MoveuseCounts
  • Fixed ladders (minus sign in front of numbers not being parsed)
  • Fixed all commands require pw
  • Players are now saved on logout
  • Fixed death save incl;
  • Exp loss value (was multiplied by exp rate)
  • Losing inventory on death (was bugged)
  • Losing skill tries
  • Blah blah cylinders blah blah

Here's some interesting facts I learned this time:
  • Did you know Real Tibia / Realots trap causes physical damage to monsters RELATIVE to its armor value which is RNG and therefor the damage output if armor >= 1 is random (30 damage ONLY against 0 armor monsters)?
  • Real Tibia / Realots continuously reloads chunks of the map every minute; the number of tiles are few enough that you won't notice the reloading as a player (or as a GM, even with max speed @ localhost during the biggest map reloads)

I wish I could be bothered to upload some screenshots or recordings, but no, I can't :D

Just to re-iterate what this server got so far:
  • Written from scratch: no OTServ or TFS or OTX or based on any such codebase whatsoever, but instead inspired by Realots and custom written
  • Anti-cheat AI network that can run on GPU or CPU, can also easily be divorced from the game server so it can run in parallel with game server instead of fighting over HW resources
  • 100% accurate monster spell formulas, melee, distance, wand, rod, armor, shielding and fishing formulas, NPC scripts (they don't perform most "actions" yet, but the loaded chat scripts / message responses should be 100% accurate; some NPCs perform simple actions, like curing poison and fire, healing if you're really low on health, giving you 1 HP if you cuss, etc), monster loot, monster speed, monster defense and armor formulas
  • Creative and proprietary optimized A* using my own special sauce (tldr backwards then forward pass)
  • Skills and conditions are stored in physical memory and files, meaning if you had only 1 tick left until food regeneration and then log out, this will be saved and loaded again when you login the next time so you will instantly regenerate health and mana upon login; these conditions are stored using the same USR system as Realots / Real Tibia
  • Server internally uses original Realots Objects.srv, MoveUse.dat, .NPC and .MON files, and .SEC files
  • Proprietary and creative MoveUse optimization
  • Map cylinders are automatically reloaded every minute in the same order, with the same behavior, as on Realots now, only on Refresh tiles and optimized by skipping sectors without Refresh flags
And so on.

Well, that's all I guess.

Cheers!
 
Time for another bi-weekly update ya'll.

Drastic changes has taken place this time as I decided to translate 5000 lines of my scripted code into more than 5000 lines of standard C++ using AI.

I have since fixed all compiler errors except for some I found out is a bug in the programming language I use, meaning I have not yet been able to test the new AI translated server where 35% of the codebase is now written in pure standard C++.
This means I don't know if my server works at all at this point in time and won't know until the developers fixed these bugs, but that doesn't stop me from coding :)
PS: Most likely fixing everything will take weeks, maybe a month or two, or will not be solvable at all, so lets hope for the best but be prepared for the worst.

Here's what I'm currently working on: looping over all active creatures in gameLoop (aka advanceGame in realots) for every every single iteration of the game loop and deciding what actions to perform depending on various conditions.
nTlgfTT.png


Previously this was done using async calls, but now players autowalks, NPC behavior, and monster behavior should be run at 50Hz by default like Realots (more or less) did it, that is behavior of all active creatures will be checked 50 times, by default, for every second on the clock.

Rewriting creatures (among other classes) in C++ will have important benefits, such as:
  • Being able to overload functions
  • For C++ coders to be able to work on it if the code is ever released, rather than having to learn an arcaic (and ugly) language of my choice
  • May let us introduce more multithreading
  • May improve generation and compilation time dramatically (has already improved generation time by 30%, from 9-10 to 6.1-6.5 seconds)
  • Potential other benefits I haven't recognized or realized yet

This is what I have been working on in the last 5 or so days.

Before that I fixed the following problems:
  • Added "log other player out of account if new player is trying to login to same char"
  • /refresh command to force either refresh single sector or all sectors
  • Outfit spells now works again
  • Added "resume cylinder refreshing time-position based on nr of minutes since server's reboot time" aka cylinder sync based on current time regardless if server crashes or force restarts or other circumstances
  • Many other bug fixes

Amazing stuff m8! Doing great things!!!
Thanks buddy :D
 
Hey everyone.

It's time for another bi-weekly update.

Changes this time:
  • Successfully compiled the server for the first time after translating over 10000 lines to C++ using Claude 3.5 :) (I said 5k the first time, I was wrong, it was actually over 10k lines, about 50% of the server codebase)
  • Fixed all bugs involved with booting the server, logging in and walking around (server was heavily bugged after first compilations including segfaults)
  • Added NPC dialogue back without async (using gameLoop aka realots advanceGame -> creature->doActions()) with proper delays and leaving target if target isn't responding
  • Added pretty much full monster behavior in doActions which is run @ 50Hz by default including movement, spells, changing targets (only if target is gone for now - I just realized I haven't added changing of targets otherwise yet), de-activation of monsters if no targets around
  • Perfected monster spell behavior including when to cast spells and the infamous double spell bug on realots which I have figured out basically how works now and can easily replicate
  • Perfected slaves (summons) follow behavior with the exact same rhytm and delays as realots
  • Fixed two segfaults introduced after translation which I have called "logout segfault" and "summons segfault", both involving altering our std::vector<basecreature*> activeCreatures while iterating over it, which I have since learned not to do ;)
  • Fixed many other bugs

This bi-weekly bugs & update video:

Thanks to everyone reading, commenting and reacting to this devlog & discussion thread :-)
 
Time for another bi-weekly update :)

This bi-weekly changes:
  • Full map saving in pure C++ (implemented this from scratch in less than 24h, was easier than I expected, should be bug-free already according to the diffs I've run)
  • Realots circle.dat parsing of file + used for spells implemented, giving perfect Realots circular spell shapes. (I thought previously I already had the formula down by dividing the radius by 1.2, but in fact after noticing a bug with UE shape and many hours of investigation, there IS NO FORMULA for Realots spell shapes, they're custom made; Euclidean, Chebyshev, Taxicab and Minkowski distance does not provide the Realots circle spell shapes, nor any other known math formula. Instead the only way to get perfect shapes is by manually creating shapes for every spell, or using circles.dat)
  • Fixed several segfaults and infinite loops by learning to use GDB (this was fun now that I'm more of an experienced developer I think)
  • Fixed line of sight; spell splash no longer go through/comes out the other side of the wall and stuff, in fact it's seemingly just like Realots line of sight now
  • Serious (about a week of) internal cleanup/fixing class methods/implement more old scripted functions in C++, added C++ header files to every CPP file so I can include any file I want without compiler errors, etc, still more to do though
  • Many other more "minor" issues, for example monsters were fleeing slowly and summons were autowalking towards master slowly, these and many others have been fixed

Other details/stats worth mentioning:
  • Map loading takes 12s on a high end $1000 budget laptop (good CPU, mediocre NVME drive, my Desktop's drive is much faster, and my Desktop's drive is not the fastest of NVME drives out there, PCI-e 3.0 instead of 4.0, among the cheaper NVME drives for a Desktop per GB, etc)
  • Map saving takes 6s on the same laptop
  • g++ standard used: c++23
  • Optimization level: -O1
  • Code generation time: 6-7s
  • Compilation time: 44s with just 1 CPU thread/core (can't be parallelized yet)
  • I've dropped Windows support for now, might be added back later (this happened a while ago)
  • Memory usage without any players cached: exactly 1.0GB (same as Realots)
  • Memory usage with all players cached (necessary for now to be able to log into player files): 1.6GB
  • Other deps: fmtlib for C++ and one or two scripting modules for the scripting language

Here's some screenshots:

I wrote some fun tools, like packet crafting from DEC numbers, map diff checker, and this spell tool trying to figure out a formula for the Realots circle spell shapes

Saved map (to drive) stats comparisons (bytes at top, lines at bot)

How it looks like when you Ctrl+C the server after boot has finished, including time elapse for saving SEC file to drive from memory

I'm also considering making a public git repository for this server with a tool I wrote that cleans all the code and just leaves the Header and empty CPP functions and methods.

It won't be very useful for running, as it won't compile, and if purely hypothetically you could compile it (you can't, because there's missing scripted code), it wouldn't do anything because the CPP files are cleared of code, BUT it's a nice way of learning a little more about my server and how to make your own server from scratch by looking at my server engine design.

Let me know if you're interested in me making the public git repo with empty code (just Header files intact and "empty"/cleared CPP files (no code, just signatures)).

Oh, and I forgot to mention, I will probably also include the map saving code (the only exception, nothing else) in the public git repo if people are interested.
Again, this code won't be useful for anyone else except for learning engineering/code designing/curiosity, etc, as the code can't even be compiled without the rest of the codebase.

Cheers!
 
Last edited:
Hey everyone. It's the 4th Wednesday of the month and you know what that means :D haha.

Changes this time:
  • Implemented loading players in C++ including loading inventories and depots + loads on 4 threads which reduces loading from old ~20 seconds to 5-6 seconds in C++ to 2.7 seconds with 4 cores
  • I will also in the future try to transition world loading from scripting to C++ using the same code I use for inventory/depot loading
  • Many new C++ functions: readFromFile, writeFromFile, fatalError in C++, loadPlayers, loadPlayer, loadAllPlayers, init, shutdown, among others
  • Added boost/stacktrace since GCC 13+ (std::stacktrace) is I believe not available yet as a official repo package for Debian based distros or Debian-backports at least
  • Accompanying debug and fatalError functions prints the stacktrace in C++ with an optional (for debug) logged message, optional skip parameter, and optional max_depth parameter
  • Saving players (untested)
  • Added cpplog in C++ which instead of using scripted logfiles uses FILE pointers (fwrite and fsync), which should hopefully be much faster.
  • Some other internal changes which fixes various bugs / still changing calls from scripting to new code in C++
  • Some mention of various bug fixes: Things like using blueberry bushes and using levers (Create, Change, CreateOnMap, DeleteOnMap) now no longer relies on things like gameSendUpdateTile (dirty solution), but instead on gameSendAddObject and gameSendRemoveObject. Just yesterday blueberry bushes would disappear when you clicked them, light poles would disappear underground and relogging would cause segfault, rookgaard bridge lever would crash client, and anything walking on rookgaard bridge would have a very very long walking delay (0xFFFF Waypoint, by comparison approximately the same as a level 300 dropping down to speed 1, which no monster or creature has on Realots)

It took me a week (or at least 5 days) to implement inventory/depot loading in C++, and less than 1 hour (probably a few minutes) to add multithreading to it, haha.

Since nobody commented or requested anything I've postponed uploading the C++ skeleton of the server to github.

Cheers!
 
Last edited:
Hey everyone. It's the 4th Wednesday of the month and you know what that means :D haha.

Changes this time:
  • Implemented loading players in C++ including loading inventories and depots + loads on 4 threads which reduces loading from old ~20 seconds to 5-6 seconds in C++ to 2.7 seconds with 4 cores
  • I will also in the future try to transition world loading from scripting to C++ using the same code I use for inventory/depot loading
  • Many new C++ functions: readFromFile, writeFromFile, fatalError in C++, loadPlayers, loadPlayer, loadAllPlayers, init, shutdown, among others
  • Added boost/stacktrace since GCC 13+ (std::stacktrace) is I believe not available yet as a official repo package for Debian based distros or Debian-backports at least
  • Accompanying debug and fatalError functions prints the stacktrace in C++ with an optional (for debug) logged message, optional skip parameter, and optional max_depth parameter
  • Saving players (untested)
  • Added cpplog in C++ which instead of using scripted logfiles uses FILE pointers (fwrite and fsync), which should hopefully be much faster.
  • Some other internal changes which fixes various bugs / still changing calls from scripting to new code in C++
  • Some mention of various bug fixes: Things like using blueberry bushes and using levers (Create, Change, CreateOnMap, DeleteOnMap) now no longer relies on things like gameSendUpdateTile (dirty solution), but instead on gameSendAddObject and gameSendRemoveObject. Just yesterday blueberry bushes would disappear when you clicked them, light poles would disappear underground and relogging would cause segfault, rookgaard bridge lever would crash client, and anything walking on rookgaard bridge would have a very very long walking delay (0xFFFF Waypoint, by comparison approximately the same as a level 300 dropping down to speed 1, which no monster or creature has on Realots)

It took me a week (or at least 5 days) to implement inventory/depot loading in C++, and less than 1 hour (probably a few minutes) to add multithreading to it, haha.

Since nobody commented or requested anything I've postponed uploading the C++ skeleton of the server to github.

Cheers!
Yoooo this is awesome stuff.

Well i will be honest. Skeleton is pretty cool to see tbh.

For me at least it would be something that could help me further progress
 
Yoooo this is awesome stuff.

Well i will be honest. Skeleton is pretty cool to see tbh.

For me at least it would be something that could help me further progress

Thanks buddy. Since my skeleton stripping code was a bit buggy I decided to generate my own PDF instead using pre-existing open source software since I don't need this code I wrote anymore / no point spending more time developing and fixing bugs in this (skeleton stripping software).

Using pre-existing software may leak a little more than I wanted to including everything in the header files I think, and you can learn some of my optimization secrets with this document now, but oh well.

At least that might mean the document is more useful, but it's also larger than expected, at like 180 pages total.

Anyway, hope it's useful for ya, cheers!
 

Attachments

Thanks buddy. Since my skeleton stripping code was a bit buggy I decided to generate my own PDF instead using pre-existing open source software since I don't need this code I wrote anymore / no point spending more time developing and fixing bugs in this (skeleton stripping software).

Using pre-existing software may leak a little more than I wanted to including everything in the header files I think, and you can learn some of my optimization secrets with this document now, but oh well.

At least that might mean the document is more useful, but it's also larger than expected, at like 180 pages total.

Anyway, hope it's useful for ya, cheers!
how did you gnerated that pdf? i dont think you wrote it from 0 dont you?
 
Is the project available as open source code? I am currently unemployed and would be curious to try my hand at programming this server
 
Is the project available as open source code? I am currently unemployed and would be curious to try my hand at programming this server
Nou, but there's a chance someone can maybe work with me if they really want. If you don't want to np :)

So far I've been working on it all on my own, and it's like 85% done and I'm expecting alpha launch this year, so I'm happy to work with people or continue all on my own, same same for me, but I have no intentions of releasing the entire codebase anytime soon.

Some things that can be expected with launch:
  • Sound system (old buggy demo from 14 months ago - audio can be easily disabled in the menu if you don't like it)
  • Only high rates in the beginning
  • Crashes and bugs haha
  • Local cam player and recorder should be working on client side, and I will try to finish mouse playback too with data that is already sent from server
  • Server sided cam player/recorder will probably be developed later, after alpha is done
 
Nou, but there's a chance someone can maybe work with me if they really want. If you don't want to np :)

So far I've been working on it all on my own, and it's like 85% done and I'm expecting alpha launch this year, so I'm happy to work with people or continue all on my own, same same for me, but I have no intentions of releasing the entire codebase anytime soon.

Some things that can be expected with launch:
  • Sound system (old buggy demo from 14 months ago - audio can be easily disabled in the menu if you don't like it)
  • Only high rates in the beginning
  • Crashes and bugs haha
  • Local cam player and recorder should be working on client side, and I will try to finish mouse playback too with data that is already sent from server
  • Server sided cam player/recorder will probably be developed later, after alpha is done

waiting for this moment
 
Nou, but there's a chance someone can maybe work with me if they really want. If you don't want to np :)

So far I've been working on it all on my own, and it's like 85% done and I'm expecting alpha launch this year, so I'm happy to work with people or continue all on my own, same same for me, but I have no intentions of releasing the entire codebase anytime soon.

Some things that can be expected with launch:
  • Sound system (old buggy demo from 14 months ago - audio can be easily disabled in the menu if you don't like it)
  • Only high rates in the beginning
  • Crashes and bugs haha
  • Local cam player and recorder should be working on client side, and I will try to finish mouse playback too with data that is already sent from server
  • Server sided cam player/recorder will probably be developed later, after alpha is done
Hi, I have a lot of free time now but I don't want to join the team too much.... I'd rather just do some source code development when I'm bored for training purposes.

offtopic: or do you know of any old edition of OTS 7.1 that is unfinished and has a small amount of source code?

ps. I have a small question, do you have any trouble reading the code when you use the prefix ‘this’ a lot?
 
Back
Top