[Discussion] TFS direction (to do list/new milestones)

[Night Wolf]

oh, so you mean actually having collaborators as GMs and realoding sv constantly? Now I get it, I was thinking about a test server but only with players

 

[Peonso]

And if it was my call(i would do this for a brand new engine), the protocol version and the targeted game features should be 7.7 and not 10.98.

What ever got the community fresh was the existence of pirate 1:1 wanna be copies of current Tibia content, that where most of the playerbase are. Everytime there was a big difference of main otserver branch and current Tibia content things went cold.

 

[Lyu]

@Lyu your server has one of the best casts out there, did you bought it or done yourself? mind sharing a few insights?

@Night Wolf I had created my own cast system with Livestream and Spectator class, but I never used it, because the current one I use on my server is stable. It is the same cast as otx2, but I converted to tfs 1.x, improved a lot and added more features created by me. maybe I will release my own version so that we can improve it together and maybe become official.

see this preview!

#ifndef __LIVESTREAM__
#define __LIVESTREAM__

#include "protocolgame.h"

class Livestream;
class Spectator;

static constexpr uint16_t MAX_SPECTATORS_BY_LIVESTREAM = 50;
static constexpr uint16_t MAX_LIVESTREAMS = 50;

using SpectatorList = std::vector<Spectator*>;
using LiveList = std::vector<Livestream*>;
using SpectatorBanVec = std::vector<uint32_t>;

enum LiveBan {
    LIVESTREAM_BAN_NOT_FOUND,
    LIVESTREAM_BAN_SUCCESS,
    LIVESTREAM_BAN_ALREADY_BAN
};

class Livestream final
{
    public:
        Livestream(ProtocolGame_ptr client, Player* player);
        Player* getPlayer() const { return player; }

        SpectatorList &getSpectators() { return spectators; }

        bool isBroadcasting() const { return broadcasting; }
        void setBroadcast(bool broadcast) { broadcasting = broadcast; }

        const std::string& getPassword() const { return password; }
        void setPassword(std::string newPass) { password = newPass; }

        void muteSpectator(std::string spectatorName, bool mute);
        LiveBan banSpectator(std::string spectatorName);

        void stop();

        void registerLive();
        void unregisterLive();

        static const LiveList& getLiveList() { return liveList; }
        bool isBanished(uint32_t ip);
        bool isAvailable();

    private:
        SpectatorList spectators;
        SpectatorBanVec IPBans;

        ProtocolGame_ptr client = nullptr;
        Player* player = nullptr;
        bool broadcasting = false;
        std::string password;

        static LiveList liveList;
};

 

[Lessaire]

If you want a Continuous Playtesting Server for this MMO engine:

• You need an actual inline AAC. *
• You need a purpose built map that provides rapid testing scenario access.
• You need purpose built ACL state toggling, so that player can escalate privileges to gameworld administrator when they need to spawn constructs that warrant further examination, or manually edit player state variables.
• If GA facilities typically include access to commands that may interact with the OS userspace underneath the network daemon, they must be sandboxed or disabled.
• You need to take care to disregard all illusions of gameplay and grind. Do not waste testers time.
• Much like how a farmer only lets their kids name the animals things like "sausage link" and "chilipot", these characters are not pets, and will be eradicated at the next testing cycle, within a week at most, do not get attached. It seems so obvious, but I've seen it happen when the test cycle was two weeks because a large content update and testers got into the new subplots.
• Ideally your serious testers will approach this in a similar fashion that a computer runs a unit test, formulating scenarios that access every possible iteration of any altered or newly introduced functionality
• You will soon find out why the only people who this seriously are the ones getting paid for it

*: Not these PHP software packages calling themseleves AACs. Those are webportals, and I don't understand why they co-opted a term that was explicitly created to differentiate the concept from a webportal. And I really don't know why it stuck. An AAC is where you can create your account and character entirely within the client software. i.e. you enter 1/1 as user/pass and the server logs you in as a fake character and lets you interact with a scripted dialogue chat to make an account. Then you relog and the same dialogue has general account management tools, like making a character.

 

[zbizu]

@Lessaire

1. we can skip aac if we allow clones on 1/1, let's go warots
2. I can help build one if needed, let's go skyrim coc qasmoke
nevermind, got bored too fast. In case anyone wants to start, can use this one >link<
3. /gamemode access, let's go minecraft
4. rewrite lua chatchannel to load files, sandbox it better than it is now, redirect prints to some chatchannel. We can use znoteAAC to let selected logged in users paste their scripts (with logs ofc). Would be nice if we could also limit the sandbox's ability to kill the server with infinite loops and giant tables, but I have no idea how to do it.
5. build a vault lol, let's go fallout
6. just set save flag to 0, let's go classic tibia
7. though I agree, the players are unpredictible and can crash every potentially working script so not every test can cover that
8. sadly my last server turned out to be a huge playtest to reproduce house item duplicate bug so I can see where it goes.

 

[Lessaire]

If I stop what I'm up to currently to fabricate tooling for this, I'll be running multiple gameworlds so that up to n slots worth of open PRs can be available for testing simultaneously. Something along the lines of:
0.) spawn clean chroot
1.) preinstall makedeps
2.) git shallow clone
3.) git checkout new %autonamed% branch,
4.) git pull foreign remote
5.) makepkg PKGBUILD
6.) lease ports and request database from an overseer process
7.) generate config.lua
8.) overwrite /world with custom testing map and inject
9.) launch daemon
10.) make announcement

Permanent testers could have actual accounts that are preserved, where the baseline database snapshot includes them. But one thing I do not want to add to this mix is PHP.
I despise PHP and usually rewrite PHP dependencies of software I want to use in NodeJS and Pug.

The biggest hurdles are definitely:
a.) Creating a suitable purpose built testing environment map
b.) Generating scenarios that adequately interface with the changed components

I believe I have a workable protocol concept for collaboration on such a thing...

OpenTibia collaborative community developer testing platform suite:

• To facilitate multiple community developers providing modules containing unit tests, tooling, and scenarios:
  • A collection of these modules consisting of maps and scripts shall be merged:
  • Hereafter community developers shall be referenced as comdevs.
  • Hereafter the root module and overall project shall be referenced as inquisition.
• I will open a repo, where each comdev who wishes to participate shall:
  • Pull-request for the inclusion of their testing module.
  • The pull-request shall target these assets into a subfolder.
  • That subfolder shall carry their identity with its name:
  • Either: The comdevs username on OTLand or The comdevs username on GitHub.
  • Hereafter the comdevs testing module shall be referenced as testbed.
• Each testbed shall contain assets consisting of:
  • An OTBM map file and it's adjunct metadata XML files.
  • A single lua script file, conforming to the provided specification.
  • A JSON file containing both the GitHub and OTLand identities of the comdev.
• Each map shall be precisely a given set of dimensions, to be determined, to facilitate programmatic stitching.
• Each map shall contain a statically located room, to be determined, to act as global entrypoint for the comdev testbed.
• It is self evident such testbeds will require their own scripts, so arrangements shall be made:
• The inquisition shall allow for the inclusion of exactly one Lua script file foreign to the native TFS base dataset.
• This single Lua script file shall hereafter be referenced as the rootscript.
• The rootscript shall be the collective merger of the comdev provided testbed script assets.
• The foundation of the rootscript shall consist of a header block providing facilities for the testbed script assets.
• The facilities provided shall conform to these specifications:
  • There shall be exactly one global variable foreign to the native TFS base dataset.
  • This header block shall harbor this global variable
  • This global variable shall be a table to act as the inquisition script namespace.
  • This global variable should be called PROVING_GROUNDS.
• The comdev testbed script assets will be appended to the rootscript after the header block in sequence.
• To facilitate this collaborative globbing of Lua, the testbed script assets shall conform to these specifications:
  • The testbed provided script assets shall be in the form of an immediately invoked function expression (IIFE)
  • The invoked function expression closure shall be decorated with the identity of the comdev.
  • Hereafter the contents of the testbed function expression closure shall be referenced as proof.
  • The proof shall consist entirely of TFS API calls, standard Lua 5.1, and RaptorJIT provided bit lib.
  • Modification of XML indices shall not be allowed. The proof must use Revscriptsys facilities.
  • The proof shall namespace itself under the inquisition script namespace.
  • The proof should further namespace itself under a variable matching the comdev identity.

I have more but I've been at this a while.
But the general idea is you can provide a test suite for your PR to facilitate it's merger.

 

[Don Daniello]

The idea of expanding a map to feature better testing and also have a database filled with characters, houses, guilds, etc, for testing purposes has been floating around and I think they are both worthwhile. Is there anybody willing to help create these?

 

[Discovery]

If I stop what I'm up to currently to fabricate tooling for this, I'll be running multiple gameworlds so that up to n slots worth of open PRs can be available for testing simultaneously. Something along the lines of:
0.) spawn clean chroot
1.) preinstall makedeps
2.) git shallow clone
3.) git checkout new %autonamed% branch,
4.) git pull foreign remote
5.) makepkg PKGBUILD
6.) lease ports and request database from an overseer process
7.) generate config.lua
8.) overwrite /world with custom testing map and inject
9.) launch daemon
10.) make announcement

Permanent testers could have actual accounts that are preserved, where the baseline database snapshot includes them. But one thing I do not want to add to this mix is PHP.
I despise PHP and usually rewrite PHP dependencies of software I want to use in NodeJS and Pug.

The biggest hurdles are definitely:
a.) Creating a suitable purpose built testing environment map
b.) Generating scenarios that adequately interface with the changed components

I believe I have a workable protocol concept for collaboration on such a thing...

OpenTibia collaborative community developer testing platform suite:

• To facilitate multiple community developers providing modules containing unit tests, tooling, and scenarios:
  • A collection of these modules consisting of maps and scripts shall be merged:
  • Hereafter community developers shall be referenced as comdevs.
  • Hereafter the root module and overall project shall be referenced as inquisition.
• I will open a repo, where each comdev who wishes to participate shall:
  • Pull-request for the inclusion of their testing module.
  • The pull-request shall target these assets into a subfolder.
  • That subfolder shall carry their identity with its name:
  • Either: The comdevs username on OTLand or The comdevs username on GitHub.
  • Hereafter the comdevs testing module shall be referenced as testbed.
• Each testbed shall contain assets consisting of:
  • An OTBM map file and it's adjunct metadata XML files.
  • A single lua script file, conforming to the provided specification.
  • A JSON file containing both the GitHub and OTLand identities of the comdev.
• Each map shall be precisely a given set of dimensions, to be determined, to facilitate programmatic stitching.
• Each map shall contain a statically located room, to be determined, to act as global entrypoint for the comdev testbed.
• It is self evident such testbeds will require their own scripts, so arrangements shall be made:
• The inquisition shall allow for the inclusion of exactly one Lua script file foreign to the native TFS base dataset.
• This single Lua script file shall hereafter be referenced as the rootscript.
• The rootscript shall be the collective merger of the comdev provided testbed script assets.
• The foundation of the rootscript shall consist of a header block providing facilities for the testbed script assets.
• The facilities provided shall conform to these specifications:
  • There shall be exactly one global variable foreign to the native TFS base dataset.
  • This header block shall harbor this global variable
  • This global variable shall be a table to act as the inquisition script namespace.
  • This global variable should be called PROVING_GROUNDS.
• The comdev testbed script assets will be appended to the rootscript after the header block in sequence.
• To facilitate this collaborative globbing of Lua, the testbed script assets shall conform to these specifications:
  • The testbed provided script assets shall be in the form of an immediately invoked function expression (IIFE)
  • The invoked function expression closure shall be decorated with the identity of the comdev.
  • Hereafter the contents of the testbed function expression closure shall be referenced as proof.
  • The proof shall consist entirely of TFS API calls, standard Lua 5.1, and RaptorJIT provided bit lib.
  • Modification of XML indices shall not be allowed. The proof must use Revscriptsys facilities.
  • The proof shall namespace itself under the inquisition script namespace.
  • The proof should further namespace itself under a variable matching the comdev identity.

I have more but I've been at this a while.
But the general idea is you can provide a test suite for your PR to facilitate it's merger.

I`m finishing that kind of server (multi-world with login separated, node.js), I`ll put on git soon

And can help developing the php execution hehe

Lets think about a web interface with map uploading(with limits) and script testing by custom tool shell;

 

[Leo32]

@Yamaken I dont think you see the value in the client feature parity.
The reason the otservbr fork is steamrolling ahead is because new things in Tibia keep pushing them (there is an objective driver that fuels dev).
Current 12.X support is done in a way where it doesn't care what client you use 10.98 or 12.x+. There are no downsides to this.

We are stale and stagnant in 10.98 due to some puritanical idea of P E R F E C T I O N and because we tied ourselves to OTC to get away from protocol parity - and now look at us. This seemed to have turned out to be a bad move.

There is nothing virtuous about a pristine corpse of a codebase, this just hands the future of opentibia to the otservbr or otx team.

Trying is failing, then making corrections, then succeeding.
You seem to champion the idea of not even trying, which is braindead.

 

[Mr. Greyshade]

I would rather have a good & clean stable distro with good performance, than the abomination that OTX and otservbr is.

 

[zbizu]

I would rather have a good & clean stable distro with good performance, than the abomination that OTX and otservbr is.

They gave us the code examples basically doing half of the job for us. Now we only have to fix/improve it and it can be merged into tfs. Be more grateful that someone actually did something when tfs repo was stale for years.

 

[Leo32]

I would rather have a good & clean stable distro with good performance, than the abomination that OTX and otservbr is.

I would rather have a good & clean stable distro play real tibia with good performance, than the abomination that OTX and otservbr opentibia is.

Define what you actually mean, cause your train of logic is just anti-ot with extra steps.
If all you think about is P E R F O R M A N C E then you're asking for TFS to stand still and go nowhere.
TFS is very well optimized and changes like that are far and few between.

Edit:

TFS could be much more optimized, there's a reason why the majority of all big servers has paid for kondra's TFS optimizations. Gudam gudam listed most systems that have a big impact on the performance.

Those changes are all tethered to the client itself, the way otc parses sprites or w/e isn't optimal this is all unrelated to TFS.
I don't think you get the whole picture here mate.

 

[Mr. Greyshade]

I would rather have a good & clean stable distro play real tibia with good performance, than the abomination that OTX and otservbr opentibia is.

Define what you actually mean, cause your train of logic is just anti-ot with extra steps.
If all you think about is P E R F O R M A N C E then you're asking for TFS to stand still and go nowhere.
TFS is very well optimized and changes like that are far and few between.

TFS could be much more optimized, there's a reason why the majority of all big servers has paid for kondra's TFS optimizations. Gudam gudam listed most systems that have a big impact on the performance.

 

[Lessaire]

And can help developing the php execution hehe

Why does a testing server need a webportal though? It seems like an added complication. That doesn't offer anything that can't be provided in some other fashion. I would be ideal to not have to expose an httpd for this at all, but if it is necessary, I'd like it to be just 1 for the whole platform.

The idea of expanding a map to feature better testing and also have a database filled with characters, houses, guilds, etc, for testing purposes has been floating around and I think they are both worthwhile. Is there anybody willing to help create these?

I'm willing to spare the server power to run something like this, and do so long term.

I really think have some sort of base map that offers teleporters to specific spots, and stitching together contributed maps like this:

would be the ideal way each dev could work on their map without having to await function other_dev()

The only trick is they have to use a single table to store all the Position variables used by their scripts, which the Proving Grounds will modify relative to their actual location in a given run. That's why a static map size has to be used. Honestly, having a single central source of truth for hardcoded positions is a best practice you should be doing anyway, so this should be a very minor request.

The other consideration is the scripts used to operate the test bed should not pollute the global namespace or interfere. It is TFS itself that is being tested, and side effects of the testbeds themselves should be nil.

In an actual test run, you'd spawn in the "control room" in core, and zip over to the teleporters pad and poof! You are in your own testing map with your scripts running.

My design idea here is this should be as automated as possible, because the one thing I know I despise is getting a wild hair to work on something at 3am but the process requires waiting on someone who won't be up for another 5 hours. So having a bot-managed GitHub repo that auto accepts PRs by devs for assets going into their own folder, which sends a message to another bot that manages deployment of the test worlds, which pulls into those changes automatic seems ideal. If the that PR is by a dev with an open PR on TFS, the relevant world be scheduled for reboot immediately. From PR to rebooted test world in under 15 minutes, or your pizza is free.

It would be a lot of work to initially create this kind of tooling but the only way to manage unit testing on a large project and not get critically burnt out is to spend your effort on automation, not managing other people. It's a lesson I hope you never have to learn the hard way.

Just my thoughts on the topic. Throwing ideas at the wall and seeing what sticks.

 

[Mr. Greyshade]

Those changes are all tethered to the client itself, the way otc parses sprites or w/e isn't optimal this is all unrelated to TFS.
I don't think you get the whole picture here mate.

Are you pretending? It's not related to OTC at all. Go ahead and read @gudan garam's first post in this thread, or actually take a peak into the tfs repository and you will see plenty of things that still need to be optimized. The reason why I mentioned kondra, is because other than just fixing otclient, he has also optimized TFS - and those changes are not related to otclient in any way.

 

[Leo32]

You're over-complicating things @Lessaire.
What he's asking for a testing environment, a datapack with a map and tools to assist in testing and scripting - Its a solid idea imo.

Something similar to this:

[TFS 1.X] Flash Client + ACC Maker (Dev Environment)

I finished cleaning up and containerizing my flash client website for release: https://github.com/Leo32onGIT/forgottenserver-flash-accmaker This is designed to work with this branch of TFS: https://github.com/Leo32onGIT/forgottenserver/tree/10.79-flash-dev Downgraded from 10.98 to 10.79 (crit...

otland.net

GitHub - Leo32onGIT/forgottenserver at 10.79-test-env

A free and open-source MMORPG server emulator written in C++ - GitHub - Leo32onGIT/forgottenserver at 10.79-test-env

github.com

but not flash

Are you pretending? It's not related to OTC at all. Go ahead and read @gudan garam's first post in this thread, or actually take a peak into the tfs repository and you will see plenty of things that still need to be optimized. The reason why I mentioned kondra, is because other than just fixing otclient, he has also optimized TFS - and those changes are not related to otclient in any way.

Adding network compression, parsing sprites improvement blah blah blah are all centered around the client.
Most other optimizations are so minor I don't understand what you're going on about.

Why are you fetishing P E F O R M A N C E? how about you suggest specific changes? do you actually understand the codebase or are you just a parrot?
How about you run a compare on the otcv8/tfs fork and fabians fork and put them into a pull request for the TFS repo?

You don't actually make any valid arguments, this is the second thread I've seen you in where you autistically screech about performance while contributing nothing.

 

[Lessaire]

You're over-complicating things Lessaire.

If it's so simple, why are there so many PRs awaiting testing?

WASTED_RESOURCES = 0; -- A tally of the total wasted human lifespan

function reality(people, time) -- A simple function of reality
  -- time: the amount of minutes each person spent
  -- people: the number of people that did this
  local truth = 'The current tally of wasted human life resources is %s minutes'

  WASTED_RESOURCES = WASTED_RESOURCES + (time * people)
  return string.format(truth, WASTED_RESOURCES)
end

I think you're missing the critical piece of the human element puzzle that makes what I'm proposing absolutely necessary. It's one thing for a contributor to have their test world they hacked out their feature branch on. But PR's sit on code review. Lets presume at least 3 testers per PR for a good validation. Now you're asking mulitple 3rd parties to each spend their own free time pulling the branch from the foreign remote reality(3, 5) then compiling it reality(3, 5) then copying over the binary to their testing server reality(3, 2) perhaps datapack changes too reality(3, 2) and then restarting their testing server reality(3, 1) and then logging in reality(3, 1)

Then the wicked math, mulitplied by PRs to get through that may require testing reality(3, WASTED_RESOURCES * (60 -1)) And this all before time spent testing even begins. And just for the currently pending changes, not predicting the future. And that compile time is presuming best-conditions of Linux devs using ccache. It gets steeply worse outside that.

Imagine living in a world where to test changes on TFS, there is a server somewhere the changes are already being acquired, built, and deployed and all you have to do is login... And you don't even have to test things alone

If that's a terrible proposition I guess I'll get back to my own pet projects.

 

[Leo32]

@Lessaire I understand what you're saying but having a live CI environment has hosting costs and relies on a maintainer.
I don't see that being realistic, just my opinion tho.

having well thought out test maps/test talkactions in the TFS repo itself or a testing branch that is stream-lined to setup and use seems more viable.

 

[hellboy]

@Lessaire @Leo32

Maybe use docker-compose? With preconfigured map, accounts, database for testing purposes?

 

[Lessaire]

has hosting costs and relies on a maintainer.

I really hate airing dirty laundry...
See this screenshot here. The 22M means 22 months uptime. That first blurred out name is a heritage/ancestry project my mom is the newsletter editor for. This org expected me to pay their membership fee for the privilege of hosting their website for them, bringing the HTML4(!) forward into the modern era, and keeping it updated for them. Despite telling them to fuck off and find someone else who was willing to put up with that shit, I kept the service running because it hosts the archive of the old newsletters, now under a subdomain.

That one the bottom? That's the blog of someone who literally tried to murder me, because they couldn't take being called out on their terrible attitude. Grabbed the steering wheel of my car and yanked hard jerking the car into the middle lane of the freeway, colliding with a minivan at 65 miles per hour, resulting in a crazy wreck and my lovely Impala being totaled. And I got it all on dash cam. And I got video of them being yanked out of my car and arrested. And I still didn't delete their shit. Don't fucking talk to me about data retention or service longevity.

I wouldn't have offered if I couldn't actually spare the resources.
...Though I really should get a move on finishing decommissioning that server. I'll hate losing that mad uptime cred though.

Maybe use docker-compose? With preconfigured map, accounts, database for testing purposes?

I'm not averse to hosting a Podman cluster. But I sure ain't building it. And I sure ain't hosting Docker.