Removing .otb – Only Using .dat

[oen432]

he separation between CID and SID was actually a deliberate design choice/

It allows servers to reuse the same client item (CID) in multiple ways. For example, a single sword sprite could represent totally different server items, each with unique stats, names, or behaviors.

That is false. You can't have multiple items with same client id. You can have multiple items with same sprite id, yes, but not client. Client-Server item communication depends on client ids, like when you use a hotkey with an item, you are sending client id. What do you think would happen if you had two potions with different server id but same client id?

 

[Nekiro]

I did read only briefly, but I think some people confuse a sprite (raw image file) and an sprite object (client object). You can reuse sprite (the image), by adding new sprite object (client object), you cannot reuse sprite object. Client relies on it being unique, when sending actions to server, server then looks it up via sid-cid map otherwise if not using the sid, will straight look it up by client id.

When creating new item, there are 3 parts.
1. sprite, an image buffer (.spr)
2. client object (.dat)
3. server object (usually .xml and .otb)

You can reuse 1. but you can't reuse 2 and 3 is directly tied to 2.
I think I said it many times, but separate sid and cid were design choice, yes, but they were provided as an aid to cipsoft id shuffling back in the day.

@topic
I'm using client ids since years now, it's kinda annoying to find content, because everything is server id (datapack, maps) an you have to manually convert it. + various custom otb with shifted server ids aren't helping too. But I wouldn't go back to otb.

 

[Gesior.pl]

I'm using client ids since years now

Client IDs on server (like canary) or server IDs in client (like BTS)?

you have to manually convert it. + various custom otb with shifted server ids aren't helping too

You could update/recreate tool used in 8.0 times. items.otb has spriteHash attribute ( open-tibia-library/modules/constants/const.ts at master · gesior/open-tibia-library (https://github.com/gesior/open-tibia-library/blob/master/modules/constants/const.ts#L189) ) that was used to track ID changes between .dat/.otb files - when client version changed from 7.72 to 7.8 etc. - using hash of item images from .spr. IDs may be different, but sprites are often the same.

 

[Tofame]

Since I've got messages of worry from several people until now, some just pondering over the compability breaking changes it might introduce, some about there being better methods,
I reported my post to administration to edit out the links with code. Code will still be public, however it makes no sense to include it in "Discussion" post, when to many it's simply a danger.

My intensions were good, but since it might indeed be a very drastic and sudden change, with apparently better alternatives, I recommend to hold off, and discuss instead not my method, but also what can be improved in general in another solutions/approach that are possible.
(Or doing what Canary did originally, ask for their tool and choose their method that has worked for years).

 

[Galdia]

@Tofame ​

"The sprites can be reused, no matter if system is designed to only use CID or use both CID and SID, so in the end this badly designed system only"

Not true my man, the server only ever sends CIDs and UIDs. Sprite IDs etc are never processes by the server and never passed over the network.

Your claim that sprites can be reused without reusing CID is false and appears to be in line with claims made by Codinablack earlier... When pressed he explained that his 'other option' was simply to try and use UnqiueID's (which are unsustainable replacements for important reasons mentioned below)

Once again the claim is this: you can't duplicate items if you don't reuse single CIDs via multiple SIDs, thus far no evidence has been presented that goes against that (tho there has been loud unsupported claims)

I would suggest you guys are in pre design stage if you haven't worked out basic item reuse (again it's the main point and purpose of SID)

"has no downsides (0 compared to what is currently) and only benefits." To be clear there are lots of options which provide a single file' the core idea of your system is the loss of desync between CID and SID; with the main side effect just being loss of support for multiple items.

This is rather important aspect; if you can't support multiple items using this ID method then that should be written front and center on the tin.

Lastly if you guys DO HAVE a method that solves this please feel free to explain it! we've heard a dozen times here 'there is a way' but when
pressed we only get half hacks that don't actually scale up (UID values for-example are way too limited for use on every instance of all items!)

@oen432 "What do you think would happen if you had two potions with different server id but same client id?"

The client sends xyz/stack pos (uniquely identifying all entities in question), CID is not relevant for any of that; you question implies a strong lack of understanding around how SID works (indeed you seem to imply it doesn't work lmao)

My server does use potions with have identical CIDs (same as any other identical instances of any other items) it works fine (the server doesn't actually need CIDs to sync anything, xyzs / UID are used for every Protocol_ParseX function).

@Nekiro seems to be the very informed and appears to be the only one in the room who both understands and is also advocating for this idea - Nekiro seems to accept that it is only an option if your going full custom with client side file distribution etc (which might be a fair trade! but it's a big limiting factor which OP seems not entirely honest or aware about)
​

@Tofame really appreciate the latest message, your contribution is super cool and appreciated! but there are some serious trade offs and side effects and it's definitely not quite as black and white as just SIDs are dumb - might be better to say Client Oriented folk (artists etc) probably appreciate CID oriented workflows, where as Server Oriented folks (quest makers, map editors, coders) likely appreciate the freedom that the SID buffer gives you TA!​

 

[Tofame]

"The sprites can be reused, no matter if system is designed to only use CID or use both CID and SID, so in the end this badly designed system only"

I meant sprite in literal sense of it. Not their ids. I meant pixel data. It can be attached as an ID, to several "Item" objects in Object Builder.
I dont know what you mean by 'sprite ID', in TFS they are used in conjunction with 'client ID', but it's of not matter to me as I haven't affected that aspect at all with any of the changes.

Once again the claim is this: you can't duplicate items if you don't reuse single CIDs via multiple SIDs, thus far no evidence has been presented that goes against that (tho there has been loud unsupported claims)


Lastly if you guys DO HAVE a method that solves this please feel free to explain it! we've heard a dozen times here 'there is a way' but when
pressed we only get half hacks that don't actually scale up (UID values for-example are way too limited for use on every instance of all items!)

I never questioned it, so argue it over with other guys. I didn't even focus on that aspect, just on the shuffling point, as to me, even if that were to be possible I'd assume it's a very risky and actually damaging concept, that is against core IT paradigms. Idk, to which exactly, but there is a reason most games aren't in such situation like this engine, and my stance is to not ever support it myself either, I guess unfortunately for you.

My post is now of 'discussion' type, instead of a one encouraging to anything, as I requested my links to be edited and no longer appear in there (thanks mods).
If anyone feels this is risky, wrong, or better methods, I once again, advise to seek those better, mentioned here methods. I will stick with mine.

edit: Even though, I cant really pinpoint the name for what people call that weird situation above, I can give a name for the .dat and .otb situation - it's serious DRY violation. If I were to add anything new to .dat, as a developer I have to:

• Understand .dat format, and add new stuff to it
• Understand object builder, add support for this .dat change, and compile it
• Understand whole new format - .otb, to do the exact same change that I did for dat.
• Understand whole new tool that is item editor, and do the change to adapt .otb load.

What's more, I also need for any other tools to also adapt either .otb or .dat, even though the change is the same, it still takes repeating myself and duplicating the workload by serious amount.
I've heard some opinions of people on my private chats (discord/here) that it's essential to upkeep .otb and .dat. I wont ever agree to that, and this discussion just tired me out. Even if I did what's wrong, maybe breaking, it's still pushing me forward.

 

[Galdia]

@Tofame yeah good points there and well made! - For art heavy work (where your making changes to sprites and dat files) keeping CID and SID in sync is logical and clever! But for servers that are just using standard dat files it's really not a tradeoff worth making.

Since It's discussion now I'll quickly mention that my own solution has been to dump the OTB file into JSON and just read / write to that, this give me nice human readable git changes (sword damage from 300 -> 350) etc, it also solves the byte-limits keeping most item values between 0-255, and its easier to build tools for (you can vibe code a near item editor once it's all json like that in an hour or two)

Thanks again for sharing! this is a really neat trick! (even if its not always compatible with every possible workflow) and I definitely agree the OTB/DAT duplication violates the (Don't Repeat Yourself) redundancy principle, confusion and complexity breeds decay so if your not using SIDs and you are in control of CIDs then yeah! totally! ditch SIDs ;D

But if your just using a compiled server to host normal clients; then forget CID's even exist, SID's are your friend

Ta

 

[oen432]

The client sends xyz/stack pos (uniquely identifying all entities in question), CID is not relevant for any of that; you question implies a strong lack of understanding around how SID works (indeed you seem to imply it doesn't work lmao)

You have no idea what you are talking about. Check how hotkeys work, there is no position or stackpos involved because client doesn't know about every single item of given type that the player possess. That's why you can use an item with a hotkey without actually having one in inventory. Iirc you will even get an error message from the server that it's not possible to use it

 

[Galdia]

something something

Let us not get rude my good and excellent dude: I know the tibia networking protocol like the back of my hand and Let me repeat; there is no server function for handling hotkeys (there are infact surprisingly few incoming parse packet commands at-all!; you just have the core things you would need like throw, use, set-outfit, turn, move, logoff, and that's about it, a hotkey would just be a zero position use command) when you 'use' in the battle window or press on a hotkey the client tries to handle it if possible acting as if you had clicked directly on the entity in question on tile / in world, if an ID DOES have to be read it will never be actually trusted and will just be used as a 'could you please use this IF IT DOES exist in my inventory', note any any such interface WOULD be subject to a CID / SID interface limitation problem (newer Tibia protocols may indeed do something like that which would sadly be disappointingly limited but atleast still not actually broken or hackable)

All low level networking commands include both position and stack pos - the fact that you saw a server using CID is not conclusive evidence that the Tibia protocol lacks this data, as much as it was a sign that the particular server you were playing on was simply doing things insecurely, it will always be possible to process a request from the client in a way that doesn't require any trust. (tho it's extremely easy for servers to fail to do that!)

The client does ALSO SEND THE CID but it's just totally client controlled / insecure (and it's considered bad practice to ever actually use it).

I'd be very curious if you can reproduce a situation which shows a modern server acting upon a CID but looking at the Protocol.cpp code its easy to see that it should really NEVER be using it for anything (it's not trustable and its also just never needed as you also receive sufficient location information in all the relevant packet types).

I do remember there were places in older servers where decoded ItemIDs were just being passes straight off to the LUA (this might have been a source of the interesting error message you saw which seemed to ''know about item types which were no longer even in your possession''!)

Those servers were really badly written! clients data is NEVER to be trusted! (after all it can just send ANY ID it likes!! we only want to trust locations since we can look those up in our server data and we should ofcoarse still verify them as being possible / valid before use)

I have no doubt there probably are servers that do this part wrong, but it's less a sign that they have to do it that way and more just a sign that they are lazy. (the Tibia protocol itself is filled with examples like this; where once initially trusted client data was revoked but not removed, with the untrusted data still being sent right next to the useful bytes) server maintainers should keep an eye out for what data NOT to trust (eg any bytes determined by the client)

If you do find a protocol packet parsing function that clearly takes an ID but not a location (perhaps in some new fandadle version) I'd be very curious to read it! (It's definitely not a thing done in any of the older protocols and I'd be quite surprised if they ever did/do break with that tradition)

Your example is thus not a valid counter argument (indeed any argument countering would have to premise with the Idea that Tibia's protocol is fundamentally insecure, and that is simply something that that has been tested quite well for some time now)

Hope that's nice and clear!, All the best!

 

[oen432]

If you do find a protocol packet parsing function that clearly takes an ID but not a location (perhaps in some new fandadle version) I'd be very curious to read it! (It's definitely not a thing done in any of the older protocols and I'd be quite surprised if they ever did/do break with that tradition)

Are you a bot or something? I feel like I'm talking to a GPT wanna be AI.

Client sends item ID with special position

otclientv8/modules/game_actionbar/actionbar.lua at master · OTAcademy/otclientv8

Mirror of official OTClientV8. Contribute to OTAcademy/otclientv8 development by creating an account on GitHub.

github.com

otclientv8/src/client/game.cpp at master · OTAcademy/otclientv8

Mirror of official OTClientV8. Contribute to OTAcademy/otclientv8 development by creating an account on GitHub.

github.com

Server even makes a check with that position to determine if the item was used through a hotkey

forgottenserver/src/game.cpp at master · otland/forgottenserver

A free and open-source MMORPG server emulator written in C++ - otland/forgottenserver

github.com

 

[Galdia]

Insults are really not appreciated my excellent dude. Also I suspect you may associate AI with thoughtful writing <a weird complement to be sure>

You said "Check how hotkeys work, there is no position or stackpos involved" (obviously you said that without checking lol, cause here they are).

As I originally stated all the packets include position and you CAN determine everything from the packet without ever trusting the client (in this case FF,00,00 just means please find where this item is and use it, its no more abusable than an inventory packet with an ID above 11, the server just needs to check the value before actually executing)
If you really did see some server report 'cant use X' without first checking you even have X then that server was just badly coded; critically: it's NOT a bug or weakness in the Tibia protocol itself. (that was just some badly written server)

Again the reason we already kind of knew this must be true is because if the client COULD simply do things (like request to use an Item that is not actually being held) then RL tibia would be vulnerable to it (as we literally use the same protocol) yet as you know, we see no suck hacks occurring.

(also nice find btw!, this is definitely the least use of position data I've seen in any packet, but sure enough it is still there and sure enough it has a uniquely identifying value that tells the server enough info to check without having to trust the client)

All the best !

 

[Gesior.pl]

As I originally stated all the packets include position and you CAN determine everything from the packet without ever trusting the client (in this case FF,00,00 just means please find where this item is and use it, its no more abusable than an inventory packet with an ID above 11, the server just needs to check the value before actually executing)

As Oen said. Client hotkeys pass all variables empty (special values to tell server that it's hotkey, not item selected by player) except clientId of item from .dat. Only information used by server to find that item on server side is clientId translated to serverId, so there can't be 2 items which CID translate to same SID, otherwise both would point to same item on server.

Anyway, I reported all @Galdia posts as AI. I counted all posts. It's 13 KB of text without quotes in 1 day, with a lot of hallucinations.
@Galdia if you want to spam programming forums, you must learn how to quote code and that after 2 negative answers to your idea, you leave given thread or start insulting people (not "Hope that's nice and clear!, All the best!")

 

[oen432]

- it's serious DRY violation

There should be shared lib for working on these files. If all the tools, servers and clients were using it then all you would need is edit the lib and recompile rest which is easy.

 

[Erikas Kontenis]

Anyway, I reported all @Galdia posts as AI. I counted all posts. It's 13 KB of text without quotes in 1 day, with a lot of hallucinations.
@Galdia if you want to spam programming forums, you must learn how to quote code and that after 2 negative answers to your idea, you leave given thread or start insulting people (not "Hope that's nice and clear!, All the best!")

There is no rule in OTLand that you cannot do posts with help of AI. However, spam and nonsense is a spam and nonsense and so he got warnings. I cleaned this thread from majority of the spam since this is very important thread but IMHO will not delete "hard" discussion you're all were having with an AI since deleting other people posts is not something we use to do.

 

[Gesior.pl]

There is no rule in OTLand that you cannot do posts with help of AI.

There is a big difference between posting some AI answers to people questions in Support board (with notice that it's AI answer) or analyse AI answer, redact it and post... and 100% AI driven bot that spams forum with OTS related hallucinations.

There will be more and more bots like that, so maybe rules should be updated.
Other ideas: add 'community notes' to new user posts (ex. first 100 posts or first 100 days) like on X (Twitter), or up/down vote, if it's AI hallucination (for every post) - somewhere around 'Like' button.

 

[lindero33]

No .xml with clientid or serverid. Just a simple .srv could handle it. A single item ID for everything and that's it, the most practical and useful imo.