Tibia Specifications Initiative

[Mazen]

There should be an initiative to help developer get a better start with supporting Open-Tibia and allow for wider adoption of Open-Tibia programs such as OTClient by providing detailed information about all Tibia-related protocols and file formats.

Open-Tibia would not have been possible without the detailed understanding of Tibia's protocols and file formats by its developers. Such would be the structure of the Tibia sprite file (.spr), and the login protocol that allows the client to communicate with the server and establish a safe connection. Throughout the years this information has continued to be collected but has never been at one place at the same time except in the form of an implementation (otserv, otclient, and many others). What we need is one source for all this information so that future development will be easier on everyone, and to allow the learning-curve for new developers to be much less steep than it is today.

The Problem
The issue we are currently facing is that many older developers end up not having time, or the will (we grow up/interests change) to keep up the cat & mouse game between us and CipSoft. What I am talking about are the new protocols (10.77, 10.90, 10.91, etc.) that are coming up every so month, and the new items, outfits, monsters, and worse of all the game-protocol and file-format changes. Although these changes have been coming at a slow phase throughout the years it has still been a rather big effort to implement all of those features into Open-Tibia. Every one of us will not be staying at one place our entire lives, someone has to eventually replace us. In the case of Open-Tibia experienced developers are not as many as they were before, those who really understand how the Tibia protocols work, they are none but a few, and they are Not being replaced. That is the problem. We have a very few new developers who are getting into C++ development of otserv and even fewer who really understand the Tibia protocols. When an experienced developer leaves, he takes all the knowledge of those protocols with him. What we have left is the code-base and fractions of information here and there of how things really work. If this continues the death of Open-Tibia will be greatly accelerated.

Symptoms
Today we have an amazing project that almost completely replaces CipSoft's Tibia client. However, there still are some things left to be done before it can actually replace Tibia Client completely, and some of these things are unfortunately taking a very long time because of this problem that we are discussing.

An example is: https://github.com/edubart/otclient/issues/642

This problem, like probably many others, are a result of lack of documentation of the Tibia file formats. Because otclient is more dependent on CipSoft's stuff it will have more of these bugs. This is why we need to document every detail of CipSoft's file-formats and protocols, and then see if otclient (and otserv for that matter) comply with the specifications. In this way we can also enable more developers (such as myself) to help the OT community out with these issues. Right now any developer who wants to help will have to guess how things work from existing code, because they don't have what only a very few people have, which are the details on how the Tibia protocols/formats work.

By the way, if you didn't notice, this bug has been there for a while, and it hasn't been fixed.

The Solution
The solution to this problem is to collect all information about all relevant protocols and file formats in one place. Wherever that place/platform is, it should be maintained by the most trusted people in the community, preferably in a public Git repository or a Wiki where changes are non-destructive and where the information can be easily downloaded. It should also be easy to contribute to it. Even if you have a very small piece of information that you see is missing, you should be able to easily, within a week see it as a part of the documentation.

The text format of the documentation should be decided before-hand, and be followed at all time to keep a consistent structure. For the file-format, it's preferable to use something simple such as Markdown, but you could also use something else that is open and non-binary such as Latex. But in case of a Wiki, you may have to use their custom format.

If many members of the community, and especially the most experienced developers, decide to be a part of this initiative we might have something useful in a very short period of time. I'm personally doing nothing here more than giving an advice, I've got a great deal of experience in running OT-servers but I have nothing to contribute in terms of Tibia protocols and file formats. This is all up to you.

 

[Yamaken]

Sometimes there is no better documentation than a clean implementation. I think it will be too much work to just make documentation on all protocold, formats and etc. A good and clean implementation may be easier and help any programmer with a minimum knowledge of the platform and language.

What may be lacking is to explain how a binary format works, how tibia organize sprite sheets into the dat file and spr file, what are packets and how tfs/opentibia works with them. All those things i needed to dig deep into a lot of source code(from tfs, otclient, otitemeditor, etc) to learn so i feel you, its hard to learn all those things.

About the issue with otclient, what is lacking is the documentation of how the feature works and then someone to work in the implementation. Basically the workflow is simple, we got a feature missing, someone report it, we get the details and then implement. I think there is no better method than this for a implementation(otclient) that wants to replace another implementation(c++ client). The issue is how many people are reporting and how many people are working on otclient. Another issue is that otclient wants to be so many things and support so many protocols(from 7.7 to 10.91).

And i'm glad you did this topic, topics with quality and important subjects are really missing here in OTLand.

 

[Shawak]

I totally agree on anything you said, nice thread.

 

[Mazen]

Sometimes there is no better documentation than a clean implementation. I think it will be too much work to just make documentation on all protocold, formats and etc. A good and clean implementation may be easier and help any programmer with a minimum knowledge of the platform and language.

What may be lacking is to explain how a binary format works, how tibia organize sprite sheets into the dat file and spr file, what are packets and how tfs/opentibia works with them. All those things i needed to dig deep into a lot of source code(from tfs, otclient, otitemeditor, etc) to learn so i feel you, its hard to learn all those things.

It's really up to the community to assemble these documentations under trusted supervision. I know it cannot happen overnight, but I believe that many developers want to contribute and this will give them a place to turn to. Just like we turn to Mark's fork of otserv when we want to contribute to otserv, we will be able to do the same to this source of documentation. It will build up over time by many willing contributors, and I mentioned that it should follow a consistent format for the very reason that these different contributors will have different styles of writing. This is where the trusted supervison comes in.

About the issue with otclient, what is lacking is the documentation of how the feature works and then someone to work in the implementation. Basically the workflow is simple, we got a feature missing, someone report it, we get the details and then implement. I think there is no better method than this for a implementation(otclient) that wants to replace another implementation(c++ client). The issue is how many people are reporting and how many people are working on otclient. Another issue is that otclient wants to be so many things and support so many protocols(from 7.7 to 10.91).

The documentation of an implementation is definitely an issue, but today the lack (non-existence) of documentations on protocols and file formats is more relevant because it affects almost every important OT project out there (otserv, otclient, RME, etc.). Implementation documentations should be written by the developers of their respective project, but this problem is on a smaller scale. The problem that we are discussing has to be addressed first, since it is more fundamental. Lets explain why through a scenario where implementation documentations are written and protocol/file-format documentations are not:

1. Tibia 10.100 is released with new .spr file-format changes.
2. otclient kept their implementation up to date, because they had experienced developers who knew how to decipher the new file-format.
3. RME, because of its lack of support is falling behind and cannot incorporate the new changes into its project.
4. CoolDeveloper1337 is an experienced C++ developer and wants to help the RME people out. But he doesn't know anything about these file formats, and the only way for him to find out is to look into otclient's code or ask its developers for guidance.

Now if CoolDeveloper1337 had a documentation for the file-formats he would have been able to fix the probem in RME without knowing anything about otclient or get to know its developers. This is a realistic scenario, because if some project implements a feature from Tibia, developers from other projects will most likely be in CoolDeveloper1337's position when they aren't able to decipher a new file-format.

 

[BeniS]

This is pretty up to date for the most part
https://github.com/edubart/otclient/blob/master/src/client/game.cpp#L1505

 

[Mazen]

This is pretty up to date for the most part
https://github.com/edubart/otclient/blob/master/src/client/game.cpp#L1505

That link is a good start, but we need this information in non-code-format. The documentation is supposed to describe every detail of how all of these features work. For example: GameIngameStore, how does the server communicate with the client to send the store, and how does the client send purchase-requests back to the server?

I believe this file can help us create a template for these features, basically empty sections that we can insert the details into.

 

[godoyxd]

several months ago I decided to learn about the dat file and spr files, it is somehow dificult because of the length of it's specification. I was able to build a python script to parse from binary to JSON. That really helps to understand the specification and it is not anything close to something with performance. Right now I sotopped doing that because I couldn't figure out some stuff about the spr file. But I've got the dat file from binary to JSON working. And vise versa.
The code is a mess, but I believe that with help we can do better.
And python could be the specification that we need. As a programmer you should know how to read pyhton.
code is in: https://github.com/Tarliton/otbinutils

 

[EdMignari]

A few months ago, I started a repository for specifications, but this is a hard work if you have many different versions and as Yamaken said, sometimes a clean implementation is better than specifications.

https://github.com/ottools/specifications

I will be available if you need help or have any questions.

 

[Shawak]

A good documentation is always better than a clean implementation. Some people might don't understand a specific programming language or the implemention itself, because they have to understand the formats itself to actually understand the implementation.

This argument is invalid and only an excuse for people who are to lazy to create a documentation.

 

[Mazen]

A few months ago, I started a repository for specifications, but this is a hard work if you have many different versions and as Yamaken said, sometimes a clean implementation is better than specifications.

https://github.com/ottools/specifications

I will be available if you need help or have any questions.

Someone need to translated this to English, it looks really good:

https://github.com/ottools/specifications/blob/master/spr/spr-pt_BR.md

several months ago I decided to learn about the dat file and spr files, it is somehow dificult because of the length of it's specification. I was able to build a python script to parse from binary to JSON. That really helps to understand the specification and it is not anything close to something with performance. Right now I sotopped doing that because I couldn't figure out some stuff about the spr file. But I've got the dat file from binary to JSON working. And vise versa.
The code is a mess, but I believe that with help we can do better.
And python could be the specification that we need. As a programmer you should know how to read pyhton.
code is in: https://github.com/Tarliton/otbinutils

Although python can look cleaner than C++ I don't agree that it is a proper format for writing documentation in. To give you an idea, it should be something like what EdMignari wrote. (In English ofc.)

Your code could help us write a proper documentation, because it's easy to read.

 

[Jano]

An example is: https://github.com/edubart/otclient/issues/642

I posted a possible fix for this issue few weeks ago but apparently it went under the radar.
https://otland.net/threads/7-4-10-9-otclient-0-6-3-2474.238797/page-2#post-2333186

@on topic

From my own experience the biggest problem are the files structure changes, protocol changes are a lot easier to track.

Also most of main big devs aren't up to date with Tibia mechanics changes (and thats perfectly ok, I'm not up to date neither), this becomes a problem when you trying to put into code a game mechanic, end users (players/ OT owners) most of the time doesn't know how to explain to a developer how things supposed to work (the little details matters)

Great initiative, I hope it doesn't end in nothing, willing to contribute if it sees the light eventually

 

[president vankk]

Someone need to translated this to English, it looks really good:

https://github.com/ottools/specifications/blob/master/spr/spr-pt_BR.md

I hope it be useful.

https://github.com/vankk/specificat...5a24daca658c814318b885902b86/spr/spr-pt_BR.md

 

[BeniS]

That link is a good start, but we need this information in non-code-format. The documentation is supposed to describe every detail of how all of these features work. For example: GameIngameStore, how does the server communicate with the client to send the store, and how does the client send purchase-requests back to the server?

I believe this file can help us create a template for these features, basically empty sections that we can insert the details into.

Can Ctrl + F GameIngameStore to see how the implementation works. Then makes it easier to reverse engineer the server end.

 

[godoyxd]

well,
I strongly agree that a documented specification is better than an implementation in any language, but if you try to document the DAT file you will have a really hard time to do if you try to make it simple.
That file structure is a mess (IMO).

You start with:
File signature: 4 bytes
quantity of items: 2 bytes
quantity of outfits: 2 bytes
quantity of effects: 2 bytes
quantity of projectiles: 2 bytes
until here is ok (you don't say)

now the fun begings, the veterans called this byte flags (1 byte).
We start with an arbritary attribute (I don't know if has an order) and end with the last value a byte can represent (255).
We can have several attributes in the middle of those values.

Then after reading the breaking byte for attributes, it is now time to read what the veterans called "frame group count". It is a byte (1 byte).
And it can be missing if our current data don't represents an item or outfit. So you only read sometimes. The default is 1 when it's missing.

If is not missing we have to read another byte (1 byte) that is the frame group ID.

Now it doesn't matter if is missing or not, so we have to read (both 1 byte):
width
height
if those are greater than 1, there is another info that they called exact size (another byte)
otherwise that value is missing and you do not need to read it!

now we got to read (all one byte):
layers
pattern width
pattern height
pattern depth
phases <<<<<<<< this can be 0!!!!!

If phases are not 0 (greater than 1):
we have to read it's attributes:
is asynchronous (1 byte)
quantity of loops (4 bytes)
start phase (1 byte)

now for the quantity of phases we have to read it's duration:
mininum (4 bytes)
maximum (4 bytes)

OK, so we are done reading phases data.
Now we are going to read which sprites belongs to this item.
That is, the sprites ID that we have in SPR file.

The math that I have to calculate how many sprites we got is (this is not mine):
quantity of sprites = width*height*layers*patternWidth*patternHeight*patternDepth*phases

now for that quantity we have several sprites of 4 bytes:
read sprite (4 bytes) until quantity of sprites

Are we done yet???????? not so fast.

that was only for the FIRST ITEM/DATA

you have to do all that for the others N items or using this formula
max items or data = quantity of items + quantity of outfits + quantity of effects + quantity of projectiles

and now we are done!

That is not hard to explain like this, but it is hard to put all that in a file, like EdMignari did for the SPR.
I tried and the file is HUGE using the 1...2...3...N for specifying loops.
That way the file is WAY more complex than a simple script in any language.

I do prefer documented specification instead code, but this file structure is wierdly complex.

anybody got any ideas?

 

[Leo32]

Documentation would be nice, but its all out there. post by post, just need to look.
Documentation isn't going to help get this community out of the shitter.

A web-based client would.
Need to expand.

Kinda too late already, community has eaten itself alive due to secrecy and monetary competition.