Tibia 11 .dat File Structure

Discussion in 'Discussion' started by Jo3Bingham, Oct 19, 2016.

  1. Jo3Bingham

    Jo3Bingham Well-Known Member

    Joined:
    Mar 3, 2008
    Messages:
    947
    Likes Received:
    391
    Best Answers:
    3
    https://tibiapf.com/showthread.php?101-dat-Structure

    Here you can see my progress in understanding the new .dat file structure. My 4th post (Update #3) contains a basically-complete breakdown of the structure. I still have a few flags to find, and some unknown values to figure out, but, for the most part, I have the whole structure decoded.
     
  2. Tarek1337

    Tarek1337 Well-Known Member

    Joined:
    Jun 9, 2012
    Messages:
    3,557
    Likes Received:
    1,131
    Best Answers:
    3
    omg, this is nice. :eek:.

    gj Jo3
     
  3. EdMignari

    EdMignari Well-Known Member

    Joined:
    Jan 8, 2011
    Messages:
    191
    Likes Received:
    312
    Best Answers:
    0
  4. Jo3Bingham

    Jo3Bingham Well-Known Member

    Joined:
    Mar 3, 2008
    Messages:
    947
    Likes Received:
    391
    Best Answers:
    3
    Update #4

    Thanks to Daniel (blackd), the weird numbers (like with object ids, and some flags) has been solved.

    If a value is greater-than-or-equal to 0x80 then there will be another value following it. If that value is greater-than-or-equal to 0x80 there will be another value following it. So on and so forth until the next value is less-than 0x80, but it should never be more than four values. Furthermore, the value of each byte can be calculated with the following formula: ((128^x)*(n))-(128^x) where x is the index of the current byte (0-based) and n is the value of the byte. Using recursion, and adding each calculated value, we can get the final value.

    For example, the object id of rope is 0xBB 0x17 which, when read as a 2-byte integer gives the value 6075, but we know that the item id of rope is 3003. If we plug 0xBB (187) into the equation we get ((128^0)*(187))-(128^0)=186. We use 0 for x because 0xBB is the first byte and we count up from 0. Now, if we plug 0x17 (23) into the equation we get ((128^1)*(23))-(128^1)=2816. We use 1 for x here because it's the second byte but we count up from 0. Take the sum of both values and you get 186+2816=3002. The item id of rope is actually 3003, so we just need to add 1 to the sum.

    pseudo-code:
    Code (Text):
    1.  
    2. int readValue(byte[] buffer, int position, int index = 0) {
    3.     byte num1 = buffer[position];
    4.     int num2 = ((128^index)*(num1))-(128^index);
    5.  
    6.     if (index == 0) {
    7.         num2++;
    8.     }
    9.  
    10.     if (num1 < 128) {
    11.         return num2;
    12.     }
    13.  
    14.     return num2 + readValue(buffer, position++, index++);
    15. }
    16.  
    I'm not the best at formulas, so if anyone can write a better one to remove that first if-statement that would be great.
     
  5. Mark

    Mark Administrator Staff Member Administrator

    Joined:
    May 27, 2007
    Messages:
    6,382
    Likes Received:
    1,357
    Best Answers:
    20
    Sounds like protobuf.
     
    Amiroslo likes this.
  6. Mkalo

    Mkalo ボーカロイド Support Team

    Joined:
    Jun 1, 2011
    Messages:
    1,120
    Likes Received:
    894
    Best Answers:
    53
    (128 ^ x) * (n - 1) and you could instead of doing recursively, use a while and after the loop is done you would sum 1 to the result, then you wouldn't have this if there.
     
  7. Yamaken

    Yamaken KeKGod

    Joined:
    Jul 27, 2013
    Messages:
    406
    Likes Received:
    276
    Best Answers:
    2
  8. Summ

    Summ (\/)(;,,;)(\/) Y not? Staff Member Global Moderator

    Joined:
    Oct 15, 2008
    Messages:
    4,157
    Likes Received:
    1,058
    Best Answers:
    7
    Deserializing the integers should work like this:

    Code (Text):
    1.  
    2. let i = 0;
    3. let result = 0;
    4. let current;
    5. do {
    6.     current = buffer[i];
    7.     let value = (current & 0b01111111) << (i++ * 7);
    8.     result = result | value;
    9. } while (current >= 0x80);
    10.  
     
    Last edited: Oct 21, 2016
    Tarek1337, MartyX and Mkalo like this.
  9. Summ

    Summ (\/)(;,,;)(\/) Y not? Staff Member Global Moderator

    Joined:
    Oct 15, 2008
    Messages:
    4,157
    Likes Received:
    1,058
    Best Answers:
    7
    Yamaken, Syntax, Ninja and 4 others like this.
  10. godoyxd

    godoyxd Noob

    Joined:
    Mar 6, 2008
    Messages:
    196
    Likes Received:
    33
    Best Answers:
    0
    Hello

    using a script from this post I could extract two .proto files from the linux version of the client 11.

    shared.proto
    Code (Text):
    1. package tibia.protobuf.shared;
    2.  
    3.  
    4. enum PLAYER_ACTION {
    5.  PLAYER_ACTION_NONE = 0;
    6.  PLAYER_ACTION_LOOK = 1;
    7.  PLAYER_ACTION_USE = 2;
    8.  PLAYER_ACTION_OPEN = 3;
    9.  PLAYER_ACTION_AUTOWALK_HIGHLIGHT = 4;
    10. }
    11.  
    12. enum ITEM_CATEGORY {
    13.  ITEM_CATEGORY_ARMORS = 1;
    14.  ITEM_CATEGORY_AMULETS = 2;
    15.  ITEM_CATEGORY_BOOTS = 3;
    16.  ITEM_CATEGORY_CONTAINERS = 4;
    17.  ITEM_CATEGORY_DECORATION = 5;
    18.  ITEM_CATEGORY_FOOD = 6;
    19.  ITEM_CATEGORY_HELMETS_HATS = 7;
    20.  ITEM_CATEGORY_LEGS = 8;
    21.  ITEM_CATEGORY_OTHERS = 9;
    22.  ITEM_CATEGORY_POTIONS = 10;
    23.  ITEM_CATEGORY_RINGS = 11;
    24.  ITEM_CATEGORY_RUNES = 12;
    25.  ITEM_CATEGORY_SHIELDS = 13;
    26.  ITEM_CATEGORY_TOOLS = 14;
    27.  ITEM_CATEGORY_VALUABLES = 15;
    28.  ITEM_CATEGORY_AMMUNITION = 16;
    29.  ITEM_CATEGORY_AXES = 17;
    30.  ITEM_CATEGORY_CLUBS = 18;
    31.  ITEM_CATEGORY_DISTANCE_WEAPONS = 19;
    32.  ITEM_CATEGORY_SWORDS = 20;
    33.  ITEM_CATEGORY_WANDS_RODS = 21;
    34.  ITEM_CATEGORY_PREMIUM_SCROLLS = 22;
    35.  ITEM_CATEGORY_TIBIA_COINS = 23;
    36. }
    37.  
    38. enum PLAYER_PROFESSION {
    39.  PLAYER_PROFESSION_ANY = -1;
    40.  PLAYER_PROFESSION_NONE = 0;
    41.  PLAYER_PROFESSION_KNIGHT = 1;
    42.  PLAYER_PROFESSION_PALADIN = 2;
    43.  PLAYER_PROFESSION_SORCERER = 3;
    44.  PLAYER_PROFESSION_DRUID = 4;
    45.  PLAYER_PROFESSION_PROMOTED = 10;
    46. }
    47.  
    48. enum ANIMATION_LOOP_TYPE {
    49.  ANIMATION_LOOP_TYPE_PINGPONG = -1;
    50.  ANIMATION_LOOP_TYPE_INFINITE = 0;
    51.  ANIMATION_LOOP_TYPE_COUNTED = 1;
    52. }
    53.  
    54. enum HOOK_TYPE {
    55.  HOOK_TYPE_SOUTH = 1;
    56.  HOOK_TYPE_EAST = 2;
    57. }
    appearances.proto
    Code (Text):
    1. package tibia.protobuf.appearances;
    2.  
    3. import "shared.proto";
    4.  
    5.  
    6. enum FIXED_FRAME_GROUP {
    7.  FIXED_FRAME_GROUP_OUTFIT_IDLE = 0;
    8.  FIXED_FRAME_GROUP_OUTFIT_MOVING = 1;
    9.  FIXED_FRAME_GROUP_OBJECT_INITIAL = 2;
    10. }
    11. message Appearances {
    12.  repeated Appearance object = 1;
    13.  repeated Appearance outfit = 2;
    14.  repeated Appearance effect = 3;
    15.  repeated Appearance missile = 4;
    16. }
    17.  
    18. message SpritePhase {
    19.  optional uint32 duration_min = 1;
    20.  optional uint32 duration_max = 2;
    21. }
    22.  
    23. message SpriteAnimation {
    24.  optional uint32 default_start_phase = 1;
    25.  optional bool synchronized = 2;
    26.  optional bool random_start_phase = 3;
    27.  optional shared.ANIMATION_LOOP_TYPE loop_type = 4;
    28.  optional uint32 loop_count = 5;
    29.  repeated SpritePhase sprite_phase = 6;
    30. }
    31.  
    32. message Box {
    33.  optional uint32 x = 1;
    34.  optional uint32 y = 2;
    35.  optional uint32 width = 3;
    36.  optional uint32 height = 4;
    37. }
    38.  
    39. message SpriteInfo {
    40.  optional uint32 pattern_width = 1;
    41.  optional uint32 pattern_height = 2;
    42.  optional uint32 pattern_depth = 3;
    43.  optional uint32 layers = 4;
    44.  repeated uint32 sprite_id = 5;
    45.  optional uint32 bounding_square = 7;
    46.  optional SpriteAnimation animation = 6;
    47.  optional bool is_opaque = 8;
    48.  repeated Box bounding_box_per_direction = 9;
    49. }
    50.  
    51. message FrameGroup {
    52.  optional FIXED_FRAME_GROUP fixed_frame_group = 1;
    53.  optional uint32 id = 2;
    54.  optional SpriteInfo sprite_info = 3;
    55. }
    56.  
    57. message Appearance {
    58.  optional uint32 id = 1;
    59.  repeated FrameGroup frame_group = 2;
    60.  optional AppearanceFlags flags = 3;
    61. }
    62.  
    63. message AppearanceFlags {
    64.  optional AppearanceFlagBank bank = 1;
    65.  optional bool clip = 2;
    66.  optional bool bottom = 3;
    67.  optional bool top = 4;
    68.  optional bool container = 5;
    69.  optional bool cumulative = 6;
    70.  optional bool usable = 7;
    71.  optional bool forceuse = 8;
    72.  optional bool multiuse = 9;
    73.  optional AppearanceFlagWrite write = 10;
    74.  optional AppearanceFlagWriteOnce write_once = 11;
    75.  optional bool liquidpool = 12;
    76.  optional bool unpass = 13;
    77.  optional bool unmove = 14;
    78.  optional bool unsight = 15;
    79.  optional bool avoid = 16;
    80.  optional bool no_movement_animation = 17;
    81.  optional bool take = 18;
    82.  optional bool liquidcontainer = 19;
    83.  optional bool hang = 20;
    84.  optional AppearanceFlagHook hook = 21;
    85.  optional bool rotate = 22;
    86.  optional AppearanceFlagLight light = 23;
    87.  optional bool dont_hide = 24;
    88.  optional bool translucent = 25;
    89.  optional AppearanceFlagShift shift = 26;
    90.  optional AppearanceFlagHeight height = 27;
    91.  optional bool lying_object = 28;
    92.  optional bool animate_always = 29;
    93.  optional AppearanceFlagAutomap automap = 30;
    94.  optional AppearanceFlagLenshelp lenshelp = 31;
    95.  optional bool fullbank = 32;
    96.  optional bool ignore_look = 33;
    97.  optional AppearanceFlagClothes clothes = 34;
    98.  optional AppearanceFlagDefaultAction default_action = 35;
    99.  optional AppearanceFlagMarket market = 36;
    100.  optional bool wrap = 37;
    101.  optional bool unwrap = 38;
    102.  optional bool topeffect = 39;
    103. }
    104.  
    105. message AppearanceFlagBank {
    106.  optional uint32 waypoints = 1;
    107. }
    108.  
    109. message AppearanceFlagWrite {
    110.  optional uint32 max_text_length = 1;
    111. }
    112.  
    113. message AppearanceFlagWriteOnce {
    114.  optional uint32 max_text_length_once = 1;
    115. }
    116.  
    117. message AppearanceFlagLight {
    118.  optional uint32 brightness = 1;
    119.  optional uint32 color = 2;
    120. }
    121.  
    122. message AppearanceFlagHeight {
    123.  optional uint32 elevation = 1;
    124. }
    125.  
    126. message AppearanceFlagShift {
    127.  optional uint32 x = 1;
    128.  optional uint32 y = 2;
    129. }
    130.  
    131. message AppearanceFlagClothes {
    132.  optional uint32 slot = 1;
    133. }
    134.  
    135. message AppearanceFlagDefaultAction {
    136.  optional shared.PLAYER_ACTION action = 1;
    137. }
    138.  
    139. message AppearanceFlagMarket {
    140.  optional shared.ITEM_CATEGORY category = 1;
    141.  optional uint32 trade_as_object_id = 2;
    142.  optional uint32 show_as_object_id = 3;
    143.  optional string name = 4;
    144.  repeated shared.PLAYER_PROFESSION restrict_to_profession = 5;
    145.  optional uint32 minimum_level = 6;
    146. }
    147.  
    148. message AppearanceFlagAutomap {
    149.  optional uint32 color = 1;
    150. }
    151.  
    152. message AppearanceFlagHook {
    153.  optional shared.HOOK_TYPE direction = 1;
    154. }
    155.  
    156. message AppearanceFlagLenshelp {
    157.  optional uint32 id = 1;
    158. }
    159.  
     
    Jo3Bingham likes this.
  11. Gold Ray

    Gold Ray New Member

    Joined:
    Jun 18, 2012
    Messages:
    8
    Likes Received:
    3
    Best Answers:
    0
    Anyone is able to extract proto files from the newest client?

    There were some changes, those posted above are not actual anymore and the mentioned script is unable to extract them from the newest client (gives only shared.proto and exits with errors).

    Thanks.
     
  12. eldera

    eldera Active Member

    Joined:
    Oct 27, 2012
    Messages:
    141
    Likes Received:
    34
    Best Answers:
    0
    Gold Ray likes this.
  13. samco

    samco 4x4 Developer.

    Joined:
    Jul 3, 2007
    Messages:
    756
    Likes Received:
    137
    Best Answers:
    4
    Im very interested in this. This may lead to a tibia 11 item editor?

    I want to up this thread.
     
    Last edited by a moderator: Jun 13, 2018 at 10:57 AM
  14. Jo3Bingham

    Jo3Bingham Well-Known Member

    Joined:
    Mar 3, 2008
    Messages:
    947
    Likes Received:
    391
    Best Answers:
    3
    Yeah, it would be very easy to make an item editor for Tibia 11 knowing these are just protobuf files. You can use Protod on the client to extract all of it's .proto files, then you can use Google's protoc executable (you can find an already compiled version under Releases) on those .proto files to generate code to use with C++, Go, Java, Python, Ruby, C#, Objective C, Javascript, or PHP. Just add the generated code to your project to easily parse and work with the protobuf files.
     
  15. samco

    samco 4x4 Developer.

    Joined:
    Jul 3, 2007
    Messages:
    756
    Likes Received:
    137
    Best Answers:
    4
    Thanks for the pointing, will make some research

    I've been working with all those links you gave me. I already managed to get the *.dat from tibia 11 loaded my c# project.
    My next step is to integrate with the sprite dump prohect is out there so i will be able to display all items information + image.

    Next step after that is: How to integrate the items.otb part? I need to check the otb editors, maybe is not that hard.

    Edit:

    I have now a "working" solution in C#, whit a simple WindowsForm application, i do load .dat and sprites.

    If someone is interested, i can upload it to github so you check the code and we all can start guessing how the sprites are managed. There are different sprites tipes and areas... iu still dont know all of them, and that will be needed to crop the sprite images correctly.
     
    Last edited by a moderator: Jun 17, 2018 at 10:14 PM
  16. gunz

    gunz GunzOT Owner Premium User

    Joined:
    Dec 30, 2007
    Messages:
    457
    Likes Received:
    138
    Best Answers:
    1

Share This Page

Loading...