• There is NO official Otland's Discord server and NO official Otland's server list. The Otland's Staff does not manage any Discord server or server list. Moderators or administrator of any Discord server or server lists have NO connection to the Otland's Staff. Do not get scammed!

[Investigation] Flash Client .DAT Spec

Leo32

Getting back into it...
Joined
Sep 21, 2007
Messages
987
Solutions
14
Reaction score
532
EDIT: Skip to the conclusion and Flash Client .DAT breakdown here.


Intro

I've been developing my project for awhile using the flash client:
KG4Nibm.gif


The client itself is working just fine and suits the project perfectly, but there is an major annoyance with item usability...
The Flash client uses smart-clicks, meaning the client dictates what to do when you right-click on an object:
  • Use
  • Use With...
  • Look
etc.

This data looks to be contained within the appearances.dat file:
https://github.com/Leo32onGIT/TFS-F...a167fb4e3d2fb93d4574fd0559d160140474e3431.dat

Task
The drama is the decoding of the .dat file.
It's spec is different to the standard tibia.dat spec which should be obvious as sprites are now stored as an atlas (?).

Preliminary
The structure of the item blocks/flags looks to be much the same:

C++ Client's Tibia.dat:


Flash Client's Appearances.dat:


So there shouldn't be any need to reinvent the wheel here.
It's a matter of understanding the structure of the old Tibia.dat file properly, then decipher the new data included in the new Appearances.dat file.

This is my first attempt at trying to reverse engineer a binary file format, so if anyone has any experience or can help in any meaningful way - it would be greatly appreciated.
 
Last edited:
A lot of things, that I don't understand.

Impressive.

Honestly this is quite amateur on my part.
I'm sure there are many out there who could:
  • Analyse the contents of the dat.
  • Recognise the pattern for each item data block.
  • Deduce the byte lengths needed for each value.
  • Wrap it all up in a pretty GUI.
Way..... quicker than I can even hope to achieve.

I'm still at the stage of dumping the single byte values to find the data pattern, literally trying to learn C# as I go xD

Byte Dump
Code:
using System;
using System.IO;

class Program
{
  static void Main()
  {
  string DATinput = @"E:\Other Apps\Rook Story\_Flash\dat spec\Appearances.dat";
  string TXToutput = @"E:\Other Apps\Rook Story\_Flash\dat spec\Appearances.txt";

  // Read all bytes in from a file on the disk.
  byte[] file = File.ReadAllBytes(DATinput);

  // Create a memory stream from those bytes.
  using (MemoryStream memory = new MemoryStream(file))
  {
  // Use the memory stream in a binary reader.
  using (BinaryReader reader = new BinaryReader(memory))
  {
  // reader.BaseStream.Position = 0;
  StreamWriter output = new StreamWriter(TXToutput);

  // Read in each byte from memory.
  for (int i = 0; i < file.Length; i++)
  {
  byte result = reader.ReadByte();
  Console.WriteLine(result);
  output.WriteLine(result);
  }
  }
  Console.WriteLine("You got a long way to go, bud.");
  }
  }
}

Still got quite a bit reading to do before my labour bears any fruit :p
 
I'm sure there are many out there who could:
  • Analyse the contents of the dat.
  • Recognise the pattern for each item data block.
  • Deduce the byte lengths needed for each value.
  • Wrap it all up in a pretty GUI.
Way..... quicker than I can even hope to achieve.
There is always someone who can do something quicker or better :p
 
Best resource I've found so far:

Pre-requisites:
DatReader.rb
Code:
require 'bindata'

class TibiaString < BinData::Record
endian :little

uint16 :len, value: -> { s.length }
string :s, read_length: :len

def get; s; end
def set v; s = v; end
end

Bank, Clip, Bottom, Top, Container, Cumulative, ForceUse,
MultiUse, Write, WriteOnce, LiquidContainer, LiquidPool, Unpass, Unmove,
Unsight, Avoid, Take, Hang, HookSouth, HookEast, Rotate,
Light, DontHide, Translucent, Shift, Height, LyingObject, AnimateAlways,
Automap, LensHelp, FullBank, IgnoreLook, Clothes, Market = *(0 .. 33)

Armors, Amulets, Boots, Containers, Decoration, Food, Helmets_Hats,
Legs, Others, Potions, Rings, Runes, Shields, Tools,
Valuables, Ammunition, Axes, Clubs, DistanceWeapons,
Swords, Wands_Rods, MetaWeapons = *(1 .. 22)

class DatObject < BinData::Record
endian :little

array :flags, read_until: -> { element.flag == 0xFF } do
uint8 :flag, check_value: -> { value <= 33 or value == 0xFF }

uint8 :waypoints, onlyif: -> { flag == Bank }
uint16 :maxtextlength, onlyif: -> { flag == Write or flag == WriteOnce }
uint16 :brightness, onlyif: -> { flag == Light }
uint16 :lightcolor, onlyif: -> { flag == Light }
uint16 :displ_x, onlyif: -> { flag == Shift }
uint16 :displ_y, onlyif: -> { flag == Shift }
uint16 :elevation, onlyif: -> { flag == Height }
uint16 :automapcolor, onlyif: -> { flag == Automap }
uint16 :lenshelp, onlyif: -> { flag == LensHelp }
uint16 :clothslot, onlyif: -> { flag == Clothes }
uint16 :marketcategory, onlyif: -> { flag == Market }
uint16 :markettradeas, onlyif: -> { flag == Market }
uint16 :marketshowas, onlyif: -> { flag == Market }
tibia_string :marketname, onlyif: -> { flag == Market }
uint16 :marketrestrictprofession, onlyif: -> { flag == Market }
uint16 :marketrestrictlevel, onlyif: -> { flag == Market }
end

uint8 :width
uint8 :height
uint8 :exactsize, onlyif: -> { width > 1 or height > 1 }

uint8 :layers
uint8 :patternwidth
uint8 :patternheight
uint8 :patterndepth
uint8 :phases

array :sprites, type: :uint16, initial_length: -> { width * height * layers * patternwidth * patternheight * patterndepth * phases }
end

class TibiaDat < BinData::Record
endian :little

uint32 :version

uint16 :numitems
uint16 :numoutfits
uint16 :numeffects
uint16 :numprojectiles

array :items, type: :dat_object, initial_length: -> { numitems - 100 }
array :outfits, type: :dat_object, initial_length: -> { numoutfits }
array :effects, type: :dat_object, initial_length: -> { numeffects }
array :projectiles, type: :dat_object, initial_length: -> { numprojectiles }
end

BinData::trace_reading do
struct = File.open('Appearances.dat') {|f| TibiaDat.read(f) }
end

It's a little dated, was made with the C++ client .dat in mind;
So it bombs out in the end and doesn't include ALL flags - but it helps with understanding the logic behind the raw dat files.

@Jo3Bingham have you already got a parser for the flash client data file?
 
Last edited:
Did anyone decompiled the client yet? that would be awesome to see how the .dat works aswell as adding custom features to the client
 
Did anyone decompiled the client yet? that would be awesome to see how the .dat works aswell as adding custom features to the client

Any SWF decompiler works, requires quite a bit of work to get it back to a compile-able form though.
If someone does have a compile-able form - they don't seem to be sharing.
 
As @Leo32 said, any SWF decompiler will work.

The main file for DAT parsing is: https://gist.github.com/Cavitt/392eb843de2a5016a796f0f9f3500f93

Pretty much every tibia.appearances class file is relevant.

Don't know why I didn't start here honestly:
https://gist.github.com/Cavitt/392eb843de2a5016a796f0f9f3500f93#file-appearancestorage-as-L9
https://gist.github.com/Cavitt/392eb843de2a5016a796f0f9f3500f93#file-appearancestorage-as-L754-L791

0x23

Makes it a easier to find what I'm after, thanks.

What I'm finding really annoying is the fact that itemID's are not present in the .dat structure.
It's just a rolling list of flags and sprite information, the itemID is basically just a count.

Makes it so much harder to just bodge fix the few items that need changing :(
 
Last edited:
I don't have the skill to create a parser, but I have achieved what I need.

Let me go through the basics, basically echoing what Jo3Bingham has out-lined here.
Here is a snippet of the start of the .dat file in the 10.78 c++ client:
Code:
CC 39 00 00 CB 55 E0 02 AA 00 36 00 00 00 00 0C
0D 0E 16 03 00 9C 00 1F FF 01 01 01 04 04 01 01
88 00 00 00 89 00 00 00 8A 00 00 00 8B 00 00 00
8C 00 00 00 8D 00 00 00 8E 00 00 00 8F 00 00 00
90 00 00 00 91 00 00 00 92 00 00 00 93 00 00 00
94 00 00 00 95 00 00 00 96 00 00 00 97 00 00 00

This includes the file header and the first item (itemID 100):
KNzaTF6.png


We're only after item information, so lets get rid of the header information:
Code:
[First 4 bytes] - dat version
[Next 2 bytes] - item count
[Next 2 bytes] - outfit count
[Next 2 bytes] - effect count
[Next 2 bytes] - distance count

So lets ignore the first 12 bytes:
Code:
CC 39 00 00 CB 55 E0 02 AA 00 36 00

And we're left with:
Code:
00 00 00 0C 0D 0E 16 03 00 9C 00 1F FF 01 01 01
04 04 01 01 88 00 00 00 89 00 00 00 8A 00 00 00
8B 00 00 00 8C 00 00 00 8D 00 00 00 8E 00 00 00
8F 00 00 00 90 00 00 00 91 00 00 00 92 00 00 00
93 00 00 00 94 00 00 00 95 00 00 00 96 00 00 00
97 00 00 00

So now we have the data for item 100.
Based on Jo3's post, 0xFF is our terminator that separates the data, so lets format this a little:
Code:
ITEM FLAG DATA:
00 00 00 0C 0D 0E 16 03 00 9C 00 1F FF

SPRITE FLAG DATA:
01 01 01 04 04 01 01

SPRITE DATA:
88 00 00 00 89 00 00 00 8A 00 00 008B 00 00 00
8C 00 00 00 8D 00 00 00 8E 00 00 00 8F 00 00 00
90 00 00 00 91 00 00 00 92 00 00 00 93 00 00 00
94 00 00 00 95 00 00 00 96 00 00 00 97 00 00 00

Cool.
Now lets try and do the same for the Flash client's .dat file:
Code:
69 56 E0 02 AA 00 36 00 00 00 00 0C 0D 0E 16 03
00 9C 00 1F FF 01 01 01 04 04 01 01 17 DE 00 00
18 DE 00 00 19 DE 00 00 1A DE 00 00 1B DE 00 00
1C DE 00 00 1D DE 00 00 1E DE 00 00 1F DE 00 00
20 DE 00 00 21 DE 00 00 22 DE 00 00 23 DE 00 00
24 DE 00 00 25 DE 00 00 26 DE 00 00

Seems there is no dat version present in the header.
This makes sense as it's present in the .xml file that serves the appearances.dat file.

So lets just ignore the first 8 bytes instead of 12:
Code:
69 56 E0 02 AA 00 36 00

Left with:
Code:
00 00 00 0C 0D 0E 16 03 00 9C 00 1F FF 01 01 01
04 04 01 01 17 DE 00 00 18 DE 00 00 19 DE 00 00
1A DE 00 00 1B DE 00 00 1C DE 00 00 1D DE 00 00
1E DE 00 00 1F DE 00 00 20 DE 00 00 21 DE 00 00
22 DE 00 00 23 DE 00 00 24 DE 00 00 25 DE 00 00
26 DE 00 00

Hey this is familiar!
Lets just cut to the chase and separate the data
Code:
ITEM FLAG DATA:
00 00 00 0C 0D 0E 16 03 00 9C 00 1F FF

SPRITE FLAG DATA:
01 01 01 04 04 01 01

SPRITE DATA:
17 DE 00 00 18 DE 00 00 19 DE 00 00 1A DE 00 00
1B DE 00 00 1C DE 00 00 1D DE 00 00 1E DE 00 00
1F DE 00 00 20 DE 00 00 21 DE 00 00 22 DE 00 00
23 DE 00 00 24 DE 00 00 25 DE 00 00 26 DE 00 00

So the structure is the same the dat version is just missing from the header and the sprite data is obviously different.

Following? awesome, now let me show you how we change the smart-click functionality.
 
Last edited:
The AppearancesStorage code from the decompiled client tells us the vital flag information that we need:
https://gist.github.com/Cavitt/392eb843de2a5016a796f0f9f3500f93

Specifically this:
Code:
public static const FLAG_DEFAULT_ACTION:int = 35;

And this:
Code:
case FLAG_DEFAULT_ACTION:
{
   _loc_6.isDefaultAction = true;
   _loc_13 = param1.readUnsignedShort();
   switch(_loc_13)
   {
     case 0:
     {
       _loc_6.defaultAction = ACTION_NONE;
       break;
     }
     case 1:
     {
       _loc_6.defaultAction = ACTION_LOOK;
       break;
     }
     case 2:
     {
       _loc_6.defaultAction = ACTION_USE;
       break;
     }
     case 3:
     {
       _loc_6.defaultAction = ACTION_OPEN;
       break;
     }
     case 4:
     {
       _loc_6.defaultAction = ACTION_AUTOWALK_HIGHLIGHT;
       break;
     }
     default:
     {
       break;
     }
   }
   break;
}

So the FLAG is 35 (0x23 in hex) and the values 0 to 4 dictate the action.
Lets verify this by searching for those values (23 01 or 23 02 or 23 03 etc) in the flash client appearances.dat

Yes, it exists for some items:
Code:
00 00 02 0C 0D 1D BA 00 23 01 00 FF

It seems to always have 00 after the value, lets keep that in mind.
(Seems to be normal for alot of item data values to span over 2 bytes).

Ok, lets append the same data to itemID 1940:
YrrPReh.png


I don't have a parser and don't want to manually go through 1940 data blocks....
Let's do this in a 'slightly' smarter way using what we've learned.

Item 1949 is unique enough in its item flags that we should be able to find it:
hBghqrM.png

xdB2olw.png


Light Color 29 (0x1D in hex) and Light Intensity 02 (ox02 in hex duh)
Minimap colour 210 (0xD2 in hex)

So going back to Joe3's post:
0x16 = light info is followed by a 2 byte value indicating the light level then another 2 byte value
0x1D = minimap color is followed by a 2 byte value indicating the minimap color

So, to find the color information we should be looking for:
Code:
16 02 00 1D 00

And to find the minimap information we should be looking for:
Code:
1D D2 00

Can we find it?
YES! earliest instance of a portal in the .dat file:
Code:
16 02 00 1D 00 1D D2 00 23 04 00 FF

Awesome, now lets traverse back 9 items.
FOUND YOU!
Code:
00 00 02 0C 0D 1D BA 00 FF 01 01 01 01 01 01 01
EF F1

So lets edit this for smart-click LOOK:
Code:
00 00 02 0C 0D 1D BA 00 23 01 00 FF 01 01 01 01
01 01 01 EF F1

We've added extra data to the .dat file so its size has changed.
Lets ensure we update this in the .xml that serves the file.

And...
EOZDUO5.gif


AHA!
Now lets do what we actually wanted to do, smart-click USE:
Code:
00 00 02 0C 0D 1D BA 00 23 02 00 FF 01 01 01 01
01 01 01 EF F1

OZLNERS.gif


Success:
 
Last edited:
Working on a primitive parser in C#:
5ypbEqJ.png

It's really needed to find specific items.

It's coming along slowly;
As mentioned above this is literally my first attempt at programming.

Fun project I'll admit.
 
Last edited:
Wish I had the time to help you guys, would have been a fun project indeed :)
 
Here is a detailed break-down.

Do note that this only goes into great detail regarding the object/item data.
The outfit, effect and projectile data do not interest me at this point.


FILE HEADER
Code:
2-bytes | object/item count
2-bytes | outfit count
2-bytes | effect count
2-bytes | projectile count

We then move onto looping Item and Sprite data.
Which consists of:

ITEM_FLAGS
ITEM_FLAGS Terminator (0xFF)
SPRITE_FLAGS
SPRITES

ITEM_FLAGS
Code:
0xFE | FLAG_USABLE
0x00 | FLAG_BANK (Seems to be what we call isGround in TFS)
   if FLAG_BANK
    {
      2-bytes | waypoints (Seems to be what we call groundSpeed in TFS)
    }
0x01 | FLAG_CLIP
0x02 | FLAG_BOTTOM
0x03 | FLAG_TOP
0x04 | FLAG_CONTAINER
0x05 | FLAG_CUMULATIVE
0x06 | FLAG_FORCEUSE
0x07 | FLAG_MULTIUSE
0x08 | FLAG_WRITE
   if FLAG_WRITE
    {
      2-bytes | maxTextLength
    }
0x09 | FLAG_WRITEONCE
   if FLAG_WRITEONCE
    {
      2-bytes | maxTextLength
    }
0x0A | FLAG_LIQUIDCONTAINER
0x0B | FLAG_LIQUIDPOOL
0x0C | FLAG_UNPASS
0x0D | FLAG_UNMOVE
0x0E | FLAG_UNSIGHT
0x0F | FLAG_AVOID
0x10 | FLAG_NOMOVEMENTANIMATION
0x11 | FLAG_TAKE
0x12 | FLAG_HANG
0x13 | FLAG_HOOKSOUTH
0x14 | FLAG_HOOKEAST
0x15 | FLAG_ROTATE
0x16 | FLAG_LIGHT
   if FLAG_LIGHT
    {
      2-bytes | brightness
      2-bytes | lightColour
    }
0x17 | FLAG_DONTHIDE
0x18 | FLAG_TRANSLUCENT
0x19 | FLAG_SHIFT
   if FLAG_SHIFT
    {
      2-bytes | displacementX
      2-bytes | displacementY
    }
0x1A | FLAG_HEIGHT
   if FLAG_HEIGHT
    {
      2-bytes | elevation
    }
0x1B | FLAG_LYINGOBJECT
0x1C | FLAG_ANIMATEALWAYS
0x1D | FLAG_AUTOMAP
   if FLAG_AUTOMAP
    {
      2-bytes | minimapColour
    }
0x1E | FLAG_LENSHELP
   if FLAG_LENSHELP (Seems to be what we call floorChange in TFS)
    {
      2-bytes | lensHelp
    }
0x1F | FLAG_FULLBANK
0x20 | FLAG_IGNORELOOK
0x21 | FLAG_CLOTHES
   if FLAG_CLOTHES
    {
      2-bytes | clothesSlot
    }
0x22 | FLAG_MARKET
   if FLAG_MARKET
    {
      2-bytes | marketCategory
      2-bytes | marketTradeAs
      2-bytes | marketShowAs
      2-bytes | marketNameLength
      (marketNameLength) * 1-bytes | marketName
      2-bytes | marketRestrictProfession
      2-bytes | marketRestrictLevel
    }
0x23 | FLAG_DEFAULT_ACTION
   if FLAG_DEFAULT_ACTION
    {
      2-bytes | action
    }
0x24 | FLAG_WRAPPABLE
0x25 | FLAG_UNWRAPPABLE

To clarify:
  • Each ITEM_FLAG is 1-byte.
  • Some ITEM_FLAGS have PARAMETERS and these are typically 2-byte values.
  • The marketName byte size is completely dependent on the value of marketNameLength.
  • The value of marketName can be decoded into a string.
  • 0xFF indicates the ITEM_FLAG data is over and we move onto SPRITE_FLAGS.

SPRITE_FLAGS
Code:
1-byte | width
1-byte | height
   if width OR height > 1
   {
      1-bytes | exactSize
   }
1-byte | blendFrames
1-byte | divX
1-byte | divY
1-byte | divZ
1-byte | phases (Seems to be what we call animationLength in TFS)
   if phases > 1
   {
      2-bytes | phaseType(?)
      2-bytes | phaseLoopCount(?)
      2-bytes | phaseStartingFrame(?)
      (animationLength * 2) * 4-bytes | phaseFrameDurations(?) (seems to be frame minimum and maximum durations)
   }

To clarify:
  • There is an extra 1-byte after height when the width or height flags are greater than 1.
  • There an extra 6-bytes after animationLength when the animationLength flag is greater than 1 which seem to be some sort of *ANIMATION_FLAGS.
  • There is an extra (animationLength * 2) * 4-bytes after the *ANIMATION_FLAGS above, which seem to be related to their duration.

    *Here is a link to the Flash Client source regarding the phases/animation flags.
    *Here is a link to the ObjectBuilder source regarding the phases/animation flags.

SPRITES

Code:
4-bytes | sprite

*repeat for every sprite

To clarify:
  • Each sprite is 4-bytes
  • The number of sprites is equal to:
    Code:
    (divX * divY * divZ * animationLength)
Done.
This is then looped through all items.

PARSER
Here is a pastebin link to a primitive C# parser.

PARSER NOTES
Just a few things to take into consideration:
  1. The while loop seems to break prematurely (ItemID 22408~).
    This seems to be due to bad looping logic, because my programming is bad and I should feel bad.
    Code:
    while(b.BaseStream.Position != b.BaseStream.Length)

  2. The exactSize and phases/animation data is irrelevant for what I'm doing here, so it has simply been skipped:
    Code:
    // Skip the exactSize byte if it exists
    if (height > 1 | width > 1)
    {
        // Skip this byte
        b.ReadBytes(1);
    }
    
    // if animation exists, ignore the animationFlags
    if (alength > 1)
    {
        // Skip the 3, 2-byte animationFlags
        b.ReadBytes(6);
    
        // Skip the other animationDuration bytes that follow
        b.ReadBytes((alength * 2) * 4);
    }
Nonetheless it works effectively and will allow you to find items you need using its CLIENT ItemID.
Before editing any items, ensure you grab the items CLIENT ItemID first using an OTB Editor.

Edit the .DAT file using a hex editor by searching for one of its SPRITES (unique data).
Long-term solution would be integration into ObjectBuilder.
 
Last edited:
Really cool to see your learning process. I'm glad that you shared your achievements, and took your time to explain in detail everything you found while learning.
Hope to see you more around here, I knew nothing about .dat and now I feel like I thoroughly understand it for both clients (C++ and Flash).

Keep up the good work! :)
 
This is just great! Thank you so much for explaining all the process.
Will you split .spr into bytes too? It will be so interesting and valuable to create some sort of basic dat/spr editor in C# with you.
I was struggling with reverse engineering for a long time and always wanted to look inside spr/dat by myself, and as mentioned above from almost zero knowledge Ive started to getting it somehow!
Good work mate

EDIT:
So far Ive found about spr is that first 4 bytes are spr signature and next 4 is sprites count (2 if ver < 9.6) and next there are 4 bytes address for each sprite and when we go to that address we have 3 bytes for colour (red/green/blue), 2 bytes for sprite size (X) and then X bytes of actual sprite.

4B - spr signature
(ver < 9.6) ? 2B : 4B - sprites count
foreach(sprites count)
{
4B - sprite address
}
foreach(sprites count)
{
3B - sprite colours (rgb)
2B - sprite memory size (X)
XB - compressed sprite bitmap
}

EDIT2:
This is as far as I went by now
H7eOOhw.png
 
Last edited:
This is just great! Thank you so much for explaining all the process.
Will you split .spr into bytes too? It will be so interesting and valuable to create some sort of basic dat/spr editor in C# with you.
I was struggling with reverse engineering for a long time and always wanted to look inside spr/dat by myself, and as mentioned above from almost zero knowledge Ive started to getting it somehow!
Good work mate

EDIT:
So far Ive found about spr is that first 4 bytes are spr signature and next 4 is sprites count (2 if ver < 9.6) and next there are 4 bytes address for each sprite and when we go to that address we have 3 bytes for colour (red/green/blue), 2 bytes for sprite size (X) and then X bytes of actual sprite.

4B - spr signature
(ver < 9.6) ? 2B : 4B - sprites count
foreach(sprites count)
{
4B - sprite address
}
foreach(sprites count)
{
3B - sprite colours (rgb)
2B - sprite memory size (X)
XB - compressed sprite bitmap
}

EDIT2:
This is as far as I went by now
H7eOOhw.png

You're using a client .SPR though?
Editors and decoders already exist for that.
The trick is finding out how the flash client .png ATLAS files are processed.
The benefits of doing this would allow us to use the sprites in external applications like an editor.
As the sprite atlas files are just .PNG files, the decoding of them will be defined directly in the client itself.

To do this, you would start with decompiling the Tibia.swf file using a SWF decompiler.
 
Last edited:
Back
Top