Protocol Version 1111

jo3bingham · Mar 9, 2017

With this morning's update, CipSoft made some changes to their packet structure. They removed the Adler-32 checksum and replaced it with an incremental counter (it's still four bytes at the same index). This goes for both client and server packets. They also added support for compressing packets using the DEFLATE algorithm. The client doesn't take advantage of this because it doesn't need to. None of the packets being sent from the client would benefit from being compressed. However, packets being sent from the server can; especially map packets, and the fact the server can send multiple packets together in one (the client does not do this). But there are also packets being sent from the server that would not benefit from being compressed. This is where the incremental counter is used. The client checks the most-significant byte of the incremental counter, and, if it's set, then it knows the packet is compressed.

pseudo-code:

Code:

bool isCompressed = false;
auto sequenceNumber = msg.get<uint32_t>();
if ((sequenceNumber & 1 << 31) != 0) {
    isCompressed = true;
}

Don Daniello · Mar 9, 2017

Good work! Interesting where they are going with this. The checksum was rather redundant.

jo3bingham · Mar 9, 2017

Don Daniello said:
Good work! Interesting where they are going with this. The checksum was rather redundant.

My guess is to combat latency. Bigger packets mean more bandwidth being used which means higher latency. Though, they could do a better job at this by moving the map to the client side like most major MMOs do, but alas.

Tampek · Mar 10, 2017

this to protocolgame.cpp?

Flatlander · Mar 10, 2017

Jo3Bingham said:
My guess is to combat latency. Bigger packets mean more bandwidth being used which means higher latency. Though, they could do a better job at this by moving the map to the client side like most major MMOs do, but alas.

Tibia cannot have local maps, the game simply doesn't work that way. **Well technically it could, but it would be horrible**

Lets say I make a modded OTClient, and I put the OTBM in the otclient for each player, so instead of sending the map to players, I just send their positions and they load their local map.

Well, then what happens when someone shovels a hole, or shoots an Mwall, opens a door, or decorates their house? You have to send those changes to the player. How do you know that changes were made? You'd have to somehow compare the original map, to the current map, and send the differences.

But not only that, this gives players the ability to look at the map and find all the secrets. They can just open the OTBM and look through the entire map themselves, and find out all my secret hidden areas.

Also, when you start up your server, you load up the entire OTBM, which is sometimes a HUGE file. The players of tibia don't always have 2-3 extra gigs of ram to safely run a giant map, so you just increased the system requirements by a large amount.

All of this insane amount of work, to avoid sending a very small amount of network traffic.
The only times where a large amount of network traffic is sent, is when a player teleport (it sends an entire new screen). But if you walk 1 sqm north, it simply forgets the previous row to the south, and sends the new row to the north. (16 tiles).
And sending a tile, isn't sending artwork. All that is really sent is the itemid, type, count, position. So basically it sends less than 10 numbers for each item. Which is an incredibly small amount of data.

jo3bingham · Mar 10, 2017

Flatlander said:
Tibia cannot have local maps, the game simply doesn't work that way. **Well technically it could, but it would be horrible**

Lets say I make a modded OTClient, and I put the OTBM in the otclient for each player, so instead of sending the map to players, I just send their positions and they load their local map.

Well, then what happens when someone shovels a hole, or shoots an Mwall, opens a door, or decorates their house? You have to send those changes to the player. How do you know that changes were made? You'd have to somehow compare the original map, to the current map, and send the differences.

It would work exactly how it does now. When a player performs an action (shovels a hole, shoots a magic wall, opens a door, etc.) it notifies the server. The server determines if the action is permitted, then notifies the player (and any surrounding players/creatures if needed). You can edit the map in the client's memory right now, make it think there's a hole where there isn't, but you can't go down that hole because the server knows there isn't a hole there. How is that any different?

Flatlander said:
But not only that, this gives players the ability to look at the map and find all the secrets. They can just open the OTBM and look through the entire map themselves, and find out all my secret hidden areas.

Yes, but just because they can see these areas doesn't mean they can figure out how to access them. The map that is distributed with the client doesn't have to contain the same information that the master map on the server does. The client map doesn't have to store whether this tile is pickable, or that this lever controls that object, or where a teleport goes to. Besides, the map data from the 7.72 server files are public now, and map trackers have been around for quite some time. CipSoft's map isn't a surprise anymore.

Flatlander said:
Also, when you start up your server, you load up the entire OTBM, which is sometimes a HUGE file. The players of tibia don't always have 2-3 extra gigs of ram to safely run a giant map, so you just increased the system requirements by a large amount.

Have you ever done any type of programming before? Loading the entire map is terrible programming. CipSoft doesn't load the whole sprite file, they open it and load information as needed. Why wouldn't they do the same thing with the map? How do you think games like World of Warcraft work that have the map files locally? Do you actually think they load the whole map file? In case you don't know, the answer is no. No, they do not because that would be terrible and inefficient programming, and probably not even possible with the size of their map.

Flatlander said:
All of this insane amount of work, to avoid sending a very small amount of network traffic.
The only times where a large amount of network traffic is sent, is when a player teleport (it sends an entire new screen). But if you walk 1 sqm north, it simply forgets the previous row to the south, and sends the new row to the north. (16 tiles).
And sending a tile, isn't sending artwork. All that is really sent is the itemid, type, count, position. So basically it sends less than 10 numbers for each item. Which is an incredibly small amount of data.

I have seen login packets from CipSoft's game servers contain almost 10,000 bytes of data (mostly depending on what's on screen where you log in). That's quite a bit of data, and the client has to parse all of it. 16 tiles? Actually, it's 18. Yes, it's hard to believe, but if you look at a map packet, they send up to 18 tiles per x-axis, and 14 per y-axis. Also, it doesn't just send data about the floor you're on. If you're on the ground floor or higher, it sends data for the ground floor all the way up to the top floor (8 floors of data; floors 0-7). If you're underground it sends data for the two floors below you, your floor, and the two floors above you. Sure, it could be as little as 18 tiles, but it could be as many as 144 tiles. Each item contains as little as 3 bytes (id [2 bytes] and mark [1 byte]), or up to 5 bytes (id, mark, count/data [1 byte], and animation phase [1 byte]). That means you're looking at anywhere from 54 bytes to 720 bytes for that one movement (assuming each tile contained only one item). Now let's imagine a full map packet: 18 tiles wide, by 14 tiles high, by 8 floors; that's 2016 tiles. If every tile contained just one item object, that would come out to 10,080 bytes of data.

Flatlander · Mar 10, 2017

Jo3Bingham said:
It would work exactly how it does now. When a player performs an action (shovels a hole, shoots a magic wall, opens a door, etc.) it notifies the server. The server determines if the action is permitted, then notifies the player (and any surrounding players/creatures if needed). You can edit the map in the client's memory right now, make it think there's a hole where there isn't, but you can't go down that hole because the server knows there isn't a hole there. How is that any different?

Yes, but just because they can see these areas doesn't mean they can figure out how to access them. The map that is distributed with the client doesn't have to contain the same information that the master map on the server does. The client map doesn't have to store whether this tile is pickable, or that this lever controls that object, or where a teleport goes to. Besides, the map data from the 7.72 server files are public now, and map trackers have been around for quite some time. CipSoft's map isn't a surprise anymore.

Have you ever done any type of programming before? Loading the entire map is terrible programming. CipSoft doesn't load the whole sprite file, they open it and load information as needed. Why wouldn't they do the same thing with the map? How do you think games like World of Warcraft work that have the map files locally? Do you actually think they load the whole map file? In case you don't know, the answer is no. No, they do not because that would be terrible and inefficient programming, and probably not even possible with the size of their map.

I have seen login packets from CipSoft's game servers contain almost 10,000 bytes of data (mostly depending on what's on screen where you log in). That's quite a bit of data, and the client has to parse all of it. 16 tiles? Actually, it's 18. Yes, it's hard to believe, but if you look at a map packet, they send up to 18 tiles per x-axis, and 14 per y-axis. Also, it doesn't just send data about the floor you're on. If you're on the ground floor or higher, it sends data for the ground floor all the way up to the top floor (8 floors of data; floors 0-7). If you're underground it sends data for the two floors below you, your floor, and the two floors above you. Sure, it could be as little as 18 tiles, but it could be as many as 144 tiles. Each item contains as little as 3 bytes (id [2 bytes] and mark [1 byte]), or up to 5 bytes (id, mark, count/data [1 byte], and animation phase [1 byte]). That means you're looking at anywhere from 54 bytes to 720 bytes for that one movement (assuming each tile contained only one item). Now let's imagine a full map packet: 18 tiles wide, by 14 tiles high, by 8 floors; that's 2016 tiles. If every tile contained just one item object, that would come out to 10,080 bytes of data.

Sigh, ok you are right that the X axis is 18 tiles not 16, I was giving it as an example, that it doesn't re-send the entire screen every time you move. The client is rather efficient.

And yes, when you first log in, it sends the most packets at once, because not only does it load the entire screen, it sends your inventory and character info all at once.

I knew when I was typing it, some of these counter-arguments may pop up. So instead of arguing every detail, I'll simply argue the main point.
Let's say I designed a perfect system, where it only pulls your screen from the local map, and you can't open it to "cheat" and see secret areas.

My original problem still exists.
Here are the possible senarios:
Scenario 1

Player A is standing Next to Player B:
Player A uses a machete and cuts grass.
Server sends item update to Player B.
Everything works.

Scenario 2

Player A is off screen from Player B:
Player A uses a machete and cuts grass.
Player B is not on screen when this happens, so his map is not updated.
Player B enters the screen, and the grass is uncut.

--Possible solution?

TFS still loads the map, and keeps it updated when players make changes.
TFS has to compare the original map, to the now edited map, and send the difference to all players when loading new tiles (Such as teleporting and walking)
To me, this is a wildy stupid way to reduce the amount of bytes being sent to a client.

You are trading one thing, for another. You trade network performance, for server/client performance. This isn't an upgrade, it is a side-grade at best.

jo3bingham · Mar 10, 2017

Flatlander said:
Sigh, ok you are right that the X axis is 18 tiles not 16, I was giving it as an example, that it doesn't re-send the entire screen every time you move. The client is rather efficient.

And yes, when you first log in, it sends the most packets at once, because not only does it load the entire screen, it sends your inventory and character info all at once.

I knew when I was typing it, some of these counter-arguments may pop up. So instead of arguing every detail, I'll simply argue the main point.
Let's say I designed a perfect system, where it only pulls your screen from the local map, and you can't open it to "cheat" and see secret areas.

My original problem still exists.
Here are the possible senarios:
Scenario 1

Player A is standing Next to Player B:

Player A uses a machete and cuts grass.

Server sends item update to Player B.

Everything works.

Scenario 2

Player A is off screen from Player B:

Player A uses a machete and cuts grass.

Player B is not on screen when this happens, so his map is not updated.

Player B enters the screen, and the grass is uncut.

--Possible solution?

TFS still loads the map, and keeps it updated when players make changes.

TFS has to compare the original map, to the now edited map, and send the difference to all players when loading new tiles (Such as teleporting and walking)

To me, this is a wildy stupid way to reduce the amount of bytes being sent to a client.

You are trading one thing, for another. You trade network performance, for server/client performance. This isn't an upgrade, it is a side-grade at best.

There wouldn't need to be any change in server performance. The server would generate map packets as it currently does, but it wouldn't have to include static objects in the packet (grounds, walls, roofs, water, borders, etc.).

Flatlander · Mar 10, 2017

Jo3Bingham said:
There wouldn't need to be any change in server performance. The server would generate map packets as it currently does, but it wouldn't have to include static objects in the packet (grounds, walls, roofs, water, borders, etc.).

The server would have to compare the original map, to the edited map, and only update the player's local map when they enter the screen with the difference.

This in itself would impact server performance.

gunz · Mar 30, 2017

Is is possible to get chunk size for the deflate algorithm ? I would like to implement this feature to OTClient.

EDIT: fixed, I meant windowBits size which is default (15)

gunz · Apr 7, 2017

Code to decompress packets using zlib in OTClient:

Code:

void InputMessage::decompress()
{
    z_stream zs;                        // z_stream is zlib's control structure
    memset(&zs, 0, sizeof(zs));

    if (inflateInit2(&zs, 15) != Z_OK)
        throw(std::runtime_error("inflateInit failed while decompressing."));

    zs.next_in = (Bytef*)m_buffer + m_readPos;
    zs.avail_in = getUnreadSize();

    int ret;
    char outbuffer[32768];
    std::string outstring;

    // get the decompressed bytes blockwise using repeated calls to inflate
    do {
        zs.next_out = reinterpret_cast<Bytef*>(outbuffer);
        zs.avail_out = sizeof(outbuffer);

        ret = inflate(&zs, Z_BLOCK);

        if (outstring.size() < zs.total_out) {
            outstring.append(outbuffer,
                             zs.total_out - outstring.size());
        }
    } while (ret == Z_OK);


    inflateEnd(&zs);

    if (ret != Z_STREAM_END) {          // an error occurred that was not EOF
        std::ostringstream oss;
        oss << "Exception during zlib decompression: (" << ret << ") "
            << zs.msg;
        throw(std::runtime_error(oss.str()));
    }

    setMessageSize(m_readPos + outstring.size() + 1);
    getU8(); // don't know why but works
    memcpy(m_buffer + m_readPos, outstring.data(), outstring.size());
}

the readChecksum function has to be edited to check whether message is compressed or not and also for removing the adler32 checksum. Decompression should be done after XTEA decryption.

jo3bingham · Apr 8, 2017

gunz said:
Decompression should be done after XTEA decryption.

Thanks for pointing that out. I forgot to mention that in my post. Also, I've noticed CipSoft has started taking advantage of the compression in client packets being sent to the server. I can't think of any packets off the top of my head that would benefit from it, but it's there.

EDIT - I lied. They're not using the compression, but some times their sequence number has the most significant bit set.

comedinha · Jul 12, 2017

Only need change to send the sequence
11.32 Support · malucooo/Otxserver-New@beda91b · GitHub

Marcus · Jul 12, 2017

For the argument about loading a local map, and Flat keeping the argument about changes in player A map wont happen in player B map.

Did you forget that counter strike exist?
their files is local, if a player runs away from you for a great distance, and open a door, when you finally get there... there door is still open (unless it auto-closes within few seconds).

And I'm gonna be honest, I've never actually looked into online games saving their files locally to spare their networks, but I do know a lot of games which uses it, and it causes no problems between the players.
GTA -online for example..
You're not having the entire map open because really.. having a country sized map open... no way..
And you can literally be a whole town away from player B (assuming you are player A), and player B does something that change the map, and you fly there in a jet or something within a minute or two, and you'll see the changed map as well

Protocol Version 1111

jo3bingham

Excellent OT User

Don Daniello

/root

jo3bingham

Excellent OT User

Tampek

ECHO 'Zarkonia.online';

Flatlander

Species Developer

jo3bingham

Excellent OT User

Flatlander

Species Developer

jo3bingham

Excellent OT User

Flatlander

Species Developer

gunz

GunzOT Owner

gunz

GunzOT Owner

jo3bingham

Excellent OT User

comedinha

Well-Known Member

Marcus

User.postCount++;

Similar threads