Feature Cam system

[kor]

Hello.

I would like to share a system I wrote about 5-6 years ago.

Spoiler: code

cams.diff

GitHub Gist: instantly share code, notes, and snippets.

gist.github.com

diff --git a/movies/.gitignore b/movies/.gitignore
new file mode 100644
index 00000000..1bc8d0f5
--- /dev/null
+++ b/movies/.gitignore
@@ -0,0 +1,3 @@
+*.byn
+*.cam
+!.gitignore
diff --git a/src/protocol.cpp b/src/protocol.cpp
index 5fb09683..de7eab83 100644
--- a/src/protocol.cpp
+++ b/src/protocol.cpp
@@ -60,8 +60,9 @@ bool XTEA_decrypt(NetworkMessage& msg, const xtea::round_keys& key)
 
 }
 
-void Protocol::onSendMessage(const OutputMessage_ptr& msg) const
+void Protocol::onSendMessage(const OutputMessage_ptr& msg)
 {
+    handleMessage(msg->getOutputBuffer(), msg->getLength());
     if (!rawMessages) {
         msg->writeMessageLength();
 
@@ -111,3 +113,80 @@ uint32_t Protocol::getIP() const
 
     return 0;
 }
+
+uint8_t DELAY_ID = 0x65;
+uint8_t PACKET_ID = 0x66;
+
+void Protocol::startCam(uint32_t guid, std::string name)
+{
+    std::stringstream path;
+    path << "movies//packets_" << guid << "_" << OTSYS_TIME() << "_" << name << ".byn";
+    camPath = path.str().c_str();
+
+    if (cam) {
+        cam.close();
+    }
+
+    cam.open(camPath, std::fstream::out | std::fstream::binary | std::fstream::trunc);
+
+    if (!cam.good()) {
+        return;
+    }
+
+    camActive = 3;
+    timeLast = OTSYS_TIME();
+}
+
+void Protocol::handleMessage(const uint8_t* buffer, const uint16_t msgSize)
+{
+    if (isConnectionExpired() || camActive != 3 || !cam.good()) {
+        return;
+    }
+
+    timeNow = OTSYS_TIME();
+    uint32_t delay = timeNow - timeLast;
+    timeLast = timeNow;
+    uint32_t i;
+    uint8_t headerDelay[3] = {DELAY_ID, (uint8_t) (delay), (uint8_t) (delay >> 8)};
+
+    for (i = 0; i < 3; i++) {
+        cam.put(headerDelay[i]);
+    }
+
+    uint8_t headerPacket[3] = {PACKET_ID, (uint8_t) (msgSize), (uint8_t) (msgSize >> 8)};
+
+    for (i = 0; i < 3; i++) {
+        cam.put(headerPacket[i]);
+    }
+
+    for (i = 0; i < msgSize; i++) {
+        cam.put(buffer[i]);
+    }
+
+    cam.flush();
+}
diff --git a/src/protocol.h b/src/protocol.h
index 8bd7d6f2..e5a91fe5 100644
--- a/src/protocol.h
+++ b/src/protocol.h
@@ -22,6 +22,7 @@
 
 #include "connection.h"
 #include "xtea.h"
+#include <fstream>
 
 class Protocol : public std::enable_shared_from_this<Protocol>
 {
@@ -35,7 +36,7 @@ class Protocol : public std::enable_shared_from_this<Protocol>
 
         virtual void parsePacket(NetworkMessage&) {}
 
-        virtual void onSendMessage(const OutputMessage_ptr& msg) const;
+        virtual void onSendMessage(const OutputMessage_ptr& msg);
         void onRecvMessage(NetworkMessage& msg);
         virtual void onRecvFirstMessage(NetworkMessage& msg) = 0;
         virtual void onConnect() {}
@@ -87,6 +88,15 @@ class Protocol : public std::enable_shared_from_this<Protocol>
 
         virtual void release() {}
 
+        void startCam(uint32_t guid, std::string name);
+        void handleMessage(const uint8_t* buffer, const uint16_t msgSize);
+        std::ofstream cam;
+        int8_t camActive;
+        std::string camPath;
+        uint32_t timeLast;
+        uint32_t timeNow;
+
     private:
         friend class Connection;
 
diff --git a/src/protocolgame.cpp b/src/protocolgame.cpp
index 83fce071..8cd31ae2 100644
--- a/src/protocolgame.cpp
+++ b/src/protocolgame.cpp
@@ -234,6 +235,7 @@ void ProtocolGame::login(const std::string& name, uint32_t accountId, OperatingS
         player->lastIP = player->getIP();
         player->lastLoginSaved = std::max<time_t>(time(nullptr), player->lastLoginSaved + 1);
         acceptPackets = true;
+        Protocol::startCam(player->getGUID(), player->getName());
     } else {
         if (eventConnect != 0 || !g_config.getBoolean(ConfigManager::REPLACE_KICK_ON_LOGIN)) {
             //Already trying to connect
@@ -244,9 +246,11 @@ void ProtocolGame::login(const std::string& name, uint32_t accountId, OperatingS
         if (foundPlayer->client) {
             foundPlayer->disconnect();
             foundPlayer->isConnecting = true;
+            Protocol::startCam(foundPlayer->getGUID(), foundPlayer->getName());
 
             eventConnect = g_scheduler.addEvent(createSchedulerTask(1000, std::bind(&ProtocolGame::connect, getThis(), foundPlayer->getID(), operatingSystem)));
         } else {
+            Protocol::startCam(foundPlayer->getGUID(), foundPlayer->getName());
             connect(foundPlayer->getID(), operatingSystem);
         }
     }

Initially it was on version 0.4, and later I moved it to 1.2 as well. During this time, it generated about 200 GB of recordings, and thanks to them, such films as https://www.youtube.com/playlist?list=PLwGkOisgt0UPKmJiNJvNp2BmQAw55Z1Bi or

(recording from a dedicated player, such as recordings generated by the system) could be created. The code is very simple and has not generated any crash since its creation, but it was tested with 20, max 30 simultaneously logged clients.

If I will find any information, I'll post them

Here I am sharing it with you today, because at the same time I would like to ask if with 100+ simultaneous clients it will not load the server in any way, or if it does not have any holes through which memory may "leak"?

It is slightly better (in my opinion) than the systems presented in Feature - [TFS 1.x+] Print all send network packets in server console (https://otland.net/threads/tfs-1-x-print-all-send-network-packets-in-server-console.279699/post-2715116) or in Cam System 10.98 (TFS 1.3) (https://otland.net/threads/cam-system-10-98-tfs-1-3.265913/post-2570020), because the packets are written directly to the file, not to the server's memory, which makes it robust in case of server crash for any reason.

The structure is very similar to the BynaCam format (Google Code Archive - Long-term storage for Google Code Project Hosting. (https://code.google.com/archive/p/bynacam/source/default/source), Player.cs file, GetFileMessage method) with the difference that I store the interval between packets as delay, and timestamp was used there and the delay was calculated from the difference of timestamps.

 

[danilopucci]

Amazing! Thanks for this contribution.

The only thing that I would say about handling more players is that the program is making disk IO operations on the same thread as the server process messages, this would not scale with more players online. Maybe from 20 to 100 is still OK.

To make it more scalable, you could create another thread which the job would be just to flush data from memory to disk (which is what the Gesior's does). And you can control the interval that the thread would flush data from memory to disk, this way in worst scenario, you would only loss some seconds of data in crash.

 

[StreamSide]

well done, great job as always

 

[kor]

you would only loss some seconds of data in crash

0.4 was (is) quite buggy, so I had many other crashes where those last seconds was priceless to find out what causes crash

Also, beside that operations are in the same thread as you noticed, I also don't like how I'm writing byte one by one with cam.put instead of putting whole packet at once, but I never didn't have enough time to improve it.

 

[danilopucci]

Also, beside that operations are in the same thread as you noticed, I also don't like how I'm writing byte one by one with cam.put instead of putting whole packet at once, but I never didn't have enough time to improve it.

Oh nice, I didn't notice that!

Well, I did not tested it, but this may work fine:

void Protocol::handleMessage(const uint8_t* buffer, const uint16_t msgSize)
{
    const int HEADER_SIZE = 3;
 
    if (isConnectionExpired() || camActive != 3 || !cam.good()) {
        return;
    }
 
    timeNow = OTSYS_TIME();
    uint32_t delay = timeNow - timeLast;
    timeLast = timeNow;

    uint8_t headerDelay[HEADER_SIZE] = {DELAY_ID, static_cast<uint8_t>(delay), static_cast<uint8_t>(delay >> 8)};
    uint8_t headerPacket[HEADER_SIZE] = {PACKET_ID, static_cast<uint8_t>(msgSize), static_cast<uint8_t>(msgSize >> 8)};
 
    cam.write(headerDelay, HEADER_SIZE);
    cam.write(headerPacket, HEADER_SIZE);
    cam.write(buffer, msgSize);
                                    
    cam.flush();
}

 

[Evolunia]

Very cool! But how would one play the .byn files?

 

[Evil Puncker]

Very cool! But how would one play the .byn files?

GitHub - exoticprgmr/bynacam: BynaCam Reloaded v2 is free, open source software to capture and play Tibia videos by network data packets in CAM file format. Software is based on original BynaCam source code. Designed exclusively for Tibia 8.6

BynaCam Reloaded v2 is free, open source software to capture and play Tibia videos by network data packets in CAM file format. Software is based on original BynaCam source code. Designed exclusivel...

github.com

I guess

 

[Evolunia]

GitHub - exoticprgmr/bynacam: BynaCam Reloaded v2 is free, open source software to capture and play Tibia videos by network data packets in CAM file format. Software is based on original BynaCam source code. Designed exclusively for Tibia 8.6

BynaCam Reloaded v2 is free, open source software to capture and play Tibia videos by network data packets in CAM file format. Software is based on original BynaCam source code. Designed exclusivel...

github.com

I guess

There's some recordings shown with otclient on his youtube channel! And I'd guess that software would only work with really old clients, unless it was updated alot

 

[kor]

There are some options - someone can adapt bynacam repository to read different structure of cams produced by system I shared (but then you need to download the recording files to the disk to play them). I'm using GitHub - Milice/forgottenloginserver (https://github.com/Milice/forgottenloginserver) as my login server (I separated machines with login and game servers), so I've added "fake" game protocol there which is basically reading packet by packet from file on server and then sends to connected client with delays as a sleep commands. That way I can connect to cam server from any client I want - Cipsoft or OTC.
I had in plans to move cam "protocol" to Node which will be much easier to maintain and develop, but never had enough time for this.

 

[Mateus Robeerto]

Does TFS 1.5 work?

 

[Evolunia]

There are some options - someone can adapt bynacam repository to read different structure of cams produced by system I shared (but then you need to download the recording files to the disk to play them). I'm using GitHub - Milice/forgottenloginserver (https://github.com/Milice/forgottenloginserver) as my login server (I separated machines with login and game servers), so I've added "fake" game protocol there which is basically reading packet by packet from file on server and then sends to connected client with delays as a sleep commands. That way I can connect to cam server from any client I want - Cipsoft or OTC.
I had in plans to move cam "protocol" to Node which will be much easier to maintain and develop, but never had enough time for this.

I see!

I played around with it a little bit, I think the easiest solution for me was to make it write the same packet structure that otclientv8 uses for their records, this worked to replay records in otcv8, I also had to add some starting packet "< 0 0000000000000000000000", not sure where it's supposed to come from

void Protocol::handleMessage(const char* buffer, const uint16_t msgSize)
{
    if (isConnectionExpired() || camActive != 3 || !cam.good()) {
        return;
    }

    timeNow = OTSYS_TIME();
    uint32_t delay = timeNow - timeLast;

    cam << "< " << (timeNow - timeLast) << " ";
    for (uint16_t i = 0; i < msgSize; i++) {
        cam << std::setfill('0') << std::setw(2) << std::hex << static_cast<uint16_t>(static_cast<uint8_t>(buffer[i]));
    }
    cam << std::dec << "\n";

    cam.flush();
}

 

[MikaelJohannese]

Does TFS 1.5 work?