C++ death crashlog

[Darius93]

Hello, I had a server crash, but I had the debugger running, and here’s the log:

https://pastebin.com/raw/FAjqE2Vr

I’ll add that it happened during a character’s death.
How can I fix it, and where did the error occur?
I don’t even know how to reproduce it, because the character has died many times before without issues...


ChatGPT told me:
Your application (TFS, meaning The Forgotten Server or its fork) tried to send data (for example, to a MySQL database through libmysqlclient), but the connection was lost.


And my question is — why?

 

[Gesior.pl]

Your application (TFS, meaning The Forgotten Server or its fork) tried to send data (for example, to a MySQL database through libmysqlclient), but the connection was lost.


And my question is — why?

That may be true. It's lost, because TFS 1.x has 2 connections to MySQL/MariaDB. One used by 99% of code (synchronous - load players, save player etc.) and one used by some Lua scripts and some parts of Tibia Market (asynchronous).
MySQL has a default timeout for inactive connections and drops them after reaching it. It's controlled by 2 MySQL config variables:

• interactive_timeout
• wait_timeout

and both are set to 8 hours by default (28800 seconds): MySQL Timeout Explained (https://www.bytebase.com/blog/mysql-timeout/#interactivetimeout-and-waittimeout)

It may also happen on server with just 1 connection to MySQL (ex. TFS 0.4), when there is no players for 8 hours and no Lua script that loads/saves something in database, but in your case it crashed on DatabaseTasks::runTask, which is function for 'asynchronous queries'.

On Linux with MariaDB you can set them in /etc/mysql/mariadb.conf.d/50-server.conf. On other systems or using MySQL (in place of MariaDB) - IDK, ask some AI, it should be easy to edit.

THERE IS ALSO OTHER POSSIBILITY // it may be unrelated coincidence
In your crash report there are 2 C++ threads executing MySQL queries at same time:

• Thread 4 (async - database thread) executing DELETE FROM `player_deaths` WHERE `player_id` = 13 ORDER BY `time` LIMIT 1
• Thread 2 (sync - dispatcher thread) executing UPDATE `players` SET `level` = 248,`group_id` = 1,`vocation` = 7,`health` = 2795,`healthmax` = 2795,`experience` = 250485922,`lookbody` = 102,`lookfeet` = 95,`lookhead` = 132,`looklegs` = 116,`looktyp`

They use other Database objects, but it's possible that they use same MySQL connection.
Sooome old servers had a bug that made 2 threads use MySQL connection, but there was only 1 connection and no mutex to prevent 2 threads from using it in same time.
On latest TFS it's implemented with databaseLock:

forgottenserver/src/database.cpp at master · otland/forgottenserver

A free and open-source MMORPG server emulator written in C++ - otland/forgottenserver

github.com

forgottenserver/src/database.h at master · otland/forgottenserver

A free and open-source MMORPG server emulator written in C++ - otland/forgottenserver

github.com

On TFS 1.0 it was also implemented with database_lock:

forgottenserver/src/database.cpp at 1.0 · otland/forgottenserver

A free and open-source MMORPG server emulator written in C++ - otland/forgottenserver

github.com

forgottenserver/src/database.h at 1.0 · otland/forgottenserver

A free and open-source MMORPG server emulator written in C++ - otland/forgottenserver

github.com

I would increase default timeout for inactive connections in MySQL to value like 30 days and wait for next crash - if it happens.

 

[Darius93]

If I understood correctly, I just need to add a short line in C++ that will make the server ignore this error, right?


Then it will only show the error in the console instead of crashing the server, and it will try to execute the query again.


Am I thinking correctly? And for extra safety, I can also increase the MySQL connection timeout to 24 hours, for example?


Would that be enough to make sure the problem doesn’t happen again?

Im use TFS 1.2

 

[Gesior.pl]

And for extra safety, I can also increase the MySQL connection timeout to 24 hours, for example?

This should work, if it's problem with inactive connection. If OTS restart is every 24h, I would set it to 25h.

If I understood correctly, I just need to add a short line in C++ that will make the server ignore this error, right?

IDK what kind of line. Maybe some try { } catch, but IDK what exception to catch. Problem is that MySQL server closes TCP/socket connection for inactivity, but it does not notify OTS that it closes connection, so OTS crashes next time it tries to execute anything using that connection.

Would that be enough to make sure the problem doesn’t happen again?

If it's inactivity problem, yes.
If it's some '2 threads use same MySQL connection' problem, then it won't help and you will have to find why it happens and how to fix it.

If it crashes again after 'inactivity' change to 25h, I would replace database/databasetask code with TFS 'master' code.

 

[Darius93]

This should work, if it's problem with inactive connection. If OTS restart is every 24h, I would set it to 25h.

IDK what kind of line. Maybe some try { } catch, but IDK what exception to catch. Problem is that MySQL server closes TCP/socket connection for inactivity, but it does not notify OTS that it closes connection, so OTS crashes next time it tries to execute anything using that connection.

If it's inactivity problem, yes.
If it's some '2 threads use same MySQL connection' problem, then it won't help and you will have to find why it happens and how to fix it.

If it crashes again after 'inactivity' change to 25h, I would replace database/databasetask code with TFS 'master' code.

Overall, I didn’t change anything — I logged into the server around 10 PM, and I’ve just logged in again now — I’m not sure if logging in also counts and whether any database queries are executed then?


But I went in and died, and the server still didn’t crash — instead, this appeared in the console:

[Error - mysql_real_query] Query: DELETE FROM `player_deaths` ...
Message: The client was disconnected by the server because of inactivity. See wait_timeout and interactive_timeout for configuring this behavior.

Is that a sign that it’s actually about the second option you mentioned — that there are two C++ threads executing MySQL queries?

 

[Gesior.pl]

Is that a sign that it’s actually about the second option you mentioned — that there are two C++ threads executing MySQL queries?

Hard to tell. Maybe it uses 2 connections, but if this error - for which you get message in console this time - appears, when other thread is processing query, it somehow destroys some "state" in MySQL library and crashes other query.

There are a lot of bugs in TFS 'master', but database code looks pretty good and way better than old code. If you know how to copy these classes and adjust them to work with TFS 1.2*, then you can try to fix that problem by using TFS master database code.

*TFS master uses string_view instead of string in a lot of places, you will have to rewrite parts of database code to make it compatible - it's easier that way, than adding string_view everywhere in TFS 1.2 source.

 

[Darius93]

Hard to tell. Maybe it uses 2 connections, but if this error - for which you get message in console this time - appears, when other thread is processing query, it somehow destroys some "state" in MySQL library and crashes other query.

There are a lot of bugs in TFS 'master', but database code looks pretty good and way better than old code. If you know how to copy these classes and adjust them to work with TFS 1.2*, then you can try to fix that problem by using TFS master database code.

*TFS master uses string_view instead of string in a lot of places, you will have to rewrite parts of database code to make it compatible - it's easier that way, than adding string_view everywhere in TFS 1.2 source.

Alright, so I did as you said — I increased the default timeout for inactive connections in MySQL to something like 30 days and will wait for the next crash — if it happens.
We’ll see in a few days

 

[Darius93]

It's been online for a few days without any error - so I think the issue is resolved, thank you Gesior!