• There is NO official Otland's Discord server and NO official Otland's server list. The Otland's Staff does not manage any Discord server or server list. Moderators or administrator of any Discord server or server lists have NO connection to the Otland's Staff. Do not get scammed!

TFS 0.X Cast problem / server freeze

GSMaster

Why? for money
Joined
Oct 26, 2008
Messages
169
Solutions
1
Reaction score
10
Location
HKS <3
Good morning, i added this cast to TFS (0.4): Comparing otland:master...CastSystem:master · otland/forgottenserver (https://github.com/otland/forgottenserver/compare/master...CastSystem:master)

and unfortunately sometimes the server freezes... it looks like packages are overlapping? (MSG / NetworkMessage)
- this is just like any player has an active cast (Viewer [3] is casting viewer)

GDB log:
C++:
 -  Thread 1 (Thread 0x7ffff4c74980 (LWP 3652)):
 -  #0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
 -  No locals.
 -  #1  0x00007ffff764d7d1 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x55555582d440 <OutputMessagePool::getInstance()::instance+96>) at ../nptl/pthread_mutex_lock.c:115
 -          ignore1 = <optimized out>
 -          ignore2 = <optimized out>
 -          ignore3 = <optimized out>
 -          id = 3652
 -          type = <optimized out>
 -          __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
 -          id = <optimized out>
 -  #2  0x00005555556fdd22 in boost::recursive_mutex::lock (this=0x55555582d440 <OutputMessagePool::getInstance()::instance+96>) at /usr/include/boost/thread/pthread/recursive_mutex.hpp:108
 -          __PRETTY_FUNCTION__ = "void boost::recursive_mutex::lock()"
 -  #3  boost::unique_lock<boost::recursive_mutex>::lock (this=<synthetic pointer>) at /usr/include/boost/thread/lock_types.hpp:346
 -  No locals.
 -  #4  boost::unique_lock<boost::recursive_mutex>::unique_lock (m_=..., this=<synthetic pointer>) at /usr/include/boost/thread/lock_types.hpp:124
 -  No locals.
 -  #5  OutputMessagePool::getOutputMessage (this=this@entry=0x55555582d3e0 <OutputMessagePool::getInstance()::instance>, protocol=protocol@entry=0x555555a1f420, autosend=autosend@entry=true) at outputmessage.cpp:146
 -          lockClass = {m = 0x55555582d440 <OutputMessagePool::getInstance()::instance+96>, is_locked = <optimized out>}
 -          outputmessage = <optimized out>
 -  #6  0x000055555571d350 in Protocol::getOutputBuffer (this=this@entry=0x555555a1f420, size=<optimized out>) at outputmessage.h:160
 -  No locals.
 -  #7  0x000055555571fc47 in ProtocolGame::writeToOutputBuffer (this=this@entry=0x555555a1f420, msg=...) at networkmessage.h:124
 -          out = {px = 0x7fffba16c440, pn = {pi_ = 0x5555556de2c8 <NetworkMessage::AddString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+106>}}
 -  #8  0x0000555555725421 in ProtocolGame::sendChannelMessage (this=this@entry=0x555555a1f420, author="Viewer [3]", text="power up", type=type@entry=SPEAK_CHANNEL_Y, channel=channel@entry=16 '\020') at protocolgame.cpp:3372
 -          msg = {_vptr.NetworkMessage = 0x55555581ce70 <vtable for NetworkMessage+16>, m_RealBuf = "\000\000\000\000\000\000\000\000\252\000\000\000\000\a\000Viewer [3]\000\000\a\020\000\b\000power uper upy danADE 20%knnie se muere tools1008]: premuim players. Say: free.", '\000' <repeats 20 times>, "D\016\000\000\000\000\000\000@ԂUUU\000\000\000\000\000\000\000\000\000\000\342\356d\367\377\177", '\000' <repeats 34 times>..., m_MsgBuf = 0x7fffffff9618 "", m_MsgSize = 29, m_ReadPos = 37, m_overrun = false}
 -  #9  0x000055555572e901 in ProtocolGame::parseSay (this=this@entry=0x5555559315c0, msg=...) at /usr/include/c++/8/bits/basic_string.h:936
 -          client = @0x7fff1b4c8920: 0x555555a1f420
 -          __for_range = std::__cxx11::list = {[0] = 0x555555a1f420, [1] = 0x55555598bba0, [2] = 0x5555558c1ce0, [3] = 0x5555559315c0}
 -          __for_begin = <optimized out>
 -          __for_end = <optimized out>
 -          receiver = ""
 -          channelId = <optimized out>
 -          type = <optimized out>
 -          text = "power up"
 -  #10 0x00005555557303a7 in ProtocolGame::parsePacket (this=0x5555559315c0, msg=...) at protocolgame.cpp:707
 -          recvbyte = <optimized out>
 -  #11 0x000055555571de38 in Protocol::onRecvMessage (this=0x5555559315c0, msg=...) at protocol.cpp:73
 -  No locals.
 -  #12 0x00005555555bb957 in Connection::parsePacket (this=0x5555559108e0, error=...) at connection.cpp:535
 -          checksum = <optimized out>
 -          len = <optimized out>
 -          recvChecksum = 786040308
 -          __PRETTY_FUNCTION__ = "void Connection::parsePacket(const boost::system::error_code&)"
 -  #13 0x00005555555c6761 in boost::_mfi::mf1<void, Connection, boost::system::error_code const&>::call<boost::shared_ptr<Connection>, boost::system::error_code const> (b1=..., u=..., this=0x7fffffffdb38) at /usr/include/boost/smart_ptr/shared_ptr.hpp:964
 -  No locals.
 -  #14 boost::_mfi::mf1<void, Connection, boost::system::error_code const&>::operator()<boost::shared_ptr<Connection> > (a1=..., u=..., this=0x7fffffffdb38) at /usr/include/boost/bind/mem_fn_template.hpp:171
 -          p = 0x0
 -          p = <optimized out>
 -  #15 boost::_bi::list2<boost::_bi::value<boost::shared_ptr<Connection> >, boost::arg<1> (*)()>::operator()<boost::_mfi::mf1<void, Connection, boost::system::error_code const&>, boost::_bi::rrlist2<boost::system::error_code const&, unsigned long> > (a=<synthetic pointer>..., f=..., this=0x7fffffffdb48) at /usr/include/boost/bind/bind.hpp:319
 -  No locals.
 -  #16 boost::_bi::bind_t<void, boost::_mfi::mf1<void, Connection, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<Connection> >, boost::arg<1> (*)()> >::operator()<boost::system::error_code const&, unsigned long> (a2=<optimized out>, a1=..., this=0x7fffffffdb38) at /usr/include/boost/bind/bind.hpp:1318
 -          a = <optimized out>
 -          a = <optimized out>
 -  #17 boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf1<void, Connection, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<Connection> >, boost::arg<1> (*)()> > >::operator() (this=0x7fffffffdb10, ec=..., bytes_transferred=<optimized out>, start=<optimized out>) at /usr/include/boost/asio/impl/read.hpp:285
 -          max_size = <optimized out>
 -  #18 0x00005555555c6a61 in boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf1<void, Connection, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<Connection> >, boost::arg<1> (*)()> > >, boost::system::error_code, unsigned long>::operator() (this=0x7fffffffdb10) at /usr/include/boost/asio/detail/bind_handler.hpp:162
 -  No locals.
 -  #19 boost::asio::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf1<void, Connection, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<Connection> >, boost::arg<1> (*)()> > >, boost::system::error_code, unsigned long> > (function=...) at /usr/include/boost/asio/handler_invoke_hook.hpp:69
 -  No locals.
 -  #20 boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf1<void, Connection, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<Connection> >, boost::arg<1> (*)()> > >, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf1<void, Connection, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<Connection> >, boost::arg<1> (*)()> > > (context=..., function=...) at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
 -  No locals.
 -  #21 boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf1<void, Connection, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<Connection> >, boost::arg<1> (*)()> > >, boost::system::error_code, unsigned long>, boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf1<void, Connection, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<Connection> >, boost::arg<1> (*)()> > > (this_handler=0x7fffffffdb10, function=...) at /usr/include/boost/asio/impl/read.hpp:338
 -  No locals.
 -  #22 boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf1<void, Connection, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<Connection> >, boost::arg<1> (*)()> > >, boost::system::error_code, unsigned long>, boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf1<void, Connection, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<Connection> >, boost::arg<1> (*)()> > > > (context=..., function=...) at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
 -  No locals.
 -  #23 boost::asio::detail::handler_work<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf1<void, Connection, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<Connection> >, boost::arg<1> (*)()> > >, boost::asio::system_executor>::complete<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf1<void, Connection, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<Connection> >, boost::arg<1> (*)()> > >, boost::system::error_code, unsigned long> > (handler=..., function=..., this=<synthetic pointer>) at /usr/include/boost/asio/detail/handler_work.hpp:82
 -  No locals.
 -  #24 boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf1<void, Connection, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<Connection> >, boost::arg<1> (*)()> > > >::do_complete (owner=0x55555587c9f0, base=<optimized out>) at /usr/include/boost/asio/detail/reactive_socket_recv_op.hpp:122
 -          b = <optimized out>
 -          o = <optimized out>
 -          p = {h = 0x7fffffffdb10, v = 0x0, p = 0x0}
 -          w = <optimized out>
 -          handler = {handler_ = {<boost::asio::detail::base_from_completion_cond<boost::asio::detail::transfer_all_t>> = {<No data fields>}, stream_ = @0x555555c05a40, buffers_ = {<boost::asio::detail::consuming_single_buffer<boost::asio::mutable_buffers_1>> = {buffer_ = {<boost::asio::mutable_buffer> = {data_ = 0x5555559108fa, size_ = 20}, <No data fields>}, total_consumed_ = 20}, <No data fields>}, start_ = 0, handler_ = {f_ = {f_ = (void (Connection::*)(Connection * const, const boost::system::error_code &)) 0x5555555bb724 <Connection::parsePacket(boost::system::error_code const&)>}, l_ = {<boost::_bi::storage2<boost::_bi::value<boost::shared_ptr<Connection> >, boost::arg<1> (*)()>> = {<boost::_bi::storage1<boost::_bi::value<boost::shared_ptr<Connection> > >> = {a1_ = {t_ = {px = 0x5555559108e0, pn = {pi_ = 0x555555c99040}}}}, <No data fields>}, <No data fields>}}}, arg1_ = {m_val = 0, m_cat = 0x7ffff7d09070}, arg2_ = 20}
 -  #25 0x0000555555758d94 in boost::asio::detail::scheduler_operation::complete (bytes_transferred=0, ec=..., owner=0x55555587c9f0, this=0x555555b54a70) at /usr/include/boost/asio/detail/scheduler_operation.hpp:40
 -  No locals.
 -  #26 boost::asio::detail::scheduler::do_run_one (ec=..., this_thread=..., lock=..., this=0x55555587c9f0) at /usr/include/boost/asio/detail/impl/scheduler.ipp:401
 -          task_result = 0
 -          on_exit = {scheduler_ = 0x55555587c9f0, lock_ = 0x7fffffffdc00, this_thread_ = 0x7fffffffdc30}
 -          o = 0x555555b54a70
 -          more_handlers = true
 -  #27 boost::asio::detail::scheduler::run (this=0x55555587c9f0, ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:154
 -          this_thread = {<boost::asio::detail::thread_info_base> = {<boost::asio::detail::noncopyable> = {<No data fields>}, reusable_memory_ = {0x7ffeb92483e0, 0x0}}, private_op_queue = {<boost::asio::detail::noncopyable> = {<No data fields>}, front_ = 0x0, back_ = 0x0}, private_outstanding_work = 0}
 -          ctx = {<boost::asio::detail::noncopyable> = {<No data fields>}, key_ = 0x55555587c9f0, value_ = 0x7fffffffdc30, next_ = 0x0}
 -          lock = {<boost::asio::detail::noncopyable> = {<No data fields>}, mutex_ = @0x55555587ca20, locked_ = false}
 -          n = 37042768
 -  #28 0x000055555575563c in boost::asio::io_context::run (this=0x7fffffffde68) at /usr/include/boost/asio/impl/io_context.ipp:62
 -          ec = {m_val = 0, m_cat = 0x7ffff7d09070}
 -          s = <optimized out>
 -          ec = <optimized out>
 -          s = <optimized out>
 -  #29 ServiceManager::run (this=0x7fffffffde30) at server.cpp:78
 -          __PRETTY_FUNCTION__ = "void ServiceManager::run()"
 -  #30 0x00005555556f2755 in main (argc=<optimized out>, argv=<optimized out>) at otserv.cpp:299
 -          args = std::vector of length 1, capacity 1 = {"/home/server/run_tfs"}
 -          servicer = {<boost::noncopyable_::noncopyable> = {<No data fields>}, m_acceptors = std::unordered_map with 2 elements = {[7171] = {px = 0x7fffb9903770, pn = {pi_ = 0x7fffb988b670}}, [7172] = {px = 0x7fffb99043f0, pn = {pi_ = 0x7fffb9951180}}}, m_io_service = {<boost::asio::execution_context> = {<boost::asio::detail::noncopyable> = {<No data fields>}, service_registry_ = 0x55555587c9a0}, impl_ = @0x55555587c9f0}, death_timer = {<boost::asio::basic_io_object<boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >, true>> = {service_ = 0x55555587caf0, implementation_ = {<boost::asio::detail::noncopyable> = {<No data fields>}, expiry = {<boost::date_time::base_time<boost::posix_time::ptime, boost::date_time::counted_time_system<boost::date_time::counted_time_rep<boost::posix_time::millisec_posix_time_system_config> > >> = {<boost::operators_impl::less_than_comparable<boost::posix_time::ptime, boost::operators_impl::equality_comparable<boost::posix_time::ptime, boost::posix_time::ptime, boost::operators_impl::operators_detail::empty_base<boost::posix_time::ptime>, boost::operators_impl::operators_detail::false_t>, boost::operators_impl::operators_detail::empty_base<boost::posix_time::ptime>, boost::operators_impl::operators_detail::true_t>> = {<boost::operators_impl::less_than_comparable1<boost::posix_time::ptime, boost::operators_impl::equality_comparable<boost::posix_time::ptime, boost::posix_time::ptime, boost::operators_impl::operators_detail::empty_base<boost::posix_time::ptime>, boost::operators_impl::operators_detail::false_t> >> = {<boost::operators_impl::equality_comparable<boost::posix_time::ptime, boost::posix_time::ptime, boost::operators_impl::operators_detail::empty_base<boost::posix_time::ptime>, boost::operators_impl::operators_detail::false_t>> = {<boost::operators_impl::equality_comparable1<boost::posix_time::ptime, boost::operators_impl::operators_detail::empty_base<boost::posix_time::ptime> >> = {<boost::operators_impl::operators_detail::empty_base<boost::posix_time::ptime>> = {<No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, time_ = {time_count_ = {value_ = 9223372036854775806}}}, <No data fields>}, might_have_pending_waits = false, timer_data = {op_queue_ = {<boost::asio::detail::noncopyable> = {<No data fields>}, front_ = 0x0, back_ = 0x0}, heap_index_ = 18446744073709551615, next_ = 0x0, prev_ = 0x0}}}, <No data fields>}, running = true}
 -          sigh = {__sigaction_handler = {sa_handler = 0x1, sa_sigaction = 0x1}, sa_mask = {__val = {0 <repeats 16 times>}}, sa_flags = 0, sa_restorer = 0x7}
 -          outPath = <error: Cannot access memory at address 0x7>
 -          errPath = <error reading variable: Cannot create a lazy string with address 0x0, and a non-zero length.>
 
I don't know, if there is any magic-multi-thread-security added in 0.4 code, but this part of code looks like a bug:

Function ProtocolGame:parseSay is called in 'network input thread' (main ots thread) and it should not manipulate player. It should only read ID of player with player->getID() and pass handling to 'dispatcher thread'. Dispatcher thread should be only thread that sends packets to players.
There are 2 possible problems:
1. Iterating over some structure in player, which may be changed in meantime by other thread:
Code:
for (auto& client : player->clients) {

2. Sending packets from 'network input thread':
Code:
client->sendChannelMessage(name, text, TALKTYPE_CHANNEL_Y, CHANNEL_CAST);

Bugs like that made otservbr-global crash often.

--------------
About freez: Does it go 100% cpu or 0% when server freez? GDB looks like 0% cpu. It looks like it stopped on mutex, which for some reason won't get unlocked. Maybe some multithread problem.

First I would comment:
and this:

It should disable cast chat and packets that send to player from 'network input thread'.
 
@Gesior.pl thanks for such a comprehensive answer and directing me. Blocking writing helped, is there any chance for it to work on this TFS? (viewers write msg etc.)
 
Back
Top