Gsp
RP
- Joined
- Jan 3, 2008
- Messages
- 250
- Solutions
- 1
- Reaction score
- 4
Hi guys, I'm hosting a server which runs on a fast dedicated server. TFS is the only major application that is running on the server. It uses %20 of the memory, around %5 of the cpu. It is not an active server but players log in from time to time. Yesterday the game server randomly crashed for the second time. When I checked the SSH command line it just said "Killed". A player was online during the crash so it made me wonder if it was done deliberately.
This is how it crashed from the syslog:
I'll be very happy if someone can help me read this log and find out what might be the problem.
This is how it crashed from the syslog:
Code:
Sep 26 01:08:39 ns3067 systemd[1]: exim4-base.service: Succeeded.
Sep 26 01:08:39 ns3067 systemd[1]: Finished exim4-base housekeeping.
Sep 26 01:08:39 ns3067 systemd[1]: exim4-base.service: Consumed 9.556s CPU time.
Sep 26 01:08:39 ns3067 systemd[1]: Starting Rotate log files...
Sep 26 01:08:53 ns3067 kernel: [1459646.523577] f2b/f.sshd invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
Sep 26 01:08:53 ns3067 kernel: [1459646.523579] f2b/f.sshd cpuset=/ mems_allowed=0
Sep 26 01:08:53 ns3067 kernel: [1459646.523583] CPU: 4 PID: 720 Comm: f2b/f.sshd Tainted: G I 4.19.0-17-amd64 #1 Debian 4.19.194-3
Sep 26 01:08:53 ns3067 kernel: [1459646.523584] Hardware name: Supermicro X8STi/X8STi, BIOS 2.0 09/17/10
Sep 26 01:08:53 ns3067 kernel: [1459646.523584] Call Trace:
Sep 26 01:08:53 ns3067 kernel: [1459646.523592] dump_stack+0x66/0x81
Sep 26 01:08:53 ns3067 kernel: [1459646.523595] dump_header+0x6b/0x283
Sep 26 01:08:53 ns3067 kernel: [1459646.523598] ? do_try_to_free_pages+0x2ec/0x370
Sep 26 01:08:53 ns3067 kernel: [1459646.523599] oom_kill_process.cold.30+0xb/0x1cf
Sep 26 01:08:53 ns3067 kernel: [1459646.523602] ? oom_badness+0x23/0x140
Sep 26 01:08:53 ns3067 kernel: [1459646.523603] out_of_memory+0x1a5/0x450
Sep 26 01:08:53 ns3067 kernel: [1459646.523605] __alloc_pages_slowpath+0xbd8/0xcb0
Sep 26 01:08:53 ns3067 kernel: [1459646.523607] __alloc_pages_nodemask+0x28b/0x2b0
Sep 26 01:08:53 ns3067 kernel: [1459646.523609] filemap_fault+0x333/0x780
Sep 26 01:08:53 ns3067 kernel: [1459646.523611] ? alloc_set_pte+0xf2/0x560
Sep 26 01:08:53 ns3067 kernel: [1459646.523612] ? filemap_map_pages+0x360/0x3a0
Sep 26 01:08:53 ns3067 kernel: [1459646.523636] ext4_filemap_fault+0x2c/0x40 [ext4]
Sep 26 01:08:53 ns3067 kernel: [1459646.523638] __do_fault+0x36/0x130
Sep 26 01:08:53 ns3067 kernel: [1459646.523639] __handle_mm_fault+0xdf9/0x11f0
Sep 26 01:08:53 ns3067 kernel: [1459646.523641] handle_mm_fault+0xd6/0x200
Sep 26 01:08:53 ns3067 kernel: [1459646.523643] __do_page_fault+0x249/0x4f0
Sep 26 01:08:53 ns3067 kernel: [1459646.523646] ? page_fault+0x8/0x30
Sep 26 01:08:53 ns3067 kernel: [1459646.523647] page_fault+0x1e/0x30
Sep 26 01:08:53 ns3067 kernel: [1459646.523649] RIP: 0033:0x5d0949
Sep 26 01:08:53 ns3067 kernel: [1459646.523652] Code: Bad RIP value.
Sep 26 01:08:53 ns3067 kernel: [1459646.523653] RSP: 002b:00007f750affbfb0 EFLAGS: 00010202
Sep 26 01:08:53 ns3067 kernel: [1459646.523654] RAX: 00007f751057ff60 RBX: 0000000000000000 RCX: 0000000000ee1ea8
Sep 26 01:08:53 ns3067 kernel: [1459646.523655] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000fc0390
Sep 26 01:08:53 ns3067 kernel: [1459646.523655] RBP: 000000000073ef00 R08: 0000000000d8bbc0 R09: 0000000000905bc0
Sep 26 01:08:53 ns3067 kernel: [1459646.523656] R10: 0000000000d8be40 R11: 0000000000000008 R12: 00007f75103bae00
Sep 26 01:08:53 ns3067 kernel: [1459646.523657] R13: 00007f75103bb710 R14: 00007f75114b70b0 R15: 00007f7510b7a990
Sep 26 01:08:53 ns3067 kernel: [1459646.523658] Mem-Info:
Sep 26 01:08:53 ns3067 kernel: [1459646.523661] active_anon:3626675 inactive_anon:384094 isolated_anon:0
Sep 26 01:08:53 ns3067 kernel: [1459646.523661] active_file:103 inactive_file:38 isolated_file:0
Sep 26 01:08:53 ns3067 kernel: [1459646.523661] unevictable:0 dirty:54 writeback:0 unstable:0
Sep 26 01:08:53 ns3067 kernel: [1459646.523661] slab_reclaimable:11496 slab_unreclaimable:24062
Sep 26 01:08:53 ns3067 kernel: [1459646.523661] mapped:1186 shmem:2097 pagetables:8958 bounce:0
Sep 26 01:08:53 ns3067 kernel: [1459646.523661] free:33200 free_pcp:1416 free_cma:0
Sep 26 01:08:53 ns3067 kernel: [1459646.523664] Node 0 active_anon:14506700kB inactive_anon:1536376kB active_file:412kB inactive_file:152kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:4744kB dirty:216kB writeback:0kB shmem:8388kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 1953792kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Sep 26 01:08:53 ns3067 kernel: [1459646.523665] Node 0 DMA free:15884kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15968kB managed:15884kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 26 01:08:53 ns3067 kernel: [1459646.523667] lowmem_reserve[]: 0 3466 15982 15982 15982
Sep 26 01:08:53 ns3067 kernel: [1459646.523669] Node 0 DMA32 free:64356kB min:14640kB low:18300kB high:21960kB active_anon:3506928kB inactive_anon:52kB active_file:56kB inactive_file:52kB unevictable:0kB writepending:4kB present:3644928kB managed:3579360kB mlocked:0kB kernel_stack:16kB pagetables:6836kB bounce:0kB free_pcp:24kB local_pcp:0kB free_cma:0kB
Sep 26 01:08:53 ns3067 kernel: [1459646.523671] lowmem_reserve[]: 0 0 12516 12516 12516
Sep 26 01:08:53 ns3067 kernel: [1459646.523673] Node 0 Normal free:52560kB min:52872kB low:66088kB high:79304kB active_anon:11000032kB inactive_anon:1536324kB active_file:408kB inactive_file:352kB unevictable:0kB writepending:92kB present:13107200kB managed:12821332kB mlocked:0kB kernel_stack:2928kB pagetables:28996kB bounce:0kB free_pcp:5640kB local_pcp:648kB free_cma:0kB
Sep 26 01:08:53 ns3067 kernel: [1459646.523676] lowmem_reserve[]: 0 0 0 0 0
Sep 26 01:08:53 ns3067 kernel: [1459646.523677] Node 0 DMA: 1*4kB (U) 1*8kB (U) 0*16kB 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15884kB
Sep 26 01:08:53 ns3067 kernel: [1459646.523682] Node 0 DMA32: 753*4kB (UME) 283*8kB (UM) 103*16kB (UM) 51*32kB (UME) 18*64kB (UME) 9*128kB (UM) 3*256kB (UME) 2*512kB (UE) 3*1024kB (UME) 2*2048kB (ME) 11*4096kB (M) = 64876kB
Sep 26 01:08:53 ns3067 kernel: [1459646.523688] Node 0 Normal: 549*4kB (UME) 502*8kB (UME) 795*16kB (UME) 328*32kB (UME) 147*64kB (UME) 45*128kB (UME) 12*256kB (UME) 7*512kB (M) 2*1024kB (M) 0*2048kB 0*4096kB = 53300kB
Sep 26 01:08:53 ns3067 kernel: [1459646.523695] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 26 01:08:53 ns3067 kernel: [1459646.523695] 2729 total pagecache pages
Sep 26 01:08:53 ns3067 kernel: [1459646.523696] 202 pages in swap cache
Sep 26 01:08:53 ns3067 kernel: [1459646.523697] Swap cache stats: add 262629, delete 262427, find 9815/10060
Sep 26 01:08:53 ns3067 kernel: [1459646.523697] Free swap = 0kB
Sep 26 01:08:53 ns3067 kernel: [1459646.523698] Total swap = 1048568kB
Sep 26 01:08:53 ns3067 kernel: [1459646.523698] 4192024 pages RAM
Sep 26 01:08:53 ns3067 kernel: [1459646.523699] 0 pages HighMem/MovableOnly
Sep 26 01:08:53 ns3067 kernel: [1459646.523699] 87880 pages reserved
Sep 26 01:08:53 ns3067 kernel: [1459646.523699] 0 pages hwpoisoned
Sep 26 01:08:53 ns3067 kernel: [1459646.523700] Tasks state (memory values in pages):
Sep 26 01:08:53 ns3067 kernel: [1459646.523700] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Sep 26 01:08:53 ns3067 kernel: [1459646.523704] [ 310] 0 310 51085 237 421888 3 -250 systemd-journal
Sep 26 01:08:53 ns3067 kernel: [1459646.523705] [ 324] 0 324 5323 276 73728 0 -1000 systemd-udevd
Sep 26 01:08:53 ns3067 kernel: [1459646.523707] [ 389] 0 389 777 45 45056 11 0 mdadm
Sep 26 01:08:53 ns3067 kernel: [1459646.523708] [ 479] 0 479 24955 305 69632 38 0 dhclient
Sep 26 01:08:53 ns3067 kernel: [1459646.523709] [ 540] 0 540 1654 52 53248 17 0 cron
Sep 26 01:08:53 ns3067 kernel: [1459646.523710] [ 541] 104 541 1983 118 53248 53 -900 dbus-daemon
Sep 26 01:08:53 ns3067 kernel: [1459646.523712] [ 547] 0 547 63408 2471 180224 2 0 php-fpm8.0
Sep 26 01:08:53 ns3067 kernel: [1459646.523713] [ 548] 0 548 55184 428 73728 13 0 rsyslogd
Sep 26 01:08:53 ns3067 kernel: [1459646.523714] [ 553] 0 553 3395 187 65536 44 0 systemd-logind
Sep 26 01:08:53 ns3067 kernel: [1459646.523715] [ 555] 105 555 1485 50 53248 26 0 nscd
Sep 26 01:08:53 ns3067 kernel: [1459646.523717] [ 557] 0 557 99232 4146 143360 316 0 fail2ban-server
Sep 26 01:08:53 ns3067 kernel: [1459646.523718] [ 566] 0 566 682 13 45056 5 0 agetty
Sep 26 01:08:53 ns3067 kernel: [1459646.523719] [ 576] 106 576 18608 147 61440 29 0 ntpd
Sep 26 01:08:53 ns3067 kernel: [1459646.523720] [ 598] 0 598 4668 194 57344 31 0 nginx
Sep 26 01:08:53 ns3067 kernel: [1459646.523722] [ 599] 33 599 4795 538 65536 19 0 nginx
Sep 26 01:08:53 ns3067 kernel: [1459646.523723] [ 600] 33 600 4936 697 65536 26 0 nginx
Sep 26 01:08:53 ns3067 kernel: [1459646.523724] [ 602] 33 602 4849 598 65536 26 0 nginx
Sep 26 01:08:53 ns3067 kernel: [1459646.523725] [ 603] 33 603 4787 531 65536 27 0 nginx
Sep 26 01:08:53 ns3067 kernel: [1459646.523727] [ 604] 0 604 3302 202 61440 32 -1000 sshd
Sep 26 01:08:53 ns3067 kernel: [1459646.523728] [ 666] 109 666 599310 97751 991232 272 0 mariadbd
Sep 26 01:08:53 ns3067 kernel: [1459646.523729] [ 1070] 108 1070 4572 157 77824 86 0 exim4
Sep 26 01:08:53 ns3067 kernel: [1459646.523731] [ 1084] 0 1084 3842 310 73728 51 0 systemd
Sep 26 01:08:53 ns3067 kernel: [1459646.523732] [ 1085] 0 1085 41688 496 94208 214 0 (sd-pam)
Sep 26 01:08:53 ns3067 kernel: [1459646.523733] [ 1121] 1001 1121 1733 2 45056 146 0 screen
Sep 26 01:08:53 ns3067 kernel: [1459646.523735] [ 1122] 1001 1122 2000 2 53248 384 0 bash
Sep 26 01:08:53 ns3067 kernel: [1459646.523736] [ 1127] 1001 1127 4197431 3894541 33419264 260307 0 theforgottenser
Sep 26 01:08:53 ns3067 kernel: [1459646.523738] [ 368445] 33 368445 63477 2599 188416 1 0 php-fpm8.0
Sep 26 01:08:53 ns3067 kernel: [1459646.523739] [ 368494] 33 368494 63464 2595 188416 1 0 php-fpm8.0
Sep 26 01:08:53 ns3067 kernel: [1459646.523740] [ 368495] 33 368495 63464 2593 188416 1 0 php-fpm8.0
Sep 26 01:08:53 ns3067 kernel: [1459646.523741] [ 369079] 0 369079 5721 2543 86016 0 0 certbot
Sep 26 01:08:53 ns3067 kernel: [1459646.523743] [ 369080] 0 369080 604 17 40960 0 0 sessionclean
Sep 26 01:08:53 ns3067 kernel: [1459646.523744] [ 369087] 0 369087 604 24 40960 0 0 sessionclean
Sep 26 01:08:53 ns3067 kernel: [1459646.523745] [ 369090] 0 369090 5500 30 53248 0 0 sort
Sep 26 01:08:53 ns3067 kernel: [1459646.523746] [ 369092] 0 369092 5500 30 53248 0 0 sort
Sep 26 01:08:53 ns3067 kernel: [1459646.523747] [ 369093] 0 369093 604 24 40960 0 0 sessionclean
Sep 26 01:08:53 ns3067 kernel: [1459646.523749] [ 369101] 0 369101 20724 796 143360 0 0 php8.0
Sep 26 01:08:53 ns3067 kernel: [1459646.523750] [ 369116] 0 369116 1617 62 53248 0 0 logrotate
Sep 26 01:08:53 ns3067 kernel: [1459646.523751] [ 369126] 0 369126 1617 62 53248 0 0 logrotate
Sep 26 01:08:53 ns3067 kernel: [1459646.523752] Out of memory: Kill process 1127 (theforgottenser) score 953 or sacrifice child
Sep 26 01:08:53 ns3067 kernel: [1459646.523820] Killed process 1127 (theforgottenser) total-vm:16789724kB, anon-rss:15578164kB, file-rss:0kB, shmem-rss:0kB
Sep 26 01:08:53 ns3067 systemd[1]: session-1.scope: A process of this unit has been killed by the OOM killer.
Sep 26 01:08:53 ns3067 systemd[1]: phpsessionclean.service: Succeeded.
Sep 26 01:08:53 ns3067 systemd[1]: Finished Clean php session files.
Sep 26 01:08:53 ns3067 systemd[1]: phpsessionclean.service: Consumed 11.894s CPU time.
Sep 26 01:08:53 ns3067 kernel: [1459647.187742] oom_reaper: reaped process 1127 (theforgottenser), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sep 26 01:08:53 ns3067 mariadbd[666]: 2021-09-26 1:08:53 6 [Warning] Aborted connection 6 to db: 'dbaseuser' user: 'root' host: 'localhost' (Got an error reading communication packets)
Sep 26 01:08:53 ns3067 systemd[1]: certbot.service: Succeeded.
Sep 26 01:08:53 ns3067 systemd[1]: Finished Certbot.
Sep 26 01:08:53 ns3067 systemd[1]: certbot.service: Consumed 12.674s CPU time.
Sep 26 01:08:54 ns3067 systemd[1]: rsyslog.service: Sent signal SIGHUP to main process 548 (rsyslogd) on client request.
I'll be very happy if someone can help me read this log and find out what might be the problem.