Od jakiegoś czasu, nagle w hosteamie zaczeły się pojawiać ataki na otsa, jak wynika z logów SYN flood na port 80 lub 7171. To co widze w logach messages:
oraz w syslog
Ataki prowadzą do przeciążenia cpu procesami: ksoftirqd i kworker i co za tym idzie OTS kickuje ludzi. Po chwili wszystko wraca do normy. Niby z neta zmodyfikowałem ustawienia w pliku /etc/sysctl.conf aby troche zniwelować straty na:
Ale oczywiście kij pomaga. Serwer www to nginx a regułki firewalla to:
Dodatkowo logi z munina:
Z hostemu napisali, że atak był tak mały, że nawet lol nie wykryli. Nie wiem czy już im się nie chce czy wina leży po serwerze. Jest ktos kto się zna na tych klockach i może doradzić?
Code:
May 11 23:23:41 d5703 kernel: [139273.749126] [B][COLOR="#FF0000"]TCP: Possible SYN flooding on port 80. Sending cookies. Check SNMP counters.[/COLOR][/B]
May 11 23:26:56 d5703 kernel: [139467.807541] kjournald D ffff8802322bd690 0 251 2 0x00000000
May 11 23:26:56 d5703 kernel: [139467.807544] ffff8802322bd690 0000000000000046 ffffffff00000000 ffff880235b160c0
May 11 23:26:56 d5703 kernel: [139467.807547] 0000000000013740 ffff88023259bfd8 ffff88023259bfd8 0000000000013740
May 11 23:26:56 d5703 kernel: [139467.807549] ffff8802322bd690 ffff88023259a010 ffffffff81013a01 000000018106a5da
May 11 23:26:56 d5703 kernel: [139467.807552] Call Trace:
May 11 23:26:56 d5703 kernel: [139467.807557] [<ffffffff81013a01>] ? read_tsc+0x5/0x16
May 11 23:26:56 d5703 kernel: [139467.807561] [<ffffffff8112b709>] ? unmap_underlying_metadata+0x4b/0x4b
May 11 23:26:56 d5703 kernel: [139467.807563] [<ffffffff8136786b>] ? io_schedule+0x84/0xc3
May 11 23:26:56 d5703 kernel: [139467.807565] [<ffffffff8112b712>] ? sleep_on_buffer+0x9/0xd
May 11 23:26:56 d5703 kernel: [139467.807567] [<ffffffff81367c66>] ? __wait_on_bit+0x3e/0x6f
May 11 23:26:56 d5703 kernel: [139467.807569] [<ffffffff81367d05>] ? out_of_line_wait_on_bit+0x6e/0x77
May 11 23:26:56 d5703 kernel: [139467.807571] [<ffffffff8112b709>] ? unmap_underlying_metadata+0x4b/0x4b
May 11 23:26:56 d5703 kernel: [139467.807574] [<ffffffff81063b2b>] ? autoremove_wake_function+0x2a/0x2a
May 11 23:26:56 d5703 kernel: [139467.807576] [<ffffffff8112b6a4>] ? wait_on_buffer+0xe/0x28
May 11 23:26:56 d5703 kernel: [139467.807578] [<ffffffff8112c6eb>] ? __sync_dirty_buffer+0x58/0x81
May 11 23:26:56 d5703 kernel: [139467.807588] [<ffffffffa00fc7f7>] ? journal_commit_transaction+0xb5f/0xec8 [jbd]
May 11 23:26:56 d5703 kernel: [139467.807590] [<ffffffff813674fc>] ? __schedule+0x5a0/0x5cd
May 11 23:26:56 d5703 kernel: [139467.807593] [<ffffffffa00fff73>] ? kjournald+0xde/0x220 [jbd]
May 11 23:26:56 d5703 kernel: [139467.807595] [<ffffffff81063b01>] ? wake_up_bit+0x20/0x20
May 11 23:26:56 d5703 kernel: [139467.807598] [<ffffffffa00ffe95>] ? commit_timeout+0xb/0xb [jbd]
May 11 23:26:56 d5703 kernel: [139467.807601] [<ffffffffa00ffe95>] ? commit_timeout+0xb/0xb [jbd]
May 11 23:26:56 d5703 kernel: [139467.807602] [<ffffffff810636b5>] ? kthread+0x7a/0x82
May 11 23:26:56 d5703 kernel: [139467.807604] [<ffffffff81370134>] ? kernel_thread_helper+0x4/0x10
May 11 23:26:56 d5703 kernel: [139467.807606] [<ffffffff8106363b>] ? kthread_worker_fn+0x147/0x147
May 11 23:26:56 d5703 kernel: [139467.807608] [<ffffffff81370130>] ? gs_change+0x13/0x13
May 11 23:26:56 d5703 kernel: [139467.808858] mysqld D ffff880232679750 0 29947 1220 0x00000000
May 11 23:26:56 d5703 kernel: [139467.808860] ffff880232679750 0000000000000086 0000000000000000 ffff880235b59610
May 11 23:26:56 d5703 kernel: [139467.808862] 0000000000013740 ffff8801394f9fd8 ffff8801394f9fd8 0000000000013740
May 11 23:26:56 d5703 kernel: [139467.808864] ffff880232679750 ffff8801394f8010 ffff88023242fc98 000000018103b9a2
May 11 23:26:56 d5703 kernel: [139467.808867] Call Trace:
May 11 23:26:56 d5703 kernel: [139467.808870] [<ffffffffa00ffdd4>] ? log_wait_commit+0xc0/0x111 [jbd]
May 11 23:26:56 d5703 kernel: [139467.808872] [<ffffffff81063b01>] ? wake_up_bit+0x20/0x20
May 11 23:26:56 d5703 kernel: [139467.808875] [<ffffffffa00ffc38>] ? __log_start_commit+0x35/0x8c [jbd]
May 11 23:26:56 d5703 kernel: [139467.808879] [<ffffffffa0113b08>] ? ext3_sync_file+0x130/0x19c [ext3]
May 11 23:26:56 d5703 kernel: [139467.808881] [<ffffffff811290c5>] ? do_fsync+0x27/0x3b
May 11 23:26:56 d5703 kernel: [139467.808883] [<ffffffff811290f6>] ? sys_fsync+0xb/0xf
May 11 23:26:56 d5703 kernel: [139467.808884] [<ffffffff8136dfd2>] ? system_call_fastpath+0x16/0x1b
May 11 23:32:56 d5703 kernel: [139826.640785] kjournald D ffff8802322bd690 0 251 2 0x00000000
May 11 23:32:56 d5703 kernel: [139826.640788] ffff8802322bd690 0000000000000046 ffffffff00000000 ffff880235b59610
May 11 23:32:56 d5703 kernel: [139826.640791] 0000000000013740 ffff88023259bfd8 ffff88023259bfd8 0000000000013740
May 11 23:32:56 d5703 kernel: [139826.640793] ffff8802322bd690 ffff88023259a010 ffffffff81013a01 000000018106a5da
May 11 23:32:56 d5703 kernel: [139826.640795] Call Trace:
May 11 23:32:56 d5703 kernel: [139826.640801] [<ffffffff81013a01>] ? read_tsc+0x5/0x16
May 11 23:32:56 d5703 kernel: [139826.640804] [<ffffffff8112b709>] ? unmap_underlying_metadata+0x4b/0x4b
May 11 23:32:56 d5703 kernel: [139826.640807] [<ffffffff8136786b>] ? io_schedule+0x84/0xc3
May 11 23:32:56 d5703 kernel: [139826.640809] [<ffffffff8112b712>] ? sleep_on_buffer+0x9/0xd
May 11 23:32:56 d5703 kernel: [139826.640811] [<ffffffff81367c66>] ? __wait_on_bit+0x3e/0x6f
May 11 23:32:56 d5703 kernel: [139826.640812] [<ffffffff81367d05>] ? out_of_line_wait_on_bit+0x6e/0x77
May 11 23:32:56 d5703 kernel: [139826.640814] [<ffffffff8112b709>] ? unmap_underlying_metadata+0x4b/0x4b
May 11 23:32:56 d5703 kernel: [139826.640817] [<ffffffff81063b2b>] ? autoremove_wake_function+0x2a/0x2a
May 11 23:32:56 d5703 kernel: [139826.640819] [<ffffffff8112b6a4>] ? wait_on_buffer+0xe/0x28
May 11 23:32:56 d5703 kernel: [139826.640821] [<ffffffff8112c6eb>] ? __sync_dirty_buffer+0x58/0x81
May 11 23:32:56 d5703 kernel: [139826.640831] [<ffffffffa00fc7f7>] ? journal_commit_transaction+0xb5f/0xec8 [jbd]
May 11 23:32:56 d5703 kernel: [139826.640833] [<ffffffff813674fc>] ? __schedule+0x5a0/0x5cd
May 11 23:32:56 d5703 kernel: [139826.640836] [<ffffffffa00fff73>] ? kjournald+0xde/0x220 [jbd]
May 11 23:32:56 d5703 kernel: [139826.640838] [<ffffffff81063b01>] ? wake_up_bit+0x20/0x20
May 11 23:32:56 d5703 kernel: [139826.640841] [<ffffffffa00ffe95>] ? commit_timeout+0xb/0xb [jbd]
May 11 23:32:56 d5703 kernel: [139826.640844] [<ffffffffa00ffe95>] ? commit_timeout+0xb/0xb [jbd]
May 11 23:32:56 d5703 kernel: [139826.640845] [<ffffffff810636b5>] ? kthread+0x7a/0x82
May 11 23:32:56 d5703 kernel: [139826.640848] [<ffffffff81370134>] ? kernel_thread_helper+0x4/0x10
May 11 23:32:56 d5703 kernel: [139826.640850] [<ffffffff8106363b>] ? kthread_worker_fn+0x147/0x147
May 11 23:32:56 d5703 kernel: [139826.640851] [<ffffffff81370130>] ? gs_change+0x13/0x13
May 11 23:32:56 d5703 kernel: [139826.642186] mysqld D ffff880233666e60 0 3818 1220 0x00000000
May 11 23:32:56 d5703 kernel: [139826.642188] ffff880233666e60 0000000000000086 0000000000000000 ffff880235b160c0
May 11 23:32:56 d5703 kernel: [139826.642190] 0000000000013740 ffff88002d4a5fd8 ffff88002d4a5fd8 0000000000013740
May 11 23:32:56 d5703 kernel: [139826.642192] ffff880233666e60 ffff88002d4a4010 ffff88023242fc98 000000018103b9a2
May 11 23:32:56 d5703 kernel: [139826.642194] Call Trace:
May 11 23:32:56 d5703 kernel: [139826.642198] [<ffffffffa00ffdd4>] ? log_wait_commit+0xc0/0x111 [jbd]
May 11 23:32:56 d5703 kernel: [139826.642200] [<ffffffff81063b01>] ? wake_up_bit+0x20/0x20
May 11 23:32:56 d5703 kernel: [139826.642203] [<ffffffffa00ffc38>] ? __log_start_commit+0x35/0x8c [jbd]
May 11 23:32:56 d5703 kernel: [139826.642207] [<ffffffffa0113b08>] ? ext3_sync_file+0x130/0x19c [ext3]
May 11 23:32:56 d5703 kernel: [139826.642209] [<ffffffff811290c5>] ? do_fsync+0x27/0x3b
May 11 23:32:56 d5703 kernel: [139826.642210] [<ffffffff811290f6>] ? sys_fsync+0xb/0xf
May 11 23:32:56 d5703 kernel: [139826.642212] [<ffffffff8136dfd2>] ? system_call_fastpath+0x16/0x1b
oraz w syslog
Code:
May 11 23:32:56 d5703 kernel: [139826.639502][COLOR="#FF0000"] INFO: task kjournald:251 blocked for more than 120 seconds.[/COLOR]
May 11 23:32:56 d5703 kernel: [139826.640139] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 11 23:32:56 d5703 kernel: [139826.640785] kjournald D ffff8802322bd690 0 251 2 0x00000000
May 11 23:32:56 d5703 kernel: [139826.640788] ffff8802322bd690 0000000000000046 ffffffff00000000 ffff880235b59610
May 11 23:32:56 d5703 kernel: [139826.640791] 0000000000013740 ffff88023259bfd8 ffff88023259bfd8 0000000000013740
May 11 23:32:56 d5703 kernel: [139826.640793] ffff8802322bd690 ffff88023259a010 ffffffff81013a01 000000018106a5da
May 11 23:32:56 d5703 kernel: [139826.640795] Call Trace:
May 11 23:32:56 d5703 kernel: [139826.640801] [<ffffffff81013a01>] ? read_tsc+0x5/0x16
May 11 23:32:56 d5703 kernel: [139826.640804] [<ffffffff8112b709>] ? unmap_underlying_metadata+0x4b/0x4b
May 11 23:32:56 d5703 kernel: [139826.640807] [<ffffffff8136786b>] ? io_schedule+0x84/0xc3
May 11 23:32:56 d5703 kernel: [139826.640809] [<ffffffff8112b712>] ? sleep_on_buffer+0x9/0xd
May 11 23:32:56 d5703 kernel: [139826.640811] [<ffffffff81367c66>] ? __wait_on_bit+0x3e/0x6f
May 11 23:32:56 d5703 kernel: [139826.640812] [<ffffffff81367d05>] ? out_of_line_wait_on_bit+0x6e/0x77
May 11 23:32:56 d5703 kernel: [139826.640814] [<ffffffff8112b709>] ? unmap_underlying_metadata+0x4b/0x4b
May 11 23:32:56 d5703 kernel: [139826.640817] [<ffffffff81063b2b>] ? autoremove_wake_function+0x2a/0x2a
May 11 23:32:56 d5703 kernel: [139826.640819] [<ffffffff8112b6a4>] ? wait_on_buffer+0xe/0x28
May 11 23:32:56 d5703 kernel: [139826.640821] [<ffffffff8112c6eb>] ? __sync_dirty_buffer+0x58/0x81
May 11 23:32:56 d5703 kernel: [139826.640831] [<ffffffffa00fc7f7>] ? journal_commit_transaction+0xb5f/0xec8 [jbd]
May 11 23:32:56 d5703 kernel: [139826.640833] [<ffffffff813674fc>] ? __schedule+0x5a0/0x5cd
May 11 23:32:56 d5703 kernel: [139826.640836] [<ffffffffa00fff73>] ? kjournald+0xde/0x220 [jbd]
May 11 23:32:56 d5703 kernel: [139826.640838] [<ffffffff81063b01>] ? wake_up_bit+0x20/0x20
May 11 23:32:56 d5703 kernel: [139826.640841] [<ffffffffa00ffe95>] ? commit_timeout+0xb/0xb [jbd]
May 11 23:32:56 d5703 kernel: [139826.640844] [<ffffffffa00ffe95>] ? commit_timeout+0xb/0xb [jbd]
May 11 23:32:56 d5703 kernel: [139826.640845] [<ffffffff810636b5>] ? kthread+0x7a/0x82
May 11 23:32:56 d5703 kernel: [139826.640848] [<ffffffff81370134>] ? kernel_thread_helper+0x4/0x10
May 11 23:32:56 d5703 kernel: [139826.640850] [<ffffffff8106363b>] ? kthread_worker_fn+0x147/0x147
May 11 23:32:56 d5703 kernel: [139826.640851] [<ffffffff81370130>] ? gs_change+0x13/0x13
May 11 23:32:56 d5703 kernel: [139826.640860] [COLOR="#FF0000"]INFO: task mysqld:3818 blocked for more than 120 seconds.[/COLOR]
May 11 23:32:56 d5703 kernel: [139826.641517] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 11 23:32:56 d5703 kernel: [139826.642186] mysqld D ffff880233666e60 0 3818 1220 0x00000000
May 11 23:32:56 d5703 kernel: [139826.642188] ffff880233666e60 0000000000000086 0000000000000000 ffff880235b160c0
May 11 23:32:56 d5703 kernel: [139826.642190] 0000000000013740 ffff88002d4a5fd8 ffff88002d4a5fd8 0000000000013740
May 11 23:32:56 d5703 kernel: [139826.642192] ffff880233666e60 ffff88002d4a4010 ffff88023242fc98 000000018103b9a2
May 11 23:32:56 d5703 kernel: [139826.642194] Call Trace:
May 11 23:32:56 d5703 kernel: [139826.642198] [<ffffffffa00ffdd4>] ? log_wait_commit+0xc0/0x111 [jbd]
May 11 23:32:56 d5703 kernel: [139826.642200] [<ffffffff81063b01>] ? wake_up_bit+0x20/0x20
May 11 23:32:56 d5703 kernel: [139826.642203] [<ffffffffa00ffc38>] ? __log_start_commit+0x35/0x8c [jbd]
May 11 23:32:56 d5703 kernel: [139826.642207] [<ffffffffa0113b08>] ? ext3_sync_file+0x130/0x19c [ext3]
May 11 23:32:56 d5703 kernel: [139826.642209] [<ffffffff811290c5>] ? do_fsync+0x27/0x3b
May 11 23:32:56 d5703 kernel: [139826.642210] [<ffffffff811290f6>] ? sys_fsync+0xb/0xf
May 11 23:32:56 d5703 kernel: [139826.642212] [<ffffffff8136dfd2>] ? system_call_fastpath+0x16/0x1b
Ataki prowadzą do przeciążenia cpu procesami: ksoftirqd i kworker i co za tym idzie OTS kickuje ludzi. Po chwili wszystko wraca do normy. Niby z neta zmodyfikowałem ustawienia w pliku /etc/sysctl.conf aby troche zniwelować straty na:
Code:
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_syn_retries = 3
net.ipv4.tcp_synack_retries = 3
net.ipv4.tcp_max_syn_backlog = 65536
net.core.wmem_max = 8388608
net.core.rmem_max = 8388608
net.core.somaxconn = 512
net.core.optmem_max = 81920
net.core.somaxconn = 4096
Ale oczywiście kij pomaga. Serwer www to nginx a regułki firewalla to:
Code:
#!/bin/bash
########################################
# Firewall #
########################################
# Politica Default - DROP
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -P FORWARD ACCEPT
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
modprobe ip_conntrack_ftp
# ------------------------------------------------
# Protection against TCP syncookies
echo 1 > /proc/sys/net/ipv4/tcp_syncookies
# Ignore ICMP
echo "1" > /proc/sys/net/ipv4/icmp_echo_ignore_all
# Default rules
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
# Block NEW connection with flag other than SYN
iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP
iptables -A FORWARD -p tcp ! --syn -m state --state NEW -j DROP
# Drop fragmented packets
iptables -A INPUT -f -j DROP
iptables -A FORWARD -f -j DROP
# Drop connections in INVALID state
iptables -A INPUT -m state --state INVALID -j DROP
iptables -A FORWARD -m state --state INVALID -j DROP
iptables -N syn_flood
iptables -A INPUT -p tcp --syn -j syn_flood
iptables -A syn_flood -m limit --limit 1/s --limit-burst 3 -j RETURN
iptables -A syn_flood -j DROP
# Allow loopback interface connections
# IMPORTANT !
iptables -A INPUT -i lo -j ACCEPT
# Allow access to localhost
iptables -I INPUT -p all -s 127.0.0.1 -j ACCEPT
# drop banned clients
iptables -A INPUT -m recent --rcheck --seconds 600 --name ban --rsource -j DROP
# Allow connections from origin
iptables -I INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# ban over 24 connections
iptables -A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m connlimit --connlimit-above 24 --connlimit-mask 32 -m recent --set --name ban --rsource -j DROP
# Allow external access to ports
iptables -I INPUT -p tcp --dport 7171 -j ACCEPT # TIBIA
iptables -I INPUT -p tcp --dport 7172 -j ACCEPT # TIBIA GAME PORT
iptables -I INPUT -p tcp --dport 21 -j ACCEPT # FTP
iptables -I INPUT -p tcp --dport 80 -j ACCEPT # HTTP
iptables -I INPUT -p icmp -m limit --limit 15/s -j ACCEPT # PING
# Limit connections on ports
iptables -A INPUT -p tcp -m recent --rcheck --seconds 60 -j REJECT
iptables -I INPUT -p tcp --dport 21 -m connlimit --connlimit-above 2 -j DROP
iptables -I INPUT -p tcp --dport 80 -m connlimit --connlimit-above 20 -j DROP
iptables -I INPUT -p tcp --dport 7171 -m connlimit --connlimit-above 7 -j REJECT --reject-with tcp-reset
iptables -I INPUT -p tcp --dport 7172 -m connlimit --connlimit-above 7 -j REJECT --reject-with tcp-reset
# Allow SSH (PUTTY)
iptables -I INPUT -p tcp --dport 22 -j ACCEPT
iptables -I INPUT -p tcp --dport xxxx -j ACCEPT
echo "Block TCP-CONNECT scan attempts (SYN bit packets)"
iptables -A INPUT -p tcp --syn -j DROP
echo "Block TCP-SYN scan attempts (only SYN bit packets)"
iptables -A INPUT -m conntrack --ctstate NEW -p tcp --tcp-flags SYN,RST,ACK,FIN,URG,PSH SYN -j DROP
echo "Block TCP-FIN scan attempts (only FIN bit packets)"
iptables -A INPUT -m conntrack --ctstate NEW -p tcp --tcp-flags SYN,RST,ACK,FIN,URG,PSH FIN -j DROP
echo "Block TCP-ACK scan attempts (only ACK bit packets)"
iptables -A INPUT -m conntrack --ctstate NEW -p tcp --tcp-flags SYN,RST,ACK,FIN,URG,PSH ACK -j DROP
echo "Block TCP-NULL scan attempts (packets without flag)"
iptables -A INPUT -m conntrack --ctstate INVALID -p tcp --tcp-flags ! SYN,RST,ACK,FIN,URG,PSH -j DROP
echo "Block "Christmas Tree" TCP-XMAS scan attempts (packets with FIN, URG, PSH bits)"
iptables -A INPUT -m conntrack --ctstate NEW -p tcp --tcp-flags SYN,RST,ACK,FIN,URG,PSH FIN,URG,PSH -j DROP
echo "Block DOS - Ping of Death"
iptables -A INPUT -p ICMP --icmp-type echo-request -m length --length 60:65535 -j ACCEPT
echo "Block DOS - Teardrop"
iptables -A INPUT -p UDP -f -j DROP
echo "Block DDOS - SYN-flood"
iptables -A INPUT -p TCP --syn -m iplimit --iplimit-above 9 -j DROP
echo "Block DDOS - Smurf"
iptables -A INPUT -m pkttype --pkt-type broadcast -j DROP
iptables -A INPUT -p ICMP --icmp-type echo-request -m pkttype --pkttype broadcast -j DROP
iptables -A INPUT -p ICMP --icmp-type echo-request -m limit --limit 3/s -j ACCEPT
echo "Block DDOS - UDP-flood (Pepsi)"
iptables -A INPUT -p UDP --dport 7 -j DROP
iptables -A INPUT -p UDP --dport 19 -j DROP
echo "Block DDOS - SMBnuke"
iptables -A INPUT -p UDP --dport 135:139 -j DROP
iptables -A INPUT -p TCP --dport 135:139 -j DROP
echo "Block DDOS - Connection-flood"
iptables -A INPUT -p TCP --syn -m iplimit --iplimit-above 3 -j DROP
echo "Block DDOS - Fraggle"
iptables -A INPUT -p UDP -m pkttype --pkt-type broadcast -j DROP
iptables -A INPUT -p UDP -m limit --limit 3/s -j ACCEPT
echo "Block DDOS - Jolt"
iptables -A INPUT -p ICMP -f -j DROP
echo "Block UDP"
iptables -A INPUT -p UDP -j DROP
Dodatkowo logi z munina:
Z hostemu napisali, że atak był tak mały, że nawet lol nie wykryli. Nie wiem czy już im się nie chce czy wina leży po serwerze. Jest ktos kto się zna na tych klockach i może doradzić?