华为9306交换机ICMP包攻击导致直连丢包但业务不受影响故障处理故报告故障现象描述与说明故障现象：Ping华为9306交换机的任何直连地址会丢包，经过交换机的业务数据不受影响。现状、拓扑与配置网络情况：华为9303交换机华为9306交换机故障现象及处理步骤一：1、在交换机9306-B上通过命令displaylogbuffer查看Apr23201214:25:16JM-SN5L-DCN-9306-2%%01QOSE/4/CPCAR_DROP_LPU(l):SomepacketsaredroppedbycpcarontheLPUinslot1.(Protocol=icmp,Drop-Count=0529546)Apr23201214:25:16JM-SN5L-DCN-9306-2%%01QOSE/4/CPCAR_DROP_MPU(l):SomepacketsaredroppedbycpcarontheMPU.(Protocol=icmp,Drop-Count=049663)Apr23201214:15:16JM-SN5L-DCN-9306-2%%01QOSE/4/CPCAR_DROP_LPU(l):SomepacketsaredroppedbycpcarontheLPUinslot1.(Protocol=icmp,Drop-Count=0489843)Apr23201214:15:16JM-SN5L-DCN-9306-2%%01QOSE/4/CPCAR_DROP_MPU(l):SomepacketsaredroppedbycpcarontheMPU.(Protocol=icmp,Drop-Count=049826)Apr23201214:09:39JM-SN5L-DCN-9306-2%%01HWCM/4/EXIT(l):Exitfromconfiguremode.Apr23201214:05:16JM-SN5L-DCN-9306-2%%01QOSE/4/CPCAR_DROP_LPU(l):SomepacketsaredroppedbycpcarontheLPUinslot1.(Protocol=icmp,Drop-Count=0483657)大量的icmp包到达设备后由主引擎和slot1的CPCAR进行丢弃。2、在交换机9306-B上通过命令displaycpu-defendstatisticsall查看CPCARonmainboard-------------------------------------------------------------------------------PacketTypePass(Bytes)Drop(Bytes)Pass(Packets)Drop(Packets)stp0000smart-link0000ldt0000lacp0000lldp0000dldp0000vrrp0000isis0000igmp0000pim0000rip0000ospf1406001320bgp88257011750mpls-rsvp0000mpls-ldp0000ttl-expired0000icmp118941948524911341828397807336eoam-3ah0000mpls-ping0000mpls-ttl-expired0000ntp0000ripng0000ospfv30000bgp4plus0000pimv60000hotlimit0000vrrp60000mld1313001350icmpv60000telnet8007350125060ssh0000ftp0000snmp0000radius0000hw-tacacs0000tcp1405201980mpls-fib-hit0000fib-hit0000arp-miss1630202070unknown-packet0000hopbyhop0000pppoe0000bpdu-tunnel0000rrpp0000udp-helper0000-------------------------------------------------------------------------------CPCARonslot1-------------------------------------------------------------------------------PacketTypePass(Bytes)Drop(Bytes)Pass(Packets)Drop(Packets)arp-request1196801760arp-reply44200660stp0000smart-link0000ldt0000lacp0000lldp0000dldp0000vrrp0000mpls-oam0000isis0000dhcp-client0000dhcp-server0000igmp0000pim0000rip0000ospf21229620229630bgp91561011750bfd0000mpls-rsvp0000mpls-ldp0000ttl-expired143622014530icmp18078814662206819626265699079349eoam-3ah0000eoam-1ag0000mpls-ping0000mpls-ttl-expired0000ntp00008021x00000000ripng0000ospfv30000bgp4plus0000pimv60000hotlimit0000vrrp60000dhcpv6-request0000dhcpv6-reply0000mld1367001350icmpv60000hvrp0000telnet853955136125532ssh0000ftp0000snmp0000radius0000hw-tacacs0000tcp1339201800mpls-fib-hit0000fib-hit90000900arp-miss1703402070unknown-packet0000unknown-multicast1371734801666540hopbyhop0000pppoe0000bpdu-tunnel0000从上述很容易看出：大量的ICMP经交换机处理不过来从而丢弃。步骤二：在交换机9306上开启ICMP的debug信息找出具体的攻击源。通过在交换机上执行debuggingipicmp，发现从鹤山上来的主要有132.103.145.0/24、132.103.146.0/24、132.103.147.0/24三个网段的源进行大量的icmp包。于是建议客户要求鹤山本地关注这些网段的终端进行病毒扫描处理。步骤三：业务恢复（在交换机上针对上述的三个网段的ICMP包进行黑名单处理）aclnumber3100rule5permiticmpsource132.103.145.00.0.0.255rule10permiticmpsource132.103.147.00.0.0.255rule15permiticmpsource132.103.146.00.0.0.255#cpu-defendpolicy1blacklist1acl3100#slot1cpu-defend-policy1处理后，icmp处理恢复正常，直连ping也不再丢包。并且观察了一天后，也正常。因此，攻击源在鹤山本地。处理过程信息LOG编号文件名说明1LOG文件应包含设备的软件版本信息、硬件配置信息、处理过程日志等内容。根本原因分析1、华为9300系列交换默认隐藏模式下有针对各种报文的QOS限速机制，当对应的报文超出设定的速率值时，由CPU-DEFENSE将后续的包进行丢弃，如ICMP，后续的包就表现为丢包现象。开启debug：debuggingarpterminalmonitorterminaldebugging关闭debug：undoterminaldebuggingundoterminalmonitorundodebuggingarp