为了解决LVSksoftirqdCPU使用率100%导致网卡软中断丢包，我和同事们一起搜索了大量的资料去分析问题，特别是感谢美团技术团队的分享帮助我们快速梳理优化思路，最后明确了如何重构RPS和RFS网卡多队列的优化脚本。个人认为这是一个大家可能普遍会遇到的问题，文章内的分析思路和解决方案未必是最优解，也欢迎各位分享自己的解决方法。
事故复盘
我们遇到的问题属于计划外的incident，现象是某产品用户在线率突然降低，LVSMaster同时收到CPUHighLoad告警，检查发现该节点出现网卡大量断开重连和丢包情况，应急切换到LVSSlave也出现上述问题，在排除掉流量异常和外部攻击后选择切换DNS到背后的NginxRealServers后服务逐步恢复。
复盘核心原因在于系统初始化时rps优化脚本没有成功执行，这个脚本起初是因为早期DBA团队遇到过CPU负载较高导致网卡异常，这个优化脚本也一直传承至今，却已经没有人知道为什么添加。现在大多数服务器没有执行成功而被大家一直所忽视显然也是postcheck没有做到位。在早期大家都停留在BashShell运维的阶段，没有专职的团队来管理确实容易失控，好在现在可以基于Ansible来做初始化和检查，运维的压力也减轻了一部分。
通过Google搜索相关知识的过程中，我们也发现在不少人都会遇到这样类似的问题。比如这篇文章提到lvs/irq
lvs的性能问题，软中断耗尽CPU单核后到达处理极限
瓶颈现象：当压力较大时，Lvs服务器CPU的其中一个核使用率达到100%（处理软中断）。当Lvs服务器处理软中断的那个核使用率达到100%，就到达系统处理上限。占用CPU的是进程“ksoftirqd”，它未能使用到多核。
做双网卡绑定，然后调试内核SMP，中断主要是来自网卡的，不是LVS本身。需要把2个网卡来的IRQ均衡在双核CPU上面。
和华为的工程师们在交换经验的时候对方分享了一个关于RSS和RPS关系图，之后的内容还会引用美团技术团队的分析
image
我们遇到的情况是缺少可用服务器资源选择把用户外部请求流量和CodisCacheCluster内部流量临时混在了同一个LVS上，虽然看上去CPU和traffic的整体压力都不算高，但是CPU的处理压力可能恰好集中在了和外网Bond1网卡相同的Core上最后引起了ksoftirqd软中断，而内网Bond0网卡就没有监控到任何丢包。虽然我们也有正常开启irqbalance，但不清楚是不是因为受到cpupowerperformance和NUMA的影响最后也没能阻止事故的发生，最终的优化方案主要是手动开启RPS和RFS，大致步骤如下：
setcpupowercpupowerfrequency-set-gperformance，CPU优化建议使用cpupower设置CPUPerformance模式
activaterps/rfsbyscript:
doubleringbuffersize:ethtool-Gp1p1[rx|tx]4096,checkethtool-gp1p1
doubleNAPIpollbudget:_budget=600
[eth0,errors,dropped,overruns]
chkconfig:23459060Provides:rpsRequired-Stop:$local_fs$remote_fs$network$syslogDefault-Stop:016Description:enabelerpswhichisakerneltweakfornetworkperformancecat/sys/devices/system/cpu/cpu0/cpufreq/scaling_governoractivaterps/rfsbyscript:[rx|tx]4096,ethtool-gp1p1toptop-19:17:57up7days,7:11,1user,loadaverage:0.00,0.01,0.05Tasks:338total,1running,337sleeping,0stopped,0zombie%Cpu0:0.0us,0.0sy,0.0ni,98.0id,0.0wa,0.0hi,2.0si,0.0st%Cpu1:0.0us,0.0sy,0.0ni,96.7id,0.0wa,0.0hi,3.3si,0.0st%Cpu2:0.0us,0.0sy,0.0ni,97.7id,0.0wa,0.0hi,2.3si,0.0st%Cpu3:0.0us,0.0sy,0.0ni,97.4id,0.0wa,0.0hi,2.6si,0.0st%Cpu4:0.0us,0.0sy,0.0ni,99.0id,0.0wa,0.0hi,1.0si,0.0st%Cpu5:0.0us,0.0sy,0.0ni,96.7id,0.0wa,0.0hi,3.3si,0.0st%Cpu6:0.0us,0.0sy,0.0ni,97.4id,0.0wa,0.0hi,2.6si,0.0st%Cpu7:0.0us,0.0sy,0.0ni,97.7id,0.0wa,0.0hi,2.3si,0.0st%Cpu8:0.0us,0.0sy,0.0ni,98.3id,0.0wa,0.0hi,1.7si,0.0st%Cpu9:0.0us,0.0sy,0.0ni,97.0id,0.0wa,0.0hi,3.0si,0.0st%Cpu10:0.0us,0.0sy,0.0ni,98.7id,0.0wa,0.0hi,1.3si,0.0st%Cpu11:0.0us,0.0sy,0.0ni,97.4id,0.0wa,0.0hi,2.6si,0.0st%Cpu12:0.0us,0.0sy,0.0ni,96.7id,0.0wa,0.0hi,3.3si,0.0st%Cpu13:0.0us,0.0sy,0.0ni,98.3id,0.0wa,0.0hi,1.7si,0.0st%Cpu14:0.0us,0.0sy,0.0ni,98.3id,0.0wa,0.0hi,1.7si,0.0st%Cpu15:0.0us,0.0sy,0.0ni,96.7id,0.0wa,0.0hi,3.3si,0.0st%Cpu16:0.0us,0.0sy,0.0ni,97.3id,0.0wa,0.0hi,2.7si,0.0st%Cpu17:0.0us,0.0sy,0.0ni,97.7id,0.0wa,0.0hi,2.3si,0.0st%Cpu18:0.3us,0.7sy,0.0ni,96.7id,0.0wa,0.0hi,2.3si,0.0st%Cpu19:0.0us,0.0sy,0.0ni,98.3id,0.0wa,0.0hi,1.7si,0.0st%Cpu20:0.0us,0.0sy,0.0ni,99.0id,0.0wa,0.0hi,1.0si,0.0st%Cpu21:0.0us,0.0sy,0.0ni,97.0id,0.0wa,0.0hi,3.0si,0.0st%Cpu22:0.0us,0.0sy,0.0ni,97.7id,0.0wa,0.0hi,2.3si,0.0st%Cpu23:0.0us,0.0sy,0.0ni,97.4id,0.0wa,0.0hi,2.6si,0.0st%Cpu24:0.0us,0.0sy,0.0ni,97.7id,0.0wa,0.0hi,2.3si,0.0st%Cpu25:0.0us,0.0sy,0.0ni,97.0id,0.0wa,0.0hi,3.0si,0.0st%Cpu26:0.0us,0.0sy,0.0ni,98.3id,0.0wa,0.0hi,1.7si,0.0st%Cpu27:0.0us,0.0sy,0.0ni,98.3id,0.0wa,0.0hi,1.7si,0.0st%Cpu28:0.0us,0.0sy,0.0ni,96.7id,0.0wa,0.0hi,3.3si,0.0st%Cpu29:0.0us,0.0sy,0.0ni,97.4id,0.0wa,0.0hi,2.6si,0.0st%Cpu30:0.0us,0.0sy,0.0ni,98.3id,0.0wa,0.0hi,1.7si,0.0st%Cpu31:0.0us,0.0sy,0.0ni,97.4id,0.0wa,0.0hi,2.6si,0.0stKiBMem:65688788total,62127224free,1742128used,1819436buff/cacheKiBSwap:66060284total,66060284free,0[root@localhostwangao]优化前[root@localhostwangao]优化后[root@localhostwangao]优化前[root@localhostwangao]优化后[root@localhostwangao]#egrep'cpu|enp175s0f0'/proc/interrupts545:0000000078728441661324182555803106208885512371620000000000412969401965430742367407680357953197910936066160151057IR-PCI-MSI-edgeenp175s0f0-TxRx-0546:0000000029333470947637279907263830033105932899494077000000003384434653425312104463040125201167IR-PCI-MSI-edgeenp175s0f0-TxRx-1547:0000000065087834747555072953222216293203384360028600000000787900982166476853638228219169981098981062757806075631IR-PCI-MSI-edgeenp175s0f0-TxRx-2548:00000000108290552624840735547355772635627950399635221069696712196600210729553139534IR-PCI-MSI-edgeenp175s0f0-TxRx-3549:000000004359768178436998498321124972000674545682667710379893406854700838IR-PCI-MSI-edgeenp175s0f0-TxRx-4550:000000001612980507053904226342262029525573500000000936684322922608690455619804612983710039IR-PCI-MSI-edgeenp175s0f0-TxRx-5551:00000000598104508566865074748733669565105350997312571270004299830232167230434654911897912IR-PCI-MSI-edgeenp175s0f0-TxRx-6552:00000000370682536379700577266310000000016847896099393002509440913049049IR-PCI-MSI-edgeenp175s0f0-TxRx-7553:00000000925533762883628379485299505383430000045773035689849748656991659950071561102436IR-PCI-MSI-edgeenp175s0f0-TxRx-8554:0000000045919755648235162908620758251119612311055230000000039979342504581901382IR-PCI-MSI-edgeenp175s0f0-TxRx-9555:000000000472620296056522695086246592405689378012774288198336045265987940901979567420IR-PCI-MSI-edgeenp175s0f0-TxRx-10556:00000000254320784536553763924325701027849237093553025572084807285335244716333437044680IR-PCI-MSI-edgeenp175s0f0-TxRx-11557:00000000355782762746096992530381653759298827296520000000012768740783470696522229642462398537950353779577448437326368IR-PCI-MSI-edgeenp175s0f0-TxRx-12558:00000000328295339700025310000000002245766616961005963404355141769770IR-PCI-MSI-edgeenp175s0f0-TxRx-13559:0000000088746890266536487768783650446000000002540382403859071974609259354802000IR-PCI-MSI-edgeenp175s0f0-TxRx-14560:0000000066464875048477443003884977961112602312668540000000687572762894922916070370247928557972200033280IR-PCI-MSI-edgeenp175s0f0-TxRx-15561:000000006086779129725205656372483046844166223009404894075510139346IR-PCI-MSI-edgeenp175s0f0-TxRx-16562:0000000070428024854751162836868107468406557855843557068573929828719983IR-PCI-MSI-edgeenp175s0f0-TxRx-17563:000000006297222106933750857663563356465863434128820806525611IR-PCI-MSI-edgeenp175s0f0-TxRx-18564:000000008238898354207547906064969848870343031988593685373679292335527IR-PCI-MSI-edgeenp175s0f0-TxRx-19565:000000006767326287046026446894232644043063655960000000030385263963275191222248226IR-PCI-MSI-edgeenp175s0f0-TxRx-20566:000000005665951419583197349309300295868322192241IR-PCI-MSI-edgeenp175s0f0-TxRx-21567:000000008326741997893259748998577498610570634823061233604620069623342IR-PCI-MSI-edgeenp175s0f0-TxRx-22568:00000000486851946241625829956099456965074404460000000029866873908525283308021907IR-PCI-MSI-edgeenp175s0f0-TxRx-23569:00000000512168295620446567489422448386252341634798200000008923355234748560797764462687411246655612746857726730942IR-PCI-MSI-edgeenp175s0f0-TxRx-24570:000000008455480562689462108440926294285438609008424525127548979250908113IR-PCI-MSI-edgeenp175s0f0-TxRx-25571:000000006990994776423234552898250604702000000005663858332108693577372016824758852410088790IR-PCI-MSI-edgeenp175s0f0-TxRx-26572:00000000973056708772928788586433599834805582824679305722510643944440994124843010548996966437382374155IR-PCI-MSI-edgeenp175s0f0-TxRx-27573:000000007210889776570469542570685843990700000000689121962737164258904860820500067947507410779903110965446881082896IR-PCI-MSI-edgeenp175s0f0-TxRx-28574:0000000069041623996859721659487595731895576555091910958306433899596358IR-PCI-MSI-edgeenp175s0f0-TxRx-29575:000000007024010984468547610703055738677640491051056000000009767644522452938549705600909797695063786565IR-PCI-MSI-edgeenp175s0f0-TxRx-30576:00000000663956289963382424649110498163119343763129927329498110470000000076096869785119763219928912285044610852324849641666346741057101551776IR-PCI-MSI-edgeenp175s0f0-TxRx-31577:000000003074223493092683000003951822317349IR-PCI-MSI-edgeenp175s0f0
