服務器日志出現大量以下錯誤:
ullorder-9000][2020-03-17 11:00:09.636][l-79965715][ERROR][c.x.r.r.n.i.n.s.NettyHttpServerHandler][OPR:|MAT:|MAN:|TID:|SKU:][] - xxl-rpc request data empty.
com.xxl.rpc.util.XxlRpcException: xxl-rpc request data empty.
at com.xxl.rpc.remoting.net.impl.netty_http.server.NettyHttpServerHandler.process(NettyHttpServerHandler.java:78)
at com.xxl.rpc.remoting.net.impl.netty_http.server.NettyHttpServerHandler.access$000(NettyHttpServerHandler.java:27)
at com.xxl.rpc.remoting.net.impl.netty_http.server.NettyHttpServerHandler$1.run(NettyHttpServerHandler.java:51)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[pullorder-9000][2020-03-17 11:00:09.638][1530585378][ERROR][c.x.r.r.n.i.n.s.NettyHttpServerHandler][OPR:|MAT:|MAN:|TID:|SKU:][] - xxl-rpc request data empty.
com.xxl.rpc.util.XxlRpcException: xxl-rpc request data empty.
at com.xxl.rpc.remoting.net.impl.netty_http.server.NettyHttpServerHandler.process(NettyHttpServerHandler.java:78)
at com.xxl.rpc.remoting.net.impl.netty_http.server.NettyHttpServerHandler.access$000(NettyHttpServerHandler.java:27)
at com.xxl.rpc.remoting.net.impl.netty_http.server.NettyHttpServerHandler$1.run(NettyHttpServerHandler.java:51)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[pullorder-9000][2020-03-17 11:00:09.639][pGroup-5-1][ERROR][c.x.r.r.n.i.n.s.NettyHttpServerHandler][OPR:|MAT:|MAN:|TID:|SKU:][] - >>>>>>>>>>> xxl-rpc provider netty server caught exception
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1125)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:617)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:534)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
上十萬的錯誤日志,著實不妥。先排查了這個錯誤不影響正常的業務操作,但是大量錯誤日志會導致日志文件非常大。網上找了一下原因,有博客說換了XXL_JOB的版本號就沒事,舊版本的XXL_JOB是使用@JobHander注解,新版本的XXL_JOB已經改成了@Xxl_job注解....覺得此方案不可行,線上環境,暫不采用此方案。
然后就采用抓包方式對線上環境進行抓包,抓包步驟:
1.ip addr可以查詢到網卡名稱
[root@centos64 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:8b:1f:3d brd ff:ff:ff:ff:ff:ff
inet 192.168.67.xxx/24 brd 192.168.67.xxx scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::75c3:3b6e:7a05:a954/64 scope link noprefixroute
valid_lft forever preferred_lft forever
上圖中ens33就是網卡名稱
2.執行抓包命令:sudo tcpdump -i 網卡名 -s 0 -w 文件名.pcap
l注意,需要使用管理員賬號才有權限執行此命令,執行后會生成一個文件名.pcap文件,需要從服務器中拉到本地進行分析。
3.使用wireshark分析
Wireshark(前稱Ethereal)是一個網絡數據包分析軟件。使用教程可參考https://www.cnblogs.com/koushuige/p/9212033.html博主的。
把文件拉到本地,用wireshark這個軟件就可以排查網絡問題。
網絡分析.png
果不其然,發現有機器不斷訪問執行器端口9999端口,導致的此異常,雖然對業務操作沒影響,但是產生大量的錯誤日志.....導致日志文件大小暴增,占用服務器空間。后經詢問,是公司的網絡部門進行端口掃描,自動暫停功能壞了導致一直訪問,后面停止端口掃描,異常消失。。。。。