ever-growing UDP socket buffer memory on 127.0.0.1 in Linux
Posted: Wed Oct 23, 2019 9:25 pm
Hi there, I'm facing an issue running a 3-node hyperconverged setup of the StarWind vSAN Linux VSA on vSphere, ESXI 6. I've noticed that on the 2 nodes where I have HA devices running, netstat indicates that a number of UDP sockets on 127.0.0.1 have an ever-growing/increasing amount of Recv-Q bytes. Odd thing is that I have configured StarWind.cfg file connections to use non-localhost IP addresses so I'm not even sure why these sockets are getting spawned in the first place.
This can be reproduced on both of my HA device nodes by running `ss -lp --udp`, pause, then execute it again and you'll see the numbers increasing in the Recv-Q column for a bunch of local addresses on host 127.0.0.1 and a high port number. The process owner appears to be wineserver/StarWindService.
then a few seconds later:
Firewall are disabled. This is the most recent version of the Linux VSA from Oct 2019.
Why is this happening? I've probably configured something incorrectly, but I have run out of ideas to troubleshoot it on this end. It is a problem because it eventually leads to socket buffer receive errors on the StarWind VSA guests, which ... seems like a bad thing.
Is there a way to eliminate the issue?
This can be reproduced on both of my HA device nodes by running `ss -lp --udp`, pause, then execute it again and you'll see the numbers increasing in the Recv-Q column for a bunch of local addresses on host 127.0.0.1 and a high port number. The process owner appears to be wineserver/StarWindService.
Code: Select all
State Recv-Q Send-Q Local Address:Port Peer Address:Port
UNCONN 0 0 *:64669 *:* users:(("wineserver",pid=10946,fd=482),("StarWindService",pid=10944,fd=616),("StarWindService",pid=10944,fd=610))
UNCONN 391680 0 127.0.0.1:40110 *:* users:(("wineserver",pid=10946,fd=184),("StarWindService",pid=10944,fd=199),("StarWindService",pid=10944,fd=197))
UNCONN 0 0 *:48466 *:* users:(("wineserver",pid=10946,fd=199),("StarWindService",pid=10944,fd=217),("StarWindService",pid=10944,fd=209))
UNCONN 0 0 *:40324 *:* users:(("wineserver",pid=10946,fd=277),("StarWindService",pid=10944,fd=334),("StarWindService",pid=10944,fd=330))
UNCONN 0 0 *:40970 *:* users:(("avahi-daemon",pid=9822,fd=13))
UNCONN 5423616 0 127.0.0.1:16450 *:* users:(("wineserver",pid=10946,fd=313),("StarWindService",pid=10944,fd=385),("StarWindService",pid=10944,fd=383))
UNCONN 0 0 *:49361 *:* users:(("wineserver",pid=10946,fd=373),("StarWindService",pid=10944,fd=464),("StarWindService",pid=10944,fd=457))
UNCONN 0 0 127.0.0.1:323 *:* users:(("chronyd",pid=9905,fd=1))
UNCONN 4976640 0 127.0.0.1:49885 *:* users:(("wineserver",pid=10946,fd=389),("StarWindService",pid=10944,fd=517),("StarWindService",pid=10944,fd=505))
UNCONN 0 0 *:25924 *:* users:(("wineserver",pid=10946,fd=207),("StarWindService",pid=10944,fd=234),("StarWindService",pid=10944,fd=227))
UNCONN 0 0 *:26078 *:* users:(("wineserver",pid=10946,fd=476),("StarWindService",pid=10944,fd=612),("StarWindService",pid=10944,fd=607))
UNCONN 0 0 *:51217 *:* users:(("wineserver",pid=10946,fd=376),("StarWindService",pid=10944,fd=468),("StarWindService",pid=10944,fd=458))
UNCONN 0 0 *:26835 *:* users:(("wineserver",pid=10946,fd=390),("StarWindService",pid=10944,fd=494),("StarWindService",pid=10944,fd=486))
UNCONN 5053440 0 127.0.0.1:35166 *:* users:(("wineserver",pid=10946,fd=326),("StarWindService",pid=10944,fd=403),("StarWindService",pid=10944,fd=402))
UNCONN 1708800 0 127.0.0.1:10850 *:* users:(("wineserver",pid=10946,fd=280),("StarWindService",pid=10944,fd=335),("StarWindService",pid=10944,fd=333))
UNCONN 372480 0 127.0.0.1:43680 *:* users:(("wineserver",pid=10946,fd=466),("StarWindService",pid=10944,fd=595),("StarWindService",pid=10944,fd=593))
UNCONN 5509632 0 127.0.0.1:11084 *:* users:(("wineserver",pid=10946,fd=194),("StarWindService",pid=10944,fd=220),("StarWindService",pid=10944,fd=216))
UNCONN 0 0 *:60357 *:* users:(("wineserver",pid=10946,fd=406),("StarWindService",pid=10944,fd=516),("StarWindService",pid=10944,fd=490))
UNCONN 5004288 0 127.0.0.1:44073 *:* users:(("wineserver",pid=10946,fd=391),("StarWindService",pid=10944,fd=495),("StarWindService",pid=10944,fd=493))
UNCONN 0 0 *:winshadow *:* users:(("wineserver",pid=10946,fd=505),("StarWindService",pid=10944,fd=648),("StarWindService",pid=10944,fd=647))
UNCONN 371712 0 127.0.0.1:19805 *:* users:(("wineserver",pid=10946,fd=378),("StarWindService",pid=10944,fd=469),("StarWindService",pid=10944,fd=467))
UNCONN 0 0 *:44450 *:* users:(("wineserver",pid=10946,fd=183),("StarWindService",pid=10944,fd=198),("StarWindService",pid=10944,fd=192))
UNCONN 398592 0 127.0.0.1:52755 *:* users:(("wineserver",pid=10946,fd=457),("StarWindService",pid=10944,fd=585),("StarWindService",pid=10944,fd=583))
UNCONN 0 0 *:61741 *:* users:(("wineserver",pid=10946,fd=311),("StarWindService",pid=10944,fd=384),("StarWindService",pid=10944,fd=378))
UNCONN 0 0 *:29007 *:* users:(("wineserver",pid=10946,fd=285),("StarWindService",pid=10944,fd=342),("StarWindService",pid=10944,fd=325))
UNCONN 0 0 *:21086 *:* users:(("wineserver",pid=10946,fd=465),("StarWindService",pid=10944,fd=594),("StarWindService",pid=10944,fd=590))
UNCONN 0 0 *:mdns *:* users:(("avahi-daemon",pid=9822,fd=12))
UNCONN 0 0 *:54575 *:* users:(("wineserver",pid=10946,fd=193),("StarWindService",pid=10944,fd=212),("StarWindService",pid=10944,fd=207))
UNCONN 0 0 *:21978 *:* users:(("wineserver",pid=10946,fd=325),("StarWindService",pid=10944,fd=399),("StarWindService",pid=10944,fd=391))
UNCONN 0 0 *:38428 *:* users:(("wineserver",pid=10946,fd=456),("StarWindService",pid=10944,fd=580),("StarWindService",pid=10944,fd=577))
UNCONN 371712 0 127.0.0.1:46776 *:* users:(("wineserver",pid=10946,fd=286),("StarWindService",pid=10944,fd=343),("StarWindService",pid=10944,fd=339))
UNCONN 372480 0 127.0.0.1:31041 *:* users:(("wineserver",pid=10946,fd=195),("StarWindService",pid=10944,fd=213),("StarWindService",pid=10944,fd=211))
UNCONN 9124608 0 127.0.0.1:22911 *:* users:(("wineserver",pid=10946,fd=374),("StarWindService",pid=10944,fd=465),("StarWindService",pid=10944,fd=461))
UNCONN 3816960 0 127.0.0.1:55975 *:* users:(("wineserver",pid=10946,fd=208),("StarWindService",pid=10944,fd=235),("StarWindService",pid=10944,fd=231))
UNCONN 5211648 0 127.0.0.1:23318 *:* users:(("wineserver",pid=10946,fd=483),("StarWindService",pid=10944,fd=617),("StarWindService",pid=10944,fd=615))
UNCONN 5137920 0 127.0.0.1:15239 *:* users:(("wineserver",pid=10946,fd=479),("StarWindService",pid=10944,fd=613),("StarWindService",pid=10944,fd=611))
Code: Select all
State Recv-Q Send-Q Local Address:Port Peer Address:Port
UNCONN 0 0 *:64669 *:* users:(("wineserver",pid=10946,fd=482),("StarWindService",pid=10944,fd=616),("StarWindService",pid=10944,fd=610))
UNCONN 432384 0 127.0.0.1:40110 *:* users:(("wineserver",pid=10946,fd=184),("StarWindService",pid=10944,fd=199),("StarWindService",pid=10944,fd=197))
UNCONN 0 0 *:48466 *:* users:(("wineserver",pid=10946,fd=199),("StarWindService",pid=10944,fd=217),("StarWindService",pid=10944,fd=209))
UNCONN 0 0 *:40324 *:* users:(("wineserver",pid=10946,fd=277),("StarWindService",pid=10944,fd=334),("StarWindService",pid=10944,fd=330))
UNCONN 0 0 *:40970 *:* users:(("avahi-daemon",pid=9822,fd=13))
UNCONN 5667840 0 127.0.0.1:16450 *:* users:(("wineserver",pid=10946,fd=313),("StarWindService",pid=10944,fd=385),("StarWindService",pid=10944,fd=383))
UNCONN 0 0 *:49361 *:* users:(("wineserver",pid=10946,fd=373),("StarWindService",pid=10944,fd=464),("StarWindService",pid=10944,fd=457))
UNCONN 0 0 127.0.0.1:323 *:* users:(("chronyd",pid=9905,fd=1))
UNCONN 5220864 0 127.0.0.1:49885 *:* users:(("wineserver",pid=10946,fd=389),("StarWindService",pid=10944,fd=517),("StarWindService",pid=10944,fd=505))
UNCONN 0 0 *:25924 *:* users:(("wineserver",pid=10946,fd=207),("StarWindService",pid=10944,fd=234),("StarWindService",pid=10944,fd=227))
UNCONN 0 0 *:26078 *:* users:(("wineserver",pid=10946,fd=476),("StarWindService",pid=10944,fd=612),("StarWindService",pid=10944,fd=607))
UNCONN 0 0 *:51217 *:* users:(("wineserver",pid=10946,fd=376),("StarWindService",pid=10944,fd=468),("StarWindService",pid=10944,fd=458))
UNCONN 0 0 *:26835 *:* users:(("wineserver",pid=10946,fd=390),("StarWindService",pid=10944,fd=494),("StarWindService",pid=10944,fd=486))
UNCONN 5297664 0 127.0.0.1:35166 *:* users:(("wineserver",pid=10946,fd=326),("StarWindService",pid=10944,fd=403),("StarWindService",pid=10944,fd=402))
UNCONN 1924608 0 127.0.0.1:10850 *:* users:(("wineserver",pid=10946,fd=280),("StarWindService",pid=10944,fd=335),("StarWindService",pid=10944,fd=333))
UNCONN 413184 0 127.0.0.1:43680 *:* users:(("wineserver",pid=10946,fd=466),("StarWindService",pid=10944,fd=595),("StarWindService",pid=10944,fd=593))
UNCONN 5753856 0 127.0.0.1:11084 *:* users:(("wineserver",pid=10946,fd=194),("StarWindService",pid=10944,fd=220),("StarWindService",pid=10944,fd=216))
UNCONN 0 0 *:60357 *:* users:(("wineserver",pid=10946,fd=406),("StarWindService",pid=10944,fd=516),("StarWindService",pid=10944,fd=490))
UNCONN 5248512 0 127.0.0.1:44073 *:* users:(("wineserver",pid=10946,fd=391),("StarWindService",pid=10944,fd=495),("StarWindService",pid=10944,fd=493))
UNCONN 0 0 *:winshadow *:* users:(("wineserver",pid=10946,fd=505),("StarWindService",pid=10944,fd=648),("StarWindService",pid=10944,fd=647))
UNCONN 412416 0 127.0.0.1:19805 *:* users:(("wineserver",pid=10946,fd=378),("StarWindService",pid=10944,fd=469),("StarWindService",pid=10944,fd=467))
UNCONN 0 0 *:44450 *:* users:(("wineserver",pid=10946,fd=183),("StarWindService",pid=10944,fd=198),("StarWindService",pid=10944,fd=192))
UNCONN 439296 0 127.0.0.1:52755 *:* users:(("wineserver",pid=10946,fd=457),("StarWindService",pid=10944,fd=585),("StarWindService",pid=10944,fd=583))
UNCONN 0 0 *:61741 *:* users:(("wineserver",pid=10946,fd=311),("StarWindService",pid=10944,fd=384),("StarWindService",pid=10944,fd=378))
UNCONN 0 0 *:29007 *:* users:(("wineserver",pid=10946,fd=285),("StarWindService",pid=10944,fd=342),("StarWindService",pid=10944,fd=325))
UNCONN 0 0 *:21086 *:* users:(("wineserver",pid=10946,fd=465),("StarWindService",pid=10944,fd=594),("StarWindService",pid=10944,fd=590))
UNCONN 0 0 *:mdns *:* users:(("avahi-daemon",pid=9822,fd=12))
UNCONN 0 0 *:54575 *:* users:(("wineserver",pid=10946,fd=193),("StarWindService",pid=10944,fd=212),("StarWindService",pid=10944,fd=207))
UNCONN 0 0 *:21978 *:* users:(("wineserver",pid=10946,fd=325),("StarWindService",pid=10944,fd=399),("StarWindService",pid=10944,fd=391))
UNCONN 0 0 *:38428 *:* users:(("wineserver",pid=10946,fd=456),("StarWindService",pid=10944,fd=580),("StarWindService",pid=10944,fd=577))
UNCONN 412416 0 127.0.0.1:46776 *:* users:(("wineserver",pid=10946,fd=286),("StarWindService",pid=10944,fd=343),("StarWindService",pid=10944,fd=339))
UNCONN 413184 0 127.0.0.1:31041 *:* users:(("wineserver",pid=10946,fd=195),("StarWindService",pid=10944,fd=213),("StarWindService",pid=10944,fd=211))
UNCONN 10227456 0 127.0.0.1:22911 *:* users:(("wineserver",pid=10946,fd=374),("StarWindService",pid=10944,fd=465),("StarWindService",pid=10944,fd=461))
UNCONN 4061184 0 127.0.0.1:55975 *:* users:(("wineserver",pid=10946,fd=208),("StarWindService",pid=10944,fd=235),("StarWindService",pid=10944,fd=231))
UNCONN 5455872 0 127.0.0.1:23318 *:* users:(("wineserver",pid=10946,fd=483),("StarWindService",pid=10944,fd=617),("StarWindService",pid=10944,fd=615))
UNCONN 5382144 0 127.0.0.1:15239 *:* users:(("wineserver",pid=10946,fd=479),("StarWindService",pid=10944,fd=613),("StarWindService",pid=10944,fd=611))
Why is this happening? I've probably configured something incorrectly, but I have run out of ideas to troubleshoot it on this end. It is a problem because it eventually leads to socket buffer receive errors on the StarWind VSA guests, which ... seems like a bad thing.
Is there a way to eliminate the issue?