From deepak-abraham.tom at hpe.com Wed Apr 3 21:07:37 2024 From: deepak-abraham.tom at hpe.com (Tom, Deepak Abraham) Date: Thu, 4 Apr 2024 04:07:37 +0000 Subject: 2nd RTM_DELLINK notification is always 1 second delayed Message-ID: Hi! I have a system configured with 2 physical eth interfaces connected to a switch. When I reboot the switch, I see that the userspace RTM_DELLINK notifications for the interfaces are always 1 second apart although both links actually go down almost simultaneously! >From /var/log/messages (logs have the granularity of a second, but can see both links go down almost simultaneously) - Apr 3 12:20:49 m2-dl360g10-244 kernel: mlx5_core 0000:5d:00.0 eno5np0: Link down Apr 3 12:20:49 m2-dl360g10-244 kernel: mlx5_core 0000:5d:00.1 eno6np1: Link down Monitoring userspace link events with ip command (linkdown is always 1 second apart; order varies from run to run) - [root]$ ip -t monitor link Timestamp: Wed Apr 3 12:20:49 2024 365932 usec 6: eno5np0: mtu 1500 qdisc mq state DOWN group default link/ether 88:e9:a4:54:89:4c brd ff:ff:ff:ff:ff:ff altname enp93s0f0np0 Timestamp: Wed Apr 3 12:20:50 2024 382616 usec 7: eno6np1: mtu 1500 qdisc mq state DOWN group default link/ether 88:e9:a4:54:89:4d brd ff:ff:ff:ff:ff:ff altname enp93s0f1np1 I wrote a simple test program that uses a netlink socket to get these link event notifications in the userspace. The exact same behavior can be seen there as well - 2nd interfaces' userspace notification is always delayed by 1 second. This behavior is consistent across Debian 11, 12 and RHEL9 distros (with their corresponding stock libnl). This behavior is also not specific to a particular system. It should be easily reproducible on all hardware. The subsequent RTM_NEWLINK notifications when the switch comes back up are however only delayed by a few microseconds between each other, which is as expected. Anyone know why there is this 1 sec delay between RTM_DELLINK notifications in the userspace always, although both physical interfaces go down almost simultaneously? Also is there a way to not have this delay? Any help is appreciated. Thanks, Deepak