Infrastructure identifiers have been anonymized. IP addresses use the documentation-only range 192.0.2.0/24, and hossein.cloud stands in for the original service domain. The DNS records below are illustrative, not the domain’s actual configuration.
I had four HAProxy servers on the same internal network: 192.0.2.11, 192.0.2.12, 192.0.2.13, and 192.0.2.14. Each server also had 192.0.2.100 configured on a loopback interface. CoreDNS returned that shared address for hossein.cloud.
hossein.cloud. IN A 192.0.2.100The intention was straightforward: all four HAProxy instances would receive traffic at the same time. The result was not. Traffic was uneven, and when one node went down, part of the network continued sending frames for 192.0.2.100 to that node’s MAC address.
HAProxy was not the component choosing a destination. DNS returned one IP, and the Layer 2 network still needed one MAC address for that IP. In effect, the design delegated node selection to an ARP race: I had created four possible answers without adding a control plane that could select healthy nodes and distribute flows.
The design looked active-active.
CoreDNS
hossein.cloud -> 192.0.2.100
|
v
Layer 2 broadcast domain
|
ARP: "Who has 192.0.2.100? Tell the requester."
/ | | \
/ | | \
192.0.2.11 192.0.2.12 192.0.2.13 192.0.2.14
HAProxy A HAProxy B HAProxy C HAProxy D
lo: 192.0.2.100 lo: 192.0.2.100 lo: 192.0.2.100 lo: 192.0.2.100All four nodes could run HAProxy, but “all nodes are active” is not the same as “the network has a valid active-active path to all nodes.” A shared address on four loopbacks only gives each kernel a local route for the VIP. It does not define how upstream devices should divide connections, detect an unhealthy HAProxy process, or stop using a failed next hop.
Follow the packet before blaming the load balancer.
A client first resolves hossein.cloud and receives 192.0.2.100. Its routing table then decides where the packet goes. If the VIP is on the client’s local subnet, the client resolves the VIP directly with ARP. If the client is on another subnet, it sends the packet to its default gateway; the router on the final hop may be the device that resolves 192.0.2.100 with ARP.
This distinction matters during troubleshooting. The stale entry may be on a client, a router, a firewall, a hypervisor, or another Layer 3 boundary. Flushing the neighbor table on an HAProxy server does nothing if a core router is the device holding the VIP-to-MAC mapping.
Once a device has selected a MAC address, the Ethernet frame is unicast to that MAC. The switch forwards it according to its MAC address table. The switch does not inspect the four HAProxy processes and choose the least busy one.
ARP maps an IPv4 address to one Layer 2 destination.
ARP answers a local question: which MAC address should receive frames for this IPv4 address on this Layer 2 segment?
- The sender broadcasts an ARP request: “Who has
192.0.2.100?” - A host that considers the address local replies with its MAC address.
- The sender stores the IP-to-MAC mapping in its neighbor table.
- Subsequent packets are sent to that MAC until the entry is refreshed, replaced, or expires.
With one legitimate owner, this is predictable. With four separate hosts claiming the same IP, multiple hosts may answer the same request. With Linux’s default arp_ignore=0 behavior, all four kernels are eligible to reply for a local address configured on loopback. Which answer is accepted and retained depends on timing, the requesting device, kernel behavior, and network controls. Different clients can learn different MAC addresses. A centralized router may learn only one and send most routed traffic to a single HAProxy.
ARP answers “where is this IP on this segment?” It does not answer “which healthy HAProxy should serve this connection?”
That is why ARP is not a load-balancing algorithm. It has no health model, capacity signal, connection hashing, or requirement to split requests evenly. Even if different requesters happen to learn different nodes, the distribution is incidental rather than controlled.
The neighbor cache explains the sticky failure.
Linux exposes ARP state through the neighbor table. An entry can move through states such as REACHABLE, STALE, DELAY, PROBE, and FAILED. A stale entry is not immediately deleted; it can still be used while the kernel confirms reachability.
If a router has learned that 192.0.2.100 is behind HAProxy B’s MAC, it can continue using that mapping after B fails. The exact duration depends on neighbor timers and traffic. If the host is still alive but only the HAProxy process is down, its kernel may continue answering ARP indefinitely even though nothing useful is listening on the service port.
The neighbor table and the switch forwarding table are related but different. A client or router caches IP-to-MAC state; a switch learns MAC-to-port state. A switch noticing that a port went down does not rewrite the router’s neighbor entry to another HAProxy MAC, and the surviving nodes do not normally accept frames addressed to the failed node’s physical MAC.
A powered-off node, a failed interface, and a failed HAProxy process therefore produce different evidence. The symptom is consistent with stale ARP, but I still verify where the mapping lives and whether the failure is at Layer 2, Layer 3, or Layer 4 before calling it the root cause.
Gratuitous ARP helps move an address; it does not distribute it.
A Gratuitous ARP is an unsolicited, link-local ARP announcement for a host’s own address. It can update peer neighbor caches and switch learning after an IP moves to a new interface or server. High-availability tools commonly send several announcements after failover so the new owner becomes reachable quickly. It does not cross routers, and whether an unsolicited mapping is accepted varies by operating system and security policy.
This is useful when ownership is exclusive: the old owner stops serving the VIP, the new owner takes it, and the network is told about the change. If four active nodes keep announcing the same VIP, Gratuitous ARP can make the neighbor mapping churn between MAC addresses. Which mapping is retained remains implementation-dependent, and the result is still not traffic distribution.
ARP flux settings can reduce ambiguity, not create active-active.
Linux follows a weak host model by default: an address belongs to the host more than to one specific interface. On a multi-homed server this can cause ARP flux, where the host answers for an address through an unexpected interface. A VIP on lo can also influence replies seen on a physical interface.
Settings such as arp_ignore and arp_announce can make this behavior stricter. For example, arp_ignore=1 limits replies to addresses configured on the receiving interface, while arp_announce=2 chooses a more appropriate source address for outbound ARP requests. The effective behavior depends on both conf/all and conf/<interface>, so it must be tested against the interface and routing design. A strict arp_ignore policy can suppress every reply for a VIP that exists only on loopback and break direct reachability.
These settings are useful when one designated device should answer for a VIP, such as an LVS or VRRP design. They cannot decide which of four independent HAProxy nodes is healthy, hash flows across them, or withdraw a failed path. Tuning ARP behavior may hide one symptom while leaving the architecture undefined.
Inspect the device that actually resolves the VIP.
I start on the client or final-hop router that is expected to resolve 192.0.2.100. On Linux, these commands show the current neighbor and routing decisions:
ip neigh
ip neigh show 192.0.2.100
ip route get 192.0.2.100
arp -aip neighlists neighbor entries, their interfaces, MAC addresses, and state.ip neigh show 192.0.2.100narrows the output to the shared VIP. I compare the learned MAC with the four server interfaces.ip route get 192.0.2.100shows the selected route, source address, next hop, and egress interface. It does not show the destination MAC, but it tells me which neighbor table and interface matter.arp -ais the older view. It is useful on systems where it is installed, butip neighexposes more state.
I also capture ARP while clearing or reproducing the issue. Competing replies from different MAC addresses make the duplicate ownership visible:
sudo tcpdump -eni <interface> 'arp or host 192.0.2.100'On the HAProxy nodes, I verify interface addresses, local routes, listening sockets, and whether more than one node replies to the same ARP request. On routers and firewalls, I check for proxy ARP, static neighbor entries, EVPN suppression, or another feature that may own the mapping.
Deleting ARP state is a diagnostic, not a repair.
To force one entry to be resolved again on Linux, I can delete it from the interface that owns the neighbor relationship:
sudo ip neigh del 192.0.2.100 dev <interface>To flush the full neighbor table:
sudo ip -s -s neigh flush allA full flush affects every neighbor, can create a burst of ARP resolution, and may cause brief disruption, so I use it carefully and usually prefer deleting one entry. More importantly, the next ARP request can learn the same failed MAC or another arbitrary HAProxy. A successful test after a flush proves that neighbor state was involved; it does not fix the design.
Active-passive makes one Layer 2 owner explicit.
The simpler correction is Keepalived with VRRP. All HAProxy nodes can have the same configuration, but only the elected active router—called MASTER in a typical Keepalived configuration—owns and advertises 192.0.2.100. Backups monitor that owner, and a track_script or equivalent health check must also monitor the local HAProxy process. When the owner fails, a backup takes the VIP and sends Gratuitous ARP to update the network.
CoreDNS: hossein.cloud -> 192.0.2.100
|
v
VRRP virtual IP owner
|
+---------------+---------------+
| |
HAProxy A: MASTER HAProxy B/C/D: BACKUP
owns 192.0.2.100 do not answer for the VIP
Failover: elect new master -> add VIP -> send Gratuitous ARPThis produces deterministic ownership and reliable failover, but it is active-passive for that VIP. It does not use all four nodes for the same service at once. Multiple VRRP groups and multiple VIPs can spread different services or DNS records across masters, but each VIP should still have one owner at a time.
Real active-active needs a Layer 3 control plane.
If all four HAProxy nodes must receive traffic for one service IP, I would keep the VIP as a loopback /32, place it in a routed service prefix, and advertise it from each node to upstream routers using BGP. With multipath enabled, the routers install equal-cost paths and stable per-flow ECMP selects a next hop. The selected next hops are the servers’ unique routed addresses, so ARP resolves 192.0.2.11, .12, .13, or .14 rather than asking four hosts to compete for one Layer 2 identity.
CoreDNS: hossein.cloud -> 192.0.2.100
|
v
Upstream L3 routers
ECMP route for 192.0.2.100/32
/ / \ \
/ / \ \
next hop .11 .12 .13 .14
HAProxy A HAProxy B HAProxy C HAProxy D
advertises the service /32 while healthyThis design requires clients to reach the VIP through the routing layer. If a client still considers 192.0.2.100 directly connected, it can ARP for the VIP and bypass ECMP entirely. In practice, the service IP should come from a routed service prefix—or be reached through an explicit host route—rather than sit inside the clients’ ordinary Layer 2 subnet.
The route must be tied to service health. If HAProxy is unhealthy, that node should withdraw the prefix. BFD can speed up detection of a failed routing peer or link, but application health still needs its own signal. New flows then avoid the withdrawn path.
ECMP is flow-aware routing, not perfect quarter-by-quarter request balancing. Hash inputs, NAT, long-lived connections, and unequal flow sizes can still create skew. For TCP, the design should use stable per-flow hashing rather than per-packet ECMP, which can reorder packets. Existing HAProxy connections can fail when a next hop disappears; routing can steer new flows after convergence, but it cannot migrate established TCP state. The improvement is that path selection and failure removal are explicit, observable, and controlled by routing.
DNS round robin is simpler, but failover is eventually consistent.
Another option is to publish the four unique HAProxy addresses instead of one shared VIP:
hossein.cloud. 30 IN A 192.0.2.11
hossein.cloud. 30 IN A 192.0.2.12
hossein.cloud. 30 IN A 192.0.2.13
hossein.cloud. 30 IN A 192.0.2.14Resolvers can rotate the record order, and clients choose an address. This removes the duplicate Layer 2 ownership and can provide coarse distribution without BGP. CoreDNS record ordering is not a service-health decision, however, and DNS is still not a connection-aware load balancer.
Recursive resolvers, operating systems, runtimes, and applications all cache DNS. TTL is a request, not a guarantee that every client will refresh on time. Some clients pin one address, some try only the first, and existing connections ignore DNS changes. A health-aware DNS system can remove a failed record, but cached answers can continue sending traffic to it until they expire. Lowering the TTL of the current single 192.0.2.100 record would not change any of this: every answer would still lead to the same ARP problem.
The recommended design depends on the real requirement.
- If one HAProxy can handle the load: use Keepalived/VRRP, one VIP owner, HAProxy health tracking, and tested Gratuitous ARP behavior. This is the smallest reliable design.
- If all four nodes must be active behind one IP: move the service boundary to Layer 3. Put the VIP in a routed service prefix, advertise it as a
/32, use BGP with per-flow ECMP, and withdraw the route when the service is unhealthy. Ensure every client path traverses that Layer 3 decision point. - If routing integration is unavailable: use multiple unique A records with health-aware DNS, or place a proper redundant Layer 4 load-balancing tier in front. Accept that DNS failover is cache-dependent.
For this network, my first question would be whether one HAProxy node has enough capacity. If yes, VRRP is operationally simpler. If active-active capacity is a hard requirement and the core network supports dynamic routing, BGP/ECMP is the clean design. I would not keep four independent Layer 2 owners for 192.0.2.100 and try to tune ARP until it behaves like a load balancer.
Conclusion.
The failure was not that a loopback address cannot be used as a service VIP. Loopbacks are common in routed anycast and ECMP designs. The failure was giving the same VIP to four hosts on one Layer 2 segment without a mechanism that controlled ownership or advertised multiple healthy routes.
ARP can locate a neighbor. Gratuitous ARP can announce that an address moved. Linux ARP settings can reduce ambiguous replies. None of them provides active-active load balancing by itself. A correct design needs either one elected owner with VRRP, multiple routed next hops with ECMP/BGP, or multiple unique addresses with the understood limitations of DNS caching.