Azure Load Balancer Troubleshooting Guide

When a service behind Azure Load Balancer becomes unreachable or behaves unexpectedly, the cause is rarely a single misconfiguration. It is usually a chain of assumptions that break at one layer—DNS, frontend IP, load‑balancing rule, health probe, network security group, route, host firewall, listener, application logic, or outbound connectivity. Successful incident response depends on isolating the exact failing layer without jumping to conclusions.

This guide is designed for engineers who are already familiar with Azure Load Balancer fundamentals and need a systematic, evidence‑driven method for diagnosing incidents. It groups symptoms by failure domain, describes the evidence required to distinguish causes, and recommends safe, reversible remediation steps. Every diagnostic action is tied to a hypothesis; no “check everything” lists.

Incident‑response principles

Before opening a single blade or command line, classify the incident:

• Scope: one client, one subnet, one backend, all backends, or all frontends.
• Direction: inbound (client → frontend), east‑west (internal frontend), or outbound (backend → internet).
• Protocol: TCP or UDP.
• State: new connections fail, existing connections fail, or both.
• Exposure: public frontend or internal (private) frontend.
• Time pattern: continuous, intermittent, correlated with a deployment, or triggered by a known failure event.
• Recent changes: rule modifications, NSG updates, route changes, VM or scale set replacements, certificate rotations, application deployments.

Preserve evidence before making any configuration change. Record the exact timestamp, affected frontend IP or DNS name, source network of failing clients, and the expected backend port. Screenshots of health probe status, NSG effective rules, and recent Activity Log entries are invaluable for post‑incident analysis.

A disciplined investigation follows a layered sequence:

1. Define the exact failing flow (source, destination, protocol, port).
2. Verify that the frontend IP resolves correctly and is reachable.
3. Check backend health and probe status.
4. Confirm the load‑balancing rule binds the frontend to the correct backend pool and port.
5. Inspect NSGs, route tables, and host firewalls.
6. Validate that the application is listening and ready.
7. Investigate outbound connectivity if backend‑to‑dependency failures are suspected.
8. Compare healthy and unhealthy instances.
9. Remediate cautiously and validate recovery at every layer.

Do not change anything until you have a hypothesis. When you do change something, change one variable at a time, verify the result, and document it.

Diagnostic layers

The following Mermaid diagram shows the layers that a packet traverses. A failure at any layer can break connectivity, but the evidence is different at each step.

Health probe traffic (dashed line) follows a different path from client traffic; it is originated by the Azure platform and must be allowed through NSGs and host firewalls separately. A successful probe does not mean that the application can process real requests—only that the probe endpoint responded correctly.

Data to collect before making changes

Before you modify anything, capture the following:

• Incident start time (UTC) and duration.
• Affected frontend IP or DNS name, public or internal.
• Failing protocol and port (e.g., TCP 443).
• Client source IP ranges and network context.
• Expected backend port and protocol.
• Current health probe configuration (protocol, port, path, interval, unhealthy threshold).
• Backend pool membership and health status per instance.
• Load‑balancing rule details (frontend IP:port, backend port, protocol, probe association).
• NSG effective security rules on the backend subnet and NICs.
• Effective routes on the backend subnet.
• Host listener state (e.g., ss -tlnp, netstat -ano on a sample backend VM).
• Application logs and system logs from the same timeframe.
• Any recent Azure Activity Log entries for the load balancer, public IP, NSG, or VM scale set.
• Outbound connectivity test results if the backend calls external services.

Collect the same information from a known‑healthy backend or a healthy region for comparison.

Symptom‑based troubleshooting

Each symptom section follows the same structure: a description of the symptom, the most likely fault domains, the first checks to perform, how to interpret the evidence, safe remediation options, and a validation checklist.

1. Public frontend is unreachable

Symptom: External clients cannot connect to a public load balancer frontend IP. TCP connection attempts timeout, are reset, or are refused.

Most likely fault domains: DNS, frontend IP configuration, load‑balancing rule, backend health, NSGs, client network.

First checks:

• Resolve the hostname from a client; verify the returned IP matches the public frontend IP.
• From the same client network, test TCP connectivity to the frontend IP:port using telnet, nc, or Test-NetConnection.
• Observe whether the connection times out, is refused, or resets. A timeout often indicates no listener or a firewall silently dropping packets; a reset suggests a network device or host actively rejecting the connection; a refusal means something sent a TCP RST, often a host with no listener.
• Check the load balancer’s frontend IP configuration in the Azure portal/CLI; confirm it is indeed a public IP and that the IP is correctly assigned to the load‑balancing rule.
• Verify that the load‑balancing rule for the expected port exists and is associated with a backend pool and health probe.
• Inspect the health probe status: are any backends healthy? If all backends are unhealthy, the load balancer stops forwarding new flows.
• Check NSG rules on the backend subnet; they must allow inbound traffic from the client source ranges on the backend port (the load balancer preserves the client’s source IP).

Evidence interpretation:

• DNS returning an unexpected IP → DNS misconfiguration or stale record. Fix DNS, not the load balancer.
• TCP timeout with all backends healthy → traffic is likely blocked by a firewall, NSG, or route before reaching the frontend. Check Azure Firewall or upstream on‑premises firewalls.
• Connection refused → the backend host OS is rejecting the connection, often because no listener is bound to that port or an NSG rule explicitly denies with a reset.
• Connection succeeds but application returns errors → the probe is passing, but the application is unhealthy. Investigate the application layer (Symptom 5).

Safe remediation:

• Correct any DNS record mismatch.
• If the frontend IP or rule is missing, create it via Infrastructure as Code—never manually in production.
• If all backends are unhealthy, troubleshoot the health probe (Symptom 3) before touching the rule.
• Do not disable NSGs; instead, add the specific missing allow rule.

Validation: After remediation, verify DNS resolves correctly, a new TCP connection succeeds, and the application returns the expected response.

2. Internal frontend is unreachable

Symptom: Clients inside the VNet, peered VNets, or connected via VPN/ExpressRoute cannot reach the internal load balancer frontend IP.

Most likely fault domains: Private DNS, VNet peering, routing, NSG, backend health.

First checks:

• Verify the client resolves the internal frontend hostname to the expected private IP. For private DNS, check Azure Private DNS zone records or custom DNS servers.
• Confirm the client is in the same VNet as the load balancer, or that VNet peering is configured and allows traffic (check peering settings for “Allow gateway transit” or “Use remote gateways” if needed).
• Test TCP connectivity to the private frontend IP:port from a client in the same VNet. If this works but cross‑VNet clients fail, the problem is in peering, routing, or NSGs on the peering link.
• Check effective routes on the client subnet: ensure a route exists to the frontend subnet, and no User‑Defined Route (UDR) forces traffic through an appliance that blocks it.
• Inspect NSG rules on the backend subnet and, if applicable, on the client subnet. The internal load balancer does not translate the client’s source IP; the NSG must allow traffic from the client subnet’s IP range on the backend port.
• Verify that the load‑balancing rule exists and that at least one backend is healthy.

Evidence interpretation:

• Private DNS resolving to the wrong IP → fix the Private DNS zone or the client’s DNS configuration.
• TCP timeout only from peered VNets → peering may be missing or misconfigured. Confirm peering status and that “Allow forwarded traffic” is enabled if traffic transits a hub.
• Connection refused from an on‑premises client → the VPN/ExpressRoute gateway may be dropping traffic; check gateway configuration and firewall rules on the on‑premises side.
• All backends unhealthy → same as Symptom 3.

Safe remediation:

• Update Private DNS records if they are wrong.
• Adjust peering settings or NSG rules after confirming the exact source range that should be permitted.
• Avoid adding a public IP to the load balancer as a workaround; it changes the trust boundary.

Validation: From each expected client network, verify TCP connectivity and application response.

3. All backends are unhealthy

Symptom: The health probe status for a backend pool shows 0 healthy instances. The load balancer stops forwarding new flows to that pool.

Most likely fault domains: Probe configuration, NSG blocking probe traffic, host firewall, listener binding, application readiness.

First checks:

• Compare the probe’s configured port, protocol (TCP/HTTP/HTTPS), and path with the actual listener on a backend VM.
• Test the probe manually from a client in the same network: can you establish a TCP connection to the backend VM on the probe port? For an HTTP probe, does curl http://<backend-ip>:<probe-port>/<probe-path> return 200?
• Check NSG effective rules on the backend subnet or NIC. There must be an inbound allow rule for the AzureLoadBalancer service tag on the probe port. If the rule is missing, the probe fails silently.
• Verify the application listener is bound to the correct interface. Use ss -tlnp (Linux) or netstat -ano (Windows). A service bound to 127.0.0.1 will not accept traffic from the load balancer.
• Examine host firewall rules (iptables, Windows Firewall). The probe port must be open.
• For HTTPS probes, certificate validation is not performed, but a TLS handshake failure (e.g., wrong protocol version) can cause a probe failure. Use openssl s_client -connect <ip>:<port> to diagnose.
• Review application logs: is the health endpoint returning 200? Is the application still initialising and returning 503?
• Check if the probe is configured with an unusually long path or one that triggers expensive operations; the probe may time out.

Evidence interpretation:

• TCP probe fails to connect from test client → listener not running, port blocked, or NSG issue.
• HTTP probe returns non‑200 → the health endpoint is misconfigured or the application is not ready. Do not change the probe; fix the endpoint or application.
• Probe succeeds from test client but platform reports unhealthy → NSG or route blocking AzureLoadBalancer source. Verify the NSG rule.
• All backends became unhealthy after a deployment → the new application version may not be starting fast enough, or the health endpoint changed. Roll back the deployment and check probe configuration.

Safe remediation:

• Correct the NSG rule to allow AzureLoadBalancer on the probe port.
• Fix the listener binding (bind to 0.0.0.0 or the appropriate interface).
• Adjust the probe’s port and path only if they genuinely do not match the application. Changing the probe while the application is broken can mask the real problem.
• If the application needs more startup time, increase the probe’s unhealthy threshold temporarily while the application team improves readiness signaling.

Validation: After remediation, monitor the probe status until it returns to the expected healthy count, then send test application traffic.

4. Only some backends are unhealthy

Symptom: A subset of backend instances is marked unhealthy; the rest are healthy.

Most likely fault domains: Instance‑specific configuration, network interface, NSG, host firewall, application state, availability zone, scale set issues.

First checks:

• Compare the VM size, OS version, application version, and configuration between a healthy and an unhealthy instance.
• Check whether the unhealthy instances were recently created (e.g., scale set scaling out). New instances may still be initialising.
• Verify the NSG on the NIC of an unhealthy instance (if a NIC‑level NSG is applied). Effective security rules may differ from the subnet‑level NSG.
• Examine the host firewall and listener on the unhealthy instance.
• Look for resource constraints (CPU, memory, disk) that might be causing the health probe to fail only on some instances.
• Check the instance’s application logs for probe‑related errors.
• If the unhealthy instances are in the same availability zone, a zonal issue may be occurring (unlikely, but possible). Test by adding an instance in a different zone.

Evidence interpretation:

• Unhealthy instances only after scale‑out → new instances not ready; probe threshold may be too aggressive.
• Unhealthy instances have a different application version → deployment issue.
• Unhealthy instances have no probe logs → probe traffic never reached the host (NSG or route difference).
• Unhealthy instances are in one zone → possible zone‑specific platform issue; check Azure Service Health.

Safe remediation:

• Bring the unhealthy instances in line with the healthy ones (same configuration, same image).
• If they are still initialising, extend the probe threshold temporarily.
• Do not remove healthy instances to “rebalance”; fix the root cause on the unhealthy ones.

Validation: Wait for the probe to turn green on the affected instances, then send traffic to confirm they handle requests correctly.

5. Backend is healthy but application requests fail

Symptom: Health probe passes, but when clients send real application traffic, they receive errors (HTTP 5xx, TLS failures, timeouts after connection establishment, or unexpected responses).

Most likely fault domains: Application logic, downstream dependency failure, TLS/certificate misconfiguration, backend port mismatch, deep health probe covering too much.

First checks:

• Review the health probe endpoint. Does it only check a local “I’m alive” signal and not exercise the application’s critical code paths? A probe that succeeds while the application cannot serve real requests is too shallow.
• Compare the backend port in the load‑balancing rule with the port the application listens on. A rule may map frontend 443 to backend 8443, but the application might only be on 8080.
• Capture application logs for a failed real request. What error is the application returning? Is it throwing exceptions, timing out, or returning a custom error page?
• Test the application directly from a client inside the VNet using the backend’s private IP and the exact application port. If that succeeds, the problem is in the load balancer’s rule or port mapping.
• Check downstream dependencies: can the backend reach its database, cache, or external API? If a dependency is down, the health probe should not be passing—unless the probe is too shallow.
• For TLS issues, verify the certificate presented by the backend matches the hostname the client expects. Use openssl s_client.

Evidence interpretation:

• Direct‑to‑backend test works → rule or port mapping error. Correct the rule’s backend port.
• Direct‑to‑backend test fails → application problem, not load balancer. Investigate application code, configuration, and dependencies.
• Application returns 503 and health probe passes → the probe is too shallow. The application is up but unable to serve requests (e.g., database connection pool exhausted). The probe should check a minimal dependency that is critical for serving.
• TLS errors → certificate invalid for the client hostname. The load balancer does not terminate TLS, so the backend must present a valid certificate for the domain the client used.

Safe remediation:

• Update the load‑balancing rule’s backend port if it is wrong.
• Do not immediately add dependency checks to the health probe; first fix the application or its dependency. If the probe is made deeper, test in staging to avoid cascading failures.
• For TLS issues, update the backend certificate.

Validation: Send real application traffic from a client and verify that the expected business response is returned, not just an HTTP 200 from a health check.

6. Traffic reaches only one backend

Symptom: All or most client traffic appears to be handled by a single backend instance, even though multiple instances are healthy.

Most likely fault domains: Session persistence, flow hash characteristics, test methodology, long‑lived connections.

First checks:

• Check the load‑balancing rule’s session persistence setting. If “Client IP” or “Client IP and protocol” is selected, all flows from the same source IP go to the same backend. This is expected.
• Determine whether the observed traffic uses a small number of unique client IPs or long‑lived connections. A single client using HTTP/2 or gRPC multiplexing will create many requests on one TCP connection, all of which go to the same backend.
• Use production metrics, not a single test. Look at per‑backend connection counts and application‑level request metrics. A slight imbalance is normal; a 90/10 split may indicate a problem or simply a small number of persistent connections.
• If session persistence is off, check the health of all backends. If one backend is unhealthy, all traffic goes to the remaining healthy ones. A recently recovered backend may only handle new flows.

Evidence interpretation:

• Session persistence enabled → expected behaviour. Do not disable it unless the application is stateless and you can accept the changed behaviour.
• One healthy backend with others unhealthy → fix the unhealthy backends.
• Imbalance with no session persistence and many clients → could be a coincidence of the 5‑tuple hash; over time it should roughly balance. If it never balances, check for a bug in the test.

Safe remediation:

• If session persistence is not required, change the distribution mode to “None” (5‑tuple hash). Test the application’s session handling beforehand.
• Ensure all backends are healthy and equally capable.

Validation: Monitor per‑backend connection counts over a longer period with real production traffic.

7. New connections fail but existing connections continue

Symptom: Services with long‑lived connections (e.g., WebSocket, gRPC streams, database replication) remain connected, but new connection attempts fail.

Most likely fault domains: Health probe failure on all backends, rule change, backend capacity exhausted, listener queue full.

First checks:

• Check backend health: are all instances now unhealthy? If so, the load balancer stops forwarding new flows, but existing flows may persist until idle timeout.
• Verify that the load‑balancing rule still exists and that the frontend IP configuration hasn’t changed.
• Examine backend metrics: is the application’s connection limit or accept queue limit reached? The OS may refuse new connections while existing ones are fine.
• Check if a deployment removed the last healthy instances, leaving only draining instances that fail probes.

Evidence interpretation:

• All backends unhealthy → troubleshoot as Symptom 3.
• Frontend IP or rule missing → restore it.
• Backend connection limits hit → scale out the backend pool.

Safe remediation:

• If all backends are unhealthy, focus on restoring health, not on the rule.
• If backend saturated, scale out or increase connection limits.

Validation: After remediation, attempt a new connection; it should succeed.

8. Existing connections drop unexpectedly

Symptom: Long‑lived TCP connections terminate before they are expected to, causing application errors.

Most likely fault domains: Backend restart, VM scale set replacement, health probe transitions, idle timeout, network path interruption.

First checks:

• Check the backend VM’s system logs: was the process restarted, or did the VM reboot?
• Check the VM scale set activity log: were instances replaced (e.g., due to auto‑healing or a deployment)?
• Review the load‑balancing rule’s idle timeout setting (default 4 minutes). If connections sit idle longer than that, they will be reset. The application should use TCP keep‑alives.
• Look for health probe transitions: if a backend was briefly marked unhealthy and then healthy, existing connections to it might have been reset if the backend process restarted.

Evidence interpretation:

• Backend restart/scale set replacement → expected; connections to that instance are lost. Clients must reconnect.
• Idle timeout reached → expected; enable keep‑alives or increase the timeout.
• Network path issue (e.g., ExpressRoute flap) → investigate network infrastructure.

Safe remediation:

• Configure the application to use TCP keep‑alives for long‑lived connections.
• If appropriate for the workload, increase the idle timeout (up to 30 minutes for TCP).
• Design clients to reconnect with exponential back‑off.

Validation: Monitor connection duration and reset counts after changes.

9. Intermittent connectivity

Symptom: Clients sometimes succeed and sometimes fail, without an obvious pattern.

Most likely fault domains: Backend health flapping, resource saturation, DNS caching, NSG rule priority conflicts, client network issues.

First checks:

• Determine if failures correlate with a specific backend instance. Collect logs from clients and backends with timestamps.
• Check backend health history: has any instance been flapping between healthy and unhealthy? This indicates a borderline probe or an overloaded instance.
• Examine backend CPU, memory, and connection counts. Saturation can cause sporadic probe failures or slow responses.
• Look at DNS: is the client resolving the frontend to different IPs (if using multiple frontends or a global service)?
• Verify NSG rules: a Deny rule with lower priority than an Allow might intermittently block traffic if the source IP changes.

Evidence interpretation:

• Health probe flapping → adjust probe threshold or fix the underlying performance issue.
• Backend saturation → scale out or optimize.
• DNS changing → clients might be connecting to an old IP; reduce TTL or ensure all frontends are healthy.

Safe remediation:

• Stabilise flapping backends before changing probe settings.
• Scale out under‑provisioned backends.

Validation: Monitor error rates over a period that previously exhibited the intermittent failures.

10. Connection reset or connection refused

Symptom: Clients receive TCP RST or Connection Refused errors.

Most likely fault domains: No listener on the destination port, NSG with a Deny rule that sends a reset, backend process not running, or a firewall actively rejecting traffic.

First checks:

• Connection refused (ECONNREFUSED) typically means a TCP RST was sent in response to a SYN, indicating that no process is listening on the port, or a firewall rule explicitly rejects the traffic.
• Connection reset (ECONNRESET) after a connection was established suggests the backend process terminated or a middlebox dropped the state.
• Test from a client in the same network directly to the backend’s private IP on the backend port. If you get the same error, the problem is at the host/application level.
• Check NSG effective rules; a Deny rule might be configured to send a reset.

Evidence interpretation:

• Direct connection refused → no listener; start the application or correct the binding.
• Connection reset after data exchange → application crash or an idle timeout on an intermediate device.

Safe remediation:

• Start the application or fix the binding.
• If an NSG Deny rule is responsible, review whether it is intentional. If not, update it.

Validation: A successful TCP handshake confirms the listener and network path are correct.

11. UDP traffic fails or behaves inconsistently

Symptom: UDP‑based applications do not receive datagrams, or responses never reach the client.

Most likely fault domains: Protocol mismatch in rule, missing listener, stateful firewalls dropping “unmatched” return traffic, application‑level assumptions.

First checks:

• Confirm the load‑balancing rule is configured for UDP, not TCP.
• On the backend, verify the application is listening for UDP on the correct port using ss -ulnp (Linux) or netstat -ano (Windows).
• Test UDP connectivity with a tool like nc -u or a dedicated UDP test client. Send a datagram and confirm the server logs receipt.
• Because UDP is connectionless, NSGs and firewalls must be configured to allow bidirectional UDP traffic. Some firewalls drop return UDP traffic if they cannot associate it with an outbound flow. Ensure the NSG allows UDP from the backend to the client on the appropriate ephemeral ports.
• Check that the client’s source port remains consistent; NAT devices may rewrite it, breaking application assumptions.

Evidence interpretation:

• No datagrams received at backend → rule incorrect, NSG blocking, or application not listening.
• Datagrams received but no response → return path blocked. Check NSG outbound rules and any stateful firewall.

Safe remediation:

• Correct the rule’s protocol.
• Adjust NSGs to allow the required bidirectional UDP flows.
• For complex UDP protocols, consider using a network virtual appliance or Application Gateway (Layer 7) if HTTP‑based alternatives exist.

Validation: Send a UDP datagram and receive a response; use packet captures to confirm bidirectional flow.

12. Inbound NAT rule or administrative access fails

Symptom: Cannot SSH or RDP to a specific backend VM using the configured inbound NAT rule.

Most likely fault domains: NAT rule misconfiguration, NSG, host firewall, service not running, source IP restriction.

First checks:

• Verify the NAT rule maps the correct frontend port to the correct backend VM’s private IP and management port.
• From the client’s public IP, test TCP connectivity to the frontend IP:port.
• Check the backend VM: is the SSH/RDP service running? Is the host firewall allowing the port from the client’s IP?
• Review the NSG on the backend subnet or NIC; it must allow inbound traffic from the client’s public IP on the management port.
• Ensure the NAT rule is not competing with a load‑balancing rule on the same frontend port.

Evidence interpretation:

• Connection timeout → traffic blocked by NSG, host firewall, or the service not running.
• Connection refused → service not running or host firewall rejecting.
• Wrong backend reached → NAT rule points to the wrong instance.

Safe remediation:

• Correct the NAT rule mapping.
• Enable and verify the SSH/RDP service.
• Adjust NSGs to allow only the required source IP ranges—never open to 0.0.0.0/0.
• Consider replacing the NAT rule with Azure Bastion for a more secure and auditable access method.

Validation: Connect via the NAT rule, then immediately review access logs and consider whether the rule should remain or be removed.

13. Outbound internet connectivity fails

Symptom: Backend instances cannot reach external services (e.g., APIs, package repositories).

Most likely fault domains: No outbound connectivity configured, SNAT exhaustion, NAT Gateway misconfiguration, NSG outbound rules.

First checks:

• Does the backend subnet have an explicit outbound solution? An internal load balancer provides no internet access. A public load balancer can provide SNAT via outbound rules, but this must be configured.
• If using Azure NAT Gateway, verify it is associated with the backend subnet and that it has a public IP.
• If using load balancer outbound rules, check the SNAT port allocation metric for exhaustion.
• Test outbound connectivity from the backend VM: curl ifconfig.me to see the outbound IP, then try connecting to the target external endpoint. Use tcpping or psping for port‑specific tests.
• Check effective routes: ensure a route to the internet exists, and no UDR forces traffic through a firewall that blocks it.
• Review NSG outbound rules; they must allow the destination IP/port.
• Inspect the external service’s firewall: does it allow the outbound IP of your backend? If the IP changed (e.g., after moving to NAT Gateway), the service may be blocking it.

Evidence interpretation:

• No route to internet → add a default route through the appropriate egress (NAT Gateway, Azure Firewall, etc.).
• SNAT port exhaustion → switch to NAT Gateway for larger port allocation, or increase the number of public IPs on the load balancer’s outbound rule.
• Connection refused by external service → the new source IP might not be allowlisted. Update the external service’s firewall rules.

Safe remediation:

• Implement NAT Gateway if outbound connectivity is critical and not yet configured.
• Update external allowlists with the new outbound IP.
• Do not disable NSGs; instead, add the specific outbound allow rule.

Validation: From a backend VM, successfully connect to the required external endpoint and verify the application can perform its external call.

14. Frontend works but backend cannot reach dependencies

Symptom: Clients can reach the application through the load balancer, but the application itself fails when calling its database, cache, or other internal services.

Most likely fault domains: Different inbound/outbound paths, private endpoint misconfiguration, NSG asymmetry, DNS for dependencies.

First checks:

• The inbound path (client → frontend → backend) works, so the backend can receive traffic. The problem is its outbound or east‑west path.
• Verify the backend can resolve the dependency’s hostname. Check Private DNS or custom DNS configuration.
• Test connectivity from the backend VM to the dependency on its expected port (e.g., TCP 1433 for SQL Database).
• Check NSG outbound rules on the backend subnet; they must allow traffic to the dependency.
• If the dependency is an Azure PaaS service accessed via Private Endpoint, verify the private endpoint is in the same VNet (or a peered VNet) and that the DNS resolves to the private IP.
• If the dependency is on‑premises, check VPN/ExpressRoute connectivity.

Evidence interpretation:

• DNS resolves to a public IP but the service is configured for private access only → use the private endpoint’s private IP or configure service endpoints.
• Connection timeout to dependency → NSG or route table blocking. Check effective routes.
• Connection refused → dependency not listening, or private endpoint not approved.

Safe remediation:

• Correct DNS to resolve to the private endpoint’s IP.
• Add missing NSG outbound rules.
• Approve the private endpoint connection.

Validation: From the backend, run a query or perform a dependency action that previously failed.

15. Performance degradation or high latency

Symptom: Application response times increase, but connections are not dropped.

Most likely fault domains: Backend saturation, cross‑zone traffic, slow dependencies, TCP keep‑alive issues, load balancer data path (rare).

First checks:

• Monitor backend CPU, memory, disk I/O, and network throughput. High utilization explains latency.
• Check per‑backend connection counts. One overloaded backend can skew latency.
• Determine if the traffic is crossing availability zones unnecessarily. Intra‑zone traffic is free and low latency; cross‑zone traffic adds a small cost and slightly higher latency.
• Profile the application: is it spending time in a slow dependency, a database query, or serialization?
• Check the load balancer’s metrics: data path availability should be high. Latency introduced by the load balancer itself is minimal (sub‑millisecond), but packet loss on the platform can cause TCP retransmissions.

Evidence interpretation:

• Backend saturated → scale out or optimize code.
• Cross‑zone traffic causing latency → adjust placement so clients and backends are in the same zone, or use a zone‑redundant frontend with zone‑pinned backends if the application is zone‑aware.
• Application‑level latency → fix the slow code or query.

Safe remediation:

• Scale out the backend pool.
• Optimize application code and dependency calls.
• If cross‑zone latency is measurable and problematic, consider a zonal architecture.

Validation: Measure 95th percentile latency before and after changes.

16. Deployment or scale operation causes failures

Symptom: After a deployment, scaling event, or scale set model update, connectivity issues appear.

Most likely fault domains: New instances not ready, health probe thresholds too aggressive, incomplete rollout, configuration drift.

First checks:

• Check the deployment timeline against incident start. Were new VMs created? Was the scale set model updated? Was a new image rolled out?
• Inspect the health probe status of new instances. They may be initialising and failing probes, causing temporary capacity loss.
• Compare the new instance configuration (OS, application version, environment variables) with the previous version.
• Review NSG changes: a deployment template might have overwritten a custom NSG rule.
• Check if the load‑balancing rule or probe was accidentally modified in the deployment.

Evidence interpretation:

• New instances unhealthy → increase probe threshold or fix application startup time.
• NSG or probe changed → revert the change in the deployment template.
• All new instances healthy but traffic fails → application incompatibility; roll back the deployment.

Safe remediation:

• Roll back the deployment if it introduced the failure.
• If the issue is startup time, extend the probe’s unhealthy threshold temporarily while a permanent fix is developed.
• Ensure IaC deployments use incremental mode or lifecycle policies to prevent accidental overwrites.

Validation: Deploy a new instance and verify it passes probes and handles traffic before scaling to full capacity.

Foundational diagnostic procedures

These procedures support the symptom sections above. They are ordered from simplest to most detailed.

Verify DNS and frontend identity

From a client that experiences the issue, resolve the hostname to an IP address using nslookup or dig. Confirm the result matches the expected frontend IP (public or private). If the hostname resolves to a different IP, investigate the DNS zone (Azure DNS, Private DNS, or external provider). For internal frontends, ensure the client is using a DNS server that can resolve the private zone.

What this proves: That the client is attempting to reach the correct IP. It does not prove anything about reachability.

Test the exact protocol and port

Use a tool that matches the application protocol. For TCP, nc -vz <ip> <port> or Test-NetConnection <ip> -Port <port>. For UDP, use nc -u. For HTTP/HTTPS, use curl -v https://<ip>:<port>/<path>.

What this proves: That a connection can be established at the transport layer, or that the application responds. It does not prove the load‑balancing rule is correct if you target a backend directly.

Inspect backend health

Review the backend pool’s health status in the Azure portal, Azure CLI (az network lb probe show or az network lb address-pool show depending on context), or via Azure Resource Graph. Note the health state of each instance and the timestamp of the last transition.

What this proves: Whether the platform considers the instance eligible for new flows. It does not indicate that the application is fully functional.

Verify the backend listener

Log on to a backend VM (via Bastion or secure channel). Run ss -tlnp (Linux) or netstat -ano (Windows) to see which processes are listening on which addresses and ports. Ensure the port matches the load‑balancing rule’s backend port and that the address is 0.0.0.0 (or the specific interface) so that traffic from the load balancer is accepted.

What this proves: That a process is listening. It does not prove that the process is ready to serve traffic.

Review NSGs and effective security rules

Use the Azure portal “Effective security rules” blade for the backend NIC or subnet. Alternatively, use Network Watcher’s “Effective security rules” or the Azure CLI. Look for an Allow rule that matches the source (client IP range or AzureLoadBalancer for probes), destination (backend port), and protocol. Confirm no higher‑priority Deny rule overrides it.

What this proves: Whether Azure’s network policy permits or blocks the traffic. It does not account for host firewalls or routes.

Review routes and effective routes

On the backend NIC/subnet, check “Effective routes.” For inbound traffic, ensure no UDR forces the response traffic through a firewall that might drop it. For outbound connectivity, ensure a route to the internet (0.0.0.0/0) exists via NAT Gateway, Azure Firewall, or the internet.

What this proves: The next‑hop for traffic leaving the subnet. Incorrect routes can cause asymmetric paths.

Use Network Watcher and Azure Monitor

• Connection troubleshoot (Network Watcher): Test connectivity from a source VM to a destination IP:port. It checks NSGs, routes, and virtual network gateways. A failure message indicates which component is blocking.
• IP flow verify: Checks whether a packet is allowed or denied based on NSGs at the NIC level.
• Next hop: Shows the next hop type and route for a given destination IP.
• Load balancer metrics: Monitor DipAvailability (data path availability) and VipAvailability (frontend availability). If these drop, open a support case.
• Activity Log: Review any recent operations on the load balancer, public IP, NSG, or scale set.

Compare backend and client captures

When a problem remains unexplained, simultaneous packet captures on a backend VM and on a test client can reveal where packets are lost or altered. Use tcpdump (Linux) or netsh trace/pktmon (Windows). Analyze with Wireshark or tcpdump. Look for SYNs reaching the backend and whether the SYN‑ACK is sent back. Verify the source IP in the SYN packet is the client’s IP (meaning the load balancer preserved it).

Important: Packet captures may contain sensitive data. Follow your organisation’s security policies.

Health probe troubleshooting matrix

Probe symptom	Likely cause	Evidence to collect	Corrective action
Probe timeout	NSG blocking AzureLoadBalancer tag, host firewall, no listener, route issue	Effective NSG rules, host firewall status, ss -tlnp, packet capture	Add NSG allow rule for AzureLoadBalancer on probe port; fix listener; open host firewall
Probe connection refused	Listener not running, service bound to localhost, host firewall rejecting	ss -tlnp, application logs, host firewall rules	Start application, bind to 0.0.0.0, adjust firewall
HTTP probe returns non‑200	Health endpoint misconfigured, application not ready, dependency check fails	Application logs, test endpoint with curl locally	Fix health endpoint to return 200 when ready; address application readiness logic
HTTPS probe handshake failure	TLS version mismatch, certificate error (though probe doesn't validate), wrong port	openssl s_client, application logs	Ensure the backend supports the TLS version used; use HTTP if TLS is not needed for probes
Only one backend fails	Instance‑specific config, different image, resource constraint	Compare instance configs, check CPU/memory, probe logs	Bring instance into same configuration as healthy ones; address resource issue
All backends fail after deployment	New version not ready, health endpoint changed, probe interval too short	Deployment logs, compare old/new probe response times	Increase unhealthy threshold temporarily; roll back if necessary; ensure probe matches new version
Probe flapping	Overloaded backend, network congestion, threshold too tight	Backend resource metrics, probe logs, network latency	Scale out, adjust threshold to be slightly more forgiving, fix performance bottleneck
Probe succeeds but real traffic fails	Probe too shallow, backend port mismatch, application logic error	Application logs, test with real request to backend private IP	Deepen probe appropriately in staging first; correct rule’s backend port; fix application bug

Configuration and change correlation

Most production incidents follow a change. Use Azure Activity Log to identify what changed around the incident start time. Filter by resource group or resource type (Microsoft.Network/loadBalancers, Microsoft.Network/networkSecurityGroups, Microsoft.Compute/virtualMachineScaleSets). Look for modifications to:

• Load balancer rules, probes, frontend IPs.
• NSG rules.
• Route tables.
• Scale set model updates.
• DNS zone records.
• Certificate updates.

Correlate these changes with Infrastructure as Code commits and deployment pipeline runs. If a change was made outside IaC (portal or CLI), that is a red flag—restore the desired state from code and investigate how the drift occurred.

Causation, not correlation: A change occurring at the same time as an incident is not automatically the root cause. Validate by reverting the change in a controlled manner (e.g., rollback deployment, reapply previous NSG rule) and observing whether the symptom disappears.

Safe remediation principles

In a production incident, the goal is to restore service with minimal risk of making the situation worse.

• Make one change at a time. If you change the health probe, the NSG, and the rule simultaneously, you won’t know which one fixed the issue, and you might introduce a new problem.
• Prefer reversible changes. Save the current configuration before modifying. Use deployment slots, canary instances, or separate test pools.
• Never disable NSGs or firewalls globally to test connectivity. Instead, add a temporary, narrowly scoped allow rule and remove it after testing.
• Do not expose administrative ports publicly as a troubleshooting shortcut. Use Azure Bastion or existing private management channels.
• When removing a backend instance, drain it first: take it out of the backend pool or make it fail its health probe, wait for existing connections to drain, then deallocate.
• Validate a fix on a single backend (if the incident permits) before applying it to all.
• After recovery, document the root cause and update runbooks and monitoring rules.

Recovery validation checklist

After remediation, verify the following, not just a single successful TCP handshake:

• DNS resolution returns the correct frontend IP.
• A new TCP/UDP connection to the frontend succeeds.
• The application returns the expected response (200 OK for a health page, correct business data).
• New connections are distributed across multiple healthy backends (check per‑backend metrics).
• Existing connections remain stable (if applicable).
• Outbound dependencies are reachable from backends.
• Error rates and latency return to baseline.
• Alerts (e.g., health probe status) return to green.
• No regression for unaffected services.

Monitor for at least 15 minutes after recovery; intermittent failures can return.

Root cause analysis

Post‑incident, a written analysis should include:

• Customer impact (what users experienced, duration, severity).
• Timeline of events, including detection, diagnosis, mitigation, and recovery.
• Root cause: the specific configuration, code defect, or operation that started the failure chain.
• Contributing factors: why monitoring didn’t catch it earlier, why the health probe didn’t prevent it, why the deployment process allowed it.
• Permanent corrective actions: changes to infrastructure, code, deployment process, monitoring, or runbooks.
• Owner and due date for each action.

A root cause is rarely “the load balancer failed.” It is usually “the health probe was too shallow, so it did not detect that the application could not reach its database after a firewall rule change.”

Common troubleshooting anti‑patterns

• Testing only from the engineer’s workstation. The client’s network path may differ. Use a test VM in the same VNet or a known‑good client.
• Using ICMP ping to validate TCP/UDP. Ping tests ICMP, which may be blocked independently of TCP. Use protocol‑specific tools.
• Checking VM health instead of backend health. A running VM can be unhealthy because the application isn’t listening or the probe is failing.
• Assuming a successful health probe means the application works. Probe endpoints often only check liveness. Always test with real application traffic.
• Changing multiple network/load balancer settings at once. This makes it impossible to isolate the fix and can cause new issues.
• Disabling NSGs or host firewalls as a first step. This eliminates a critical security layer and can cause more harm if other rules are missing.
• Blaming the load balancer for any timeout. Timeouts can come from firewalls, application backlogs, or client networks.
• Ignoring return‑path routing. Asymmetric routes cause TCP resets and silent failures.
• Ignoring DNS caching. A stale DNS record on the client can send traffic to an old IP.
• Treating UDP like TCP. UDP lacks handshake and state; tools and expectations must differ.
• Relying on a single successful test. Intermittent failures require monitoring over time.
• Omitting timestamps in evidence. Correlating logs is impossible without accurate timestamps.

Representative incident walkthrough

Scenario: An internal API behind an internal Azure Load Balancer becomes intermittently unavailable after a routine application deployment. The API serves HTTP traffic on TCP 8443.

Initial symptom: Clients report HTTP 503 errors; sometimes requests succeed.

Scope classification: The API is internal, using a private frontend IP 10.20.1.50. New connections are failing intermittently; existing connections are fine initially. The issue started immediately after a scale‑set model update that deployed a new application version.

Evidence collection:

• Health probe status in Azure portal: 2 of 3 backends healthy, one flapping.
• The flapping instance’s probe configuration: HTTP probe on port 8080 path /ready.
• Test /ready on the unhealthy instance from a management VM: returns 200 after a 5‑second delay. On the healthy instances, returns 200 immediately.
• Application logs on the flapping instance show warnings: “Initialising connection pool…” after each probe request. The new code version introduced a synchronous database health check inside the /ready endpoint that takes several seconds.
• The probe’s unhealthy threshold is 2 failures (consecutive). The probe interval is 5 seconds. The slow /ready endpoint times out the probe, causing it to flap.

Hypothesis testing:

• The new deep health probe is causing cascading failure detection on that instance. When the database momentarily slows, the endpoint takes >5 seconds, probe fails, instance marked unhealthy, then recovers → flap.
• Why only one instance? That instance might have a slightly different database connection path or started after a database connection pool reset.

Root cause: The deployment changed the health endpoint to include a deep dependency check that is too slow, violating the “fast and deterministic” probe principle.

Safe remediation:

1. Temporarily increase the probe’s unhealthy threshold to 5 on the flapping instance (or the whole pool) to buy time.
2. Roll back the application deployment to the previous version that had a shallow probe endpoint.
3. In the development environment, redesign the /ready endpoint to only check local readiness (e.g., server loop running, configuration loaded). Move database connectivity monitoring to a separate endpoint or background health check that does not block the probe.

Recovery validation:

• All backends stable and healthy.
• No more 503 errors.
• Latency back to baseline.
• After hours, the application team deploys a fix with a correct probe.

Preventive changes:

• Update deployment acceptance tests to include probe response time under 1 second.
• Add a monitoring alert for probe response time.
• Document the probe design principle in the team’s wiki.

Explore further

• Azure Load Balancer – Topic hub with high‑level orientation and decision guidance.
• What Is Azure Load Balancer? Architecture, Components, and Use Cases – Foundational concepts and component responsibilities.
• Azure Load Balancer Architecture Guide – Detailed traffic flow, configuration analysis, and design trade‑offs.
• Azure Load Balancer Best Practices for Production Workloads – Practices that prevent the incidents described in this guide.

---

This troubleshooting guide provides a systematic, evidence‑driven approach to diagnosing and resolving Azure Load Balancer issues. By classifying the scope and direction of failures, collecting targeted evidence, and forming hypotheses before acting, you can restore service quickly and prevent recurrence. Remember that the load balancer is just one link in the chain; always validate the full path from client to application and back.