Fixing the "Cannot connect to backend server" error in Azure Application Gateway health checks

When an Application Gateway reports backend health check failures, a frequent but easy-to-miss cause is a TLS negotiation mismatch between the gateway and the backend. This article shows how to diagnose the problem, align TLS settings and probes, validate the fix, and codify the correct configuration in IaC.

Identifying the Issue

1. In the Azure Portal, navigate to your Application Gateway (you can find it under Load balancing services)
2. Go to Monitoring and click on Backend health

3. Look for any unhealthy backend pools with the error message:

   Cannot connect to backend server. Check whether any NSG/UDR/Firewall is blocking access to the server. Check if application is running on correct port.
   

Common Root Cause: TLS Version Mismatch

While the error message suggests checking NSG rules and ports, a common cause is a TLS version mismatch between your Application Gateway and the backend service, which in our case is an Azure Function.

Step-by-Step Resolution

1. Check the backend’s minimum TLS

For an Azure Function:

1. Navigate to your Function App
2. Go to Settings → Configuration
3. Look for Minimum Inbound TLS Version
4. If it’s TLS 1.3, the backend will reject TLS 1.0/1.1/1.2

2. Verify Application Gateway SSL Policy

1. In your Application Gateway, go to Settings → Listeners
2. Note the Selected SSL Policy
3. Check the policy’s supported TLS versions in the Microsoft documentation. Predefined policies like AppGwSslPolicy20220101 are recommended for security and compatibility.

3. Align TLS Versions

The gateway must allow the minimum TLS version enforced by the backend. For exaample:

• If backend enforces TLS 1.3
• The Application Gateway must permit TLS 1.3 or higher

Apart from that, ensure ciphers allowed by the gateway include at least one cipher suite the backend accepts.

4. Update SSL Policy

Choose a predefined policy that matches your security requirements. Microsoft’s recommended policy (AppGwSslPolicy20220101 at the time of writing) usually provides the best balance of security and compatibility.

We first see how to update the policy by using CLI commands:

az network application-gateway update \
  --name <application-gateway-name> \
  --resource-group <rresource-group-name> \
  --set sslPolicy.policyType=Predefined sslPolicy.policyName=AppGwSslPolicy20220101

And this is the Infrastructure as Code Solution by using Bicep:

resource applicationGateway 'Microsoft.Network/applicationGateways@2021-05-01' = {
  // ... other properties
  properties: {
    // ... other properties
    sslPolicy: {
      policyType: 'Predefined'
      policyName: 'AppGwSslPolicy20220101' // Current recommended policy
    }
  }
}

Best Practices

1. Always use the latest recommended SSL policy unless you have specific compatibility requirements
2. Document any custom TLS version requirements in your infrastructure code
3. Regularly review and update your TLS configurations to maintain security
4. Consider using Azure Policy to enforce minimum TLS versions across your resources