Ellipsoidal Geodesic vs Spherical Haversine Distance Variance
Six Benchmark Routes from the Equator to 78° N on the WGS84 Ellipsoid
Many web distance calculators rely on the spherical Haversine formula, assuming Earth is a perfect ball with a uniform radius of 6,371 km. This study quantifies the systematic error introduced by spherical approximations compared to Charles Karney's WGS84 ellipsoidal geodesic algorithm (geographiclib), using six long-haul city pairs from the Equator to the high Arctic.
1. Mathematical Origin of the Divergence
Earth is an oblate spheroid flattened at the poles due to centrifugal rotation. Under the WGS84 datum (EPSG:7030):
- Semi-major axis (Equatorial radius a): 6,378,137.0 meters
- Semi-minor axis (Polar radius b): 6,356,752.3142 meters
- Difference a − b: 21,384.7 meters (13.29 miles)
- Flattening (f): 1 / 298.257223563
When a software tool applies the Haversine formula using the standard IUGG mean radius (R1 = 6,371,008.8 meters), it uses one radius everywhere. The ellipsoid's east-west radius of curvature at the Equator is 6,378.1 km, and its north-south (meridional) radius of curvature rises from 6,335.4 km at the Equator to 6,399.6 km at the poles. A route therefore comes out short on the sphere wherever the local radius of curvature along its path exceeds 6,371 km, which is the case for every route in this sample.
2. Empirical Latitude Distortion Matrix
| Latitude Band | Sample Flight Leg | WGS84 Ellipsoidal | Spherical Haversine | Delta (Error) | Variance % |
|---|---|---|---|---|---|
| Equatorial (~0°) | Quito, EC to Pontianak, ID (along the Equator) | 19,167.7 km | 19,146.3 km | -21.40 km (-13.30 mi) | -0.112% |
| Sub-Tropics (~25° N) | Miami, FL to Taipei, TW | 13,920.5 km | 13,902.5 km | -18.06 km (-11.22 mi) | -0.130% |
| Mid-Latitudes (~40° N) | New York, NY to Madrid, ES | 5,782.6 km | 5,768.0 km | -14.59 km (-9.07 mi) | -0.252% |
| High Mid-Latitudes (~50° N) | London, UK to Vancouver, CA | 7,601.4 km | 7,578.2 km | -23.23 km (-14.43 mi) | -0.306% |
| Sub-Arctic (~60° N) | Oslo, NO to Anchorage, AK | 6,467.3 km | 6,443.9 km | -23.46 km (-14.58 mi) | -0.363% |
| Polar (~75° N) | Longyearbyen, Svalbard to Utqiagvik (Barrow), AK | 3,397.0 km | 3,382.7 km | -14.26 km (-8.86 mi) | -0.420% |
In this sample the spherical Haversine formula underestimated every route, by -0.11% along the Equator up to -0.42% on the Svalbard to Alaska polar leg. The largest absolute gaps were about 23 km (14.5 miles) on London to Vancouver and Oslo to Anchorage. Haversine error is bounded at roughly 0.5% in either direction, which is fine for estimates but not for surveying, aviation flight planning or legal distance tests; use an ellipsoidal geodesic there. You can reproduce any row with the straight-line distance calculator, which uses the same WGS84 geodesic.