Verifique a diretiva UseCanonicalName no httpd.conf
#
# UseCanonicalName: Determines how Apache constructs self-referencing
# URLs and the SERVER_NAME and SERVER_PORT variables.
# When set "Off", Apache will use the Hostname and Port supplied
# by the client. When set "On", Apache will use the value of the
# ServerName directive.
#
UseCanonicalName Off
Você também pode removê-lo no verniz, tente isto:
sub vcl_fetch {
if (beresp.status == 301 || beresp.status == 302)
{
set beresp.http.Location = regsub(beresp.http.Location, "^(\w+://[^/]+):\d+", "");
}
}