Timeout for asynchronous SslStream methods AuthenticateAsClient takes too long

I'm developing a .NET service which tries to establish a secure connection to a server. The service does not know in advance if the server supports secure connections. That's why I simply try to establish a secure connection and if that fails I'll fallback to the unsecure connection.

I'm using TcpClient together with SslStream:

tcpClient.BeginConnect(hostname, port, Start, null);
private void Start(IAsyncResult asyncResult)
{
  X509CertificateCollection certCollection = new X509CertificateCollection();
  certCollection.Add(new X509Certificate2("cert.p12", "pw"));

  SslStream sslStream = new SslStream(tcpClient.GetStream(), false, new RemoteCertificateValidationCallback(ValidateCertificate));

  sslStream.ReadTimeout = 2000;

  sslStream.BeginAuthenticateAsClient("name", certCollection, SslProtocols.Tls, false, AuthenticateCallback, sslStream);
}

private void AuthenticateCallback(IAsyncResult asyncResult)
{
stream.ReadTimeout = -1;

// secure connection has been established
}

In the case that the server does not support the requested encryption it takes 2 minutes before the callback method is being called. Using the synchronous method AuthenticateAsClient() instead takes only the expected 2 seconds (as requested by setting the ReadTimeout):

sslStream.AuthenticateAsClient("ESLD Server", certCollection, SslProtocols.Tls, false);

Why does the timeout only apply to the synchronous method? AuthenticateAsClientAsync() takes 2 minutes as well. How can I reduce the callback time for the asynchronous method? Or is there a better approach for checking if the server supports secure connections?