Manually checking HTTPS certificates on Android

Introduction

The basic question is: should we hang the people who designed TLS with the guts of the people who designed Android, or would the other way around be better?

Note: the previous sentence is a joke; I do not condone the death penalty, even for bad programming.
Note 2: the end of the previous note is also a joke, I do not consider bad programming to be worse than most crimes usually punished by the death penalty, historically or in the barbaric countries that still have it.
Note 3: “barbaric” in the previous note is not a joke.

Imagine you want to write a client-server application, with the client running on a smartphone or tablet or something like that. You want it to be secure, of course, so you must use cryptography to protect the network communications.

Security experts advise not to roll your own cryptography, because you would get it wrong; they advise to use standard protocols and well-tested implementations. Well, experts' livelihood is to make people believe their field is more complicated than it actually is, but in this particular instance, they are mostly right: there are so many subtle pitfalls in cryptography that can make a protocol completely insecure that most developers would get it wrong.

Anyway, some of your users would probably have to deal with stupid ISPs or BOFHs that think Internet is only the Web and put a firewall on everything else. And maybe you also want to use free or cheap web hosting for the server.

So you will go with standard HTTPS, using the built-in implementation of Android. And now the nightmare begins because the server's certificate is not trusted.

The difficult problem that the certificates try to address

Assuming the bad guys do not have an alien supercomputer in their attic, a theoretical breakthrough unknown to the public or a backdoor in the protocol or the implementation, modern cryptography correctly used can guarantee you that nobody can eavesdrop or tamper your communications, except...

... Except if they can intercept your communications from the very beginning. If they can, they can impersonate your peer completely, and then they can use the knowledge gained to impersonate you speaking to your peer. This is called man-in-the-middle.

Cryptography can still help with that, using cryptographic signatures. In the physical world, if you need to check someone's signature, you need a model of it, and the same model would allow you to forge the signature. In the digital world, it works much better, because the know-how behind a cryptographic signature has two components: a piece of data called “private key” that is needed to generate the signature and a piece of data called “public key” that allows to verify the signature but can not be used to forge it.

If you know your peer's public key, or have reliable means of getting it, then you can exclude the man-in-the-middle: your peer has to sign the cryptographic parameters used on the communication channel; if the signature matches, nobody can be listening or tampering.

You can use an trusted intermediary to obtain your peer's public key reliably. You then only need to have the intermediary's key. The intermediaries themselves can use trusted intermediaries to reach the peer, building a chain or web of trust between you and your peer.

Of course, the longer the chain, the weaker the security.

Certificates are just that: a signature, made by a well-known entity, of a less-known entity's identity and public key. This signature establishes one link in the web of trust. The certificate can be distributed by any means: by the intermediary, by the peer, by public repositories, etc.

But for this web of trust to work, you need the initial link(s), the link(s) going from you to the rest of the world. If you want to check your peer's identity, you need to know someone important's public key; if you want your peer to check your identity, you need a certificate from someone important.

Why the certificate model of HTTPS is broken

In the certificate model used by TLS, the cryptographic protocol used by HTTPS, the servers present at connect time a chain of certificates from the server key to a certification authority. If the certification authority is known to the client application, the connection is considered secure.

With HTTPS, the keys of the certification authorities are stored in the users' web browser. The users can add (and remove) certification authorities, but let us be realistic, they never do: the accepted certification authorities are the ones that are bundled with the major web browsers.

The browser distributors have quite a few requirements before accepting a certification authority in their bundle, but these requirements are mostly about corporate solidity. In terms of actual security, this is mostly ridiculous.

The certificates certify that the public key was really issued by the legitimate owner of the DNS domain name. Nothing more.

They do not certify that the site is honest. I could register i-will-rob-you-blind.com and get a certificate for it.

They do not certify that the site's goods or services are good.

They do not certify that the site is secure. All the visitors' passwords and the site's private key could be laying around for any intern or external contractor to grab, or worse.

The keystone of the problem is that the chain of certificates is unique and chosen by the server. This is ridiculous because the certification is addressed to the client, not the server. If the clients were to choose the certification chain, they could, for example, choose one that comes with insurance against scams.

Actually, no, this is not ridiculous, this is a racket: if you want the green lock that will make your visitors feel secure, get your credit card. To be convinced of that, just look how alarming and complex the messages for unknown certification authorities are compared to the messages (or the absence of messages) for unencrypted connections. Remember that encrypted connections without certification are still much more secure than unencrypted, a man-in-the-middle attack being much more difficult to achieve than just eavesdropping.

Alternate solutions to the problem

There are a lot of situations where you have means of getting your public key to your users without relying on a third-party certification authority.

If you are a bank, then you must have some secure way of getting your clients their credit cards. Just give them the public key's fingerprint at the same time.

The same kind of solution can apply whenever you have another means of contacting your users that is considered more secure, or at least more authoritative. Maybe your users must sign a contract on paper, or maybe they need to get their badge from the security desk or their initial password in person from the sysadmin, etc.

Any of these situations can be used as the initial link for the web of trust, but nobody uses them because the user interfaces are designed to make it hard to stray from the official model.

The actual Android implementation

If you want to use another model with Android's built-in HTTP implementation, you will have a lot of trouble to get it to accept another model. Actually, you will have a lot of trouble even getting it to accept certification authorities not bundled with the device.

You can find on the web a lot of ridiculously complex code snippets showing how to disable certificate verification entirely. Most of them do not work. And even those that work are accompanied by comments promising you every catastrophe short of the actual apocalypse if you use them.

You can find official recommendations on how to add your own certificate to your application. The code snippets are still ridiculously complex. And anyway, you would not require your users to copy half a page of base64-encoded certificate from the sheet of paper where their password is printed.

Here is a code snippet that allows you to implement custom certificate verification.

It does not work as is. As is, it only prints the certificate's fingerprint to the internal logs and accepts it. You need to add code to get the certificate or the fingerprint from the callback to your user interface and get the user to confirm it somehow.

You can consider this piece of code to be in the Public Domain, or if does not work, the 3-clause BSD license. There are, of course, no guarantees.

HttpClient get_new_http_client() {
    try {
        KeyStore trust_store = KeyStore.getInstance(KeyStore.getDefaultType());
        trust_store.load(null, null);

        SSLSocketFactory ssl_sock_factory = new SSLSocketFactory(trust_store) {

            SSLContext ssl_context = SSLContext.getInstance("TLS");
            {
                TrustManager[] trust_manager = new TrustManager[1];

                trust_manager[0] = new X509TrustManager() {

                    public void checkClientTrusted(X509Certificate[] chain, String authType) 
                        throws CertificateException {
                    }

                    public void checkServerTrusted(X509Certificate[] chain, String authType) 
                        throws CertificateException {
                        MessageDigest md = null;
                        try {
                            md = MessageDigest.getInstance("SHA-1");
                        } catch (Exception e) { /* will raise null */ }
                        md.update(chain[0].getEncoded());
                        byte[] digest = md.digest();
                        StringBuffer buf = new StringBuffer(digest.length * 3);
                        for (int i = 0; i < digest.length; i++)
                            buf.append(String.format("%02x:", digest[i]));
                        buf.delete(digest.length * 3 - 1, digest.length * 3);
                        String r = buf.toString();
                        warn("fpr = %s for %s", r, chain[0].getSubjectDN().getName());
                    }

                    public X509Certificate[] getAcceptedIssuers() {
                        return null;
                    }
                };

                ssl_context.init(null, trust_manager, null);
            }

            public Socket createSocket(Socket socket, String host, int port, boolean autoClose)
                throws IOException, UnknownHostException {
                return ssl_context.getSocketFactory().createSocket(socket, host, port, autoClose);
            }

            public Socket createSocket() throws IOException {
                return ssl_context.getSocketFactory().createSocket();
            }

        };

        //ssl_sock_factory.setHostnameVerifier(SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
        HttpParams params = new BasicHttpParams();
        HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
        HttpProtocolParams.setContentCharset(params, HTTP.UTF_8);
        SchemeRegistry registry = new SchemeRegistry();
        //registry.register(new Scheme("http", PlainSocketFactory.getSocketFactory(), 80));
        registry.register(new Scheme("https", ssl_sock_factory, 443));
        ClientConnectionManager ccm = new ThreadSafeClientConnManager(params, registry);
        return new DefaultHttpClient(ccm, params);

    } catch(Exception e) {
        warn("Exception creating client: ", e);
        e.printStackTrace();
    }
    return null;
}