[Planetlab-devel] PyCurl and ssl certificates, comments on changes in PyCurl.py

[Thierry Parmentelat]

Hi guys

This is a quick update on the PyCurl code.
I am looking forward to an audio/chat conferencing on this issue, as 
soon as possible.

So as to prepare this meeting, I tried to make a synthesis of my 
observations. Unfortunately I do not have the full source of the 
production code, but Tony gave me some partial information about it.
Here is what I gathered so far:

* The PyCurl class is used by
** Peers.py in the context that we are now interested in:
 >>>
    def connect(self, **kwds):
        """
        Connect to this peer via XML-RPC.
        """

        import xmlrpclib
        from PLC.PyCurl import PyCurlTransport
        self.server = xmlrpclib.ServerProxy(self['peer_url'],
                                            
PyCurlTransport(self['peer_url'], self['cacert']),
                                            allow_none = 1, **kwds)
<<<

** Shell.py: in the following code
 >>>
            if url is None:
                if self.config is None:
                    raise Exception, "Must specify API URL"

                url = "https://" + self.config.PLC_API_HOST + \
                      ":" + str(self.config.PLC_API_PORT) + \
                      "/" + self.config.PLC_API_PATH + "/"

                if cacert is None:
                    cacert = self.config.PLC_API_CA_SSL_CRT

            self.url = url
            if cacert is not None:
                self.server = xmlrpclib.ServerProxy(url, 
PyCurlTransport(url, cacert), allow_none = 1)
            else:
                self.server = xmlrpclib.ServerProxy(url, allow_none = 1)
<<<

* so a first note here: it very well looks like, at least right now, we 
*always* create objects typed PyCurl.PyCurlTransport with a non-empty cacert

* The changes I am aware of are outlined below

 >>>
        # Set URL
        self.url = uri
        self.curl.setopt(pycurl.URL, str(uri))

+        # Set Verify Peer
+        self.curl.setopt(pycurl.SSL_VERIFYPEER, 0)
+        self.curl.setopt(pycurl.SSL_VERIFYHOST, 1)
+
+        cert = "/etc/pki/tls/certs/ca-bundle.crt"

        # Set certificate path
        if cert is not None:
            if os.path.exists(cert):
                cert_path = str(cert)
            else:
                # Keep a reference so that it does not get deleted
                self.cert = NamedTemporaryFile(prefix = "cert")
                self.cert.write(cert)
                self.cert.flush()
                cert_path = self.cert.name
            self.curl.setopt(pycurl.CAINFO, cert_path)
            self.curl.setopt(pycurl.SSL_VERIFYPEER, 2)
<<<

* so the first thing that looks wrong is that the cert parameter is 
overwritten regardless of the actual value that is provided as 'cert' to 
the class constructor

* another weird thing is about setting VERIFYHOST to 1.
The useful link http://curl.haxx.se/libcurl/c/curl_easy_setopt.html 
gives useful about the various settings and defaults used in the 
library. To make things worse these defaults were revised in the 7.10 
release, but we run based on 7.13.
So the semantics and defaults (I checked most of this) are

** SSL_VERIFYPEER : default=true
two modes are supported:
*** 0 or false means do not check the server's cert,
*** 1 or 2 or true means do check it
http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTSSLVERIFYPEER

** SSL_VERIFYHOST : default=2
three modes are supported:
*** 0 : succeeds regardless of what the CN in the certificate is
*** 1 : only checks that the server certificate does have a CN field
*** 2: checks the CN matches the hostname used for reaching the server
http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTSSLVERIFYHOST

* from what Tony seems to remember, this change could be related to the 
step when Faiyax installed the new SSL certificates a few months back. 
So coming back to overriding cert to a hard-wired constant, I'd propose 
that we try removing this line
+        cert = "/etc/pki/tls/certs/ca-bundle.crt"
because it is clearly very wrong - in any case it prevents the peer code 
from working.
and instead, either mention /etc/pki/tls/certs/ca-bundle.crt as the 
PLC_API_CA_SSL_CRT in the config, or rather copy this bundle into  
/etc/planetlab/api_ca_ssl.crt

* I am unsure about this VERIFYHOST setting.
Probably there is something tricky here, since you run in a much more 
complicated, multi-servers, environment, than I do so there probably are 
issues that I am not aware of. However I wish to point out that setting 
this will relax peer-checkings too and this is maybe not want you want.
(also, if this is really needed it should go in the 'if cert is not 
None' part because it does not make much sense to set it in the other 
case, but that's a detail since as we've seen above, we always have 
not-None certs...)

Here is a method for checking the settings independently (although I 
could not find any command-line option for playing with VERIFYHOST, so 
all this assumes VERIFYHOST=2).
Just run the following kind of things from your myplc chroot jail

<plc> -bash-3.00# curl --cacert /etc/planetlab/api_ca_ssl.crt 
https://localhost:443/
curl: (51) SSL: certificate subject name 'www.planet-lab.eu' does not 
match target host name 'localhost'

<plc> -bash-3.00# curl --cacert /etc/planetlab/api_ca_ssl.crt 
https://planet-lab.eu:443/
curl: (51) SSL: certificate subject name 'www.planet-lab.eu' does not 
match target host name 'planet-lab.eu'

<plc> -bash-3.00# curl --cacert /etc/planetlab/api_ca_ssl.crt 
https://www.planet-lab.eu:443/
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
....


-- Thierry