Web Browsers and HTTP/S connections


This is the first of a series of posts about HTTP/S that you will find in this blog during the next days. I will explain technical aspects about these protocols, how they work, how to test and validate connections, sites and certificates. I will also write about the encryption algorithms and more.

HTTP Connection


First of all, let’s analyze what a browser needs to do to display the website/application that we request.

Imagine you want to open the website http://www.google.com and you type the address in the address bar.
The browser first needs to connect to the server which hosts the site but for that to happen it needs to know the server IP. It makes a DNS query to the DNS server asking the IP address or addresses.
In plain text it would be something like this:

#nslookup google.com
Server: 172.20.25.24
Address: 172.20.25.24#53

Non-authoritative answer:
Name:	google.com
Addresses:  2607:f8b0:4000:803::1005
          173.194.46.9
          173.194.46.7
          173.194.46.3
          173.194.46.14
          173.194.46.2
          173.194.46.8
          173.194.46.6
          173.194.46.4
          173.194.46.1
          173.194.46.0
          173.194.46.5

After making this query the browser selects one of them and establishes a TCP connection to the port 80 of the server with the selected IP address. I will make this connection with ncat to simulate the process:

# ncat 173.194.46.9 80

If the TCP connection started by the browser is successful then it asks for the application under the domain you specified in the address bar. In the form of the HTTP protocol:

# ncat 173.194.46.9 80
GET / HTTP/1.1
Host: www.google.com

The server sends the response as HTTP protocol with HTTP headers – body and finally the browser extracts all this information and sends it to the engine to be rendered and displayed it in the screen.

HTTP/1.1 302 Found
Location: http://www.google.com.mx/
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Set-Cookie: PREF=ID=2b72ebdb0be8ed0d:FF=0:TM=1356032320:LM=1356032320:S=l-CTizrywMG3GXXR; expires=Sat, 20-Dec-2014 19:38:40 GMT; path=/; domain=.google.com
Set-Cookie: NID=67=dqEUHeOSZsR3K6hgUCDM-VxVJaxODX-GNvyJfXJsdjSj1L2ze_47R02CVZw2cAl_kciC6unhtm8ZIzCJc48D1_iiljffmRDu3OMJ7IDV5s4z2Abp0_toCtmu1G4zBa4l; expires=Fri, 21-Jun-2013 19:38:40 GMT; path=/; domain=.google.com; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Date: Thu, 20 Dec 2012 19:38:40 GMT
Server: gws
Content-Length: 222
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN

Because the protocol is HTTP the content of the requests and responses are in plain text. Once established the connection between the server and the client (browser) all the communcation flows in the same way.

HTTPS Connection


With HTTPS the things are a little bit different. After the TCP connection and before de HTTP request there is a negotiation also between the client – server but by SSL/TLS protocol. This protocol is responsible for making the connection and the data secure.

The web browsers have an engine called Public Key Infrastructure also known as PKI. This handles the connections by HTTPS among other encrypted connections.

We can check the SSL negotiation with openssl and its client feature.
Before analyzing the next url (https://www.google.com) I will explain a few things.
When the browser detects an “https” into the address bar it connects by TCP to the port 443 instead of the port 80 as in “http”.
The reason of this is that the port 80 expects an HTTP request method as the first command it receives, a command like GET / POST / TRACE / OPTIONS / HEAD / CONNECT / PUT / DELETE, but by protocol, the HTTPS does not send the HTTP method immediately after the TCP connection, instead it starts a handshake called TLS handshake to encrypt all the data that will be send and receive by such port.
Let’s see how this handshake looks like:

# openssl s_client -state -connect www.google.com:443
CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
depth=1 /C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
verify error:num=20:unable to get local issuer certificate
verify return:0
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
   i:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
 1 s:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDITCCAoqgAwIBAgIQT52W2WawmStUwpV8tBV9TTANBgkqhkiG9w0BAQUFADBM
MQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkg
THRkLjEWMBQGA1UEAxMNVGhhd3RlIFNHQyBDQTAeFw0xMTEwMjYwMDAwMDBaFw0x
MzA5MzAyMzU5NTlaMGgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlh
MRYwFAYDVQQHFA1Nb3VudGFpbiBWaWV3MRMwEQYDVQQKFApHb29nbGUgSW5jMRcw
FQYDVQQDFA53d3cuZ29vZ2xlLmNvbTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkC
gYEA3rcmQ6aZhc04pxUJuc8PycNVjIjujI0oJyRLKl6g2Bb6YRhLz21ggNM1QDJy
wI8S2OVOj7my9tkVXlqGMaO6hqpryNlxjMzNJxMenUJdOPanrO/6YvMYgdQkRn8B
d3zGKokUmbuYOR2oGfs5AER9G5RqeC1prcB6LPrQ2iASmNMCAwEAAaOB5zCB5DAM
BgNVHRMBAf8EAjAAMDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9jcmwudGhhd3Rl
LmNvbS9UaGF3dGVTR0NDQS5jcmwwKAYDVR0lBCEwHwYIKwYBBQUHAwEGCCsGAQUF
BwMCBglghkgBhvhCBAEwcgYIKwYBBQUHAQEEZjBkMCIGCCsGAQUFBzABhhZodHRw
Oi8vb2NzcC50aGF3dGUuY29tMD4GCCsGAQUFBzAChjJodHRwOi8vd3d3LnRoYXd0
ZS5jb20vcmVwb3NpdG9yeS9UaGF3dGVfU0dDX0NBLmNydDANBgkqhkiG9w0BAQUF
AAOBgQAhrNWuyjSJWsKrUtKyNGadeqvu5nzVfsJcKLt0AMkQH0IT/GmKHiSgAgDp
ulvKGQSy068Bsn5fFNum21K5mvMSf3yinDtvmX3qUA12IxL/92ZzKbeVCq3Yi7Le
IOkKcGQRCMha8X2e7GmlpdWC1ycenlbN0nbVeSv3JUMcafC4+Q==
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
issuer=/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
---
No client certificate CA names sent
---
SSL handshake has read 1772 bytes and written 307 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-SHA
    Session-ID: FF198D617DDDB8CBDB6766C191D5EE1882761F34C4F4C9C5DC9077AC2D0229D1
    Session-ID-ctx: 
    Master-Key: 02AA3B39DCAB1DAAF57AC9E5C70DA305C45C20CC2615250AB9C9C8D6E02BA80E670B43247AE27F6A0E7A61F84C8ABDEF
    Key-Arg   : None
    Start Time: 1356062682
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---

After a successful handshake the port is ready for receiving HTTP commands.

# openssl s_client -state -connect google.com:443
CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
depth=1 /C=US/O=Google Inc/CN=Google Internet Authority
verify error:num=20:unable to get local issuer certificate
verify return:0
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=*.google.com
   i:/C=US/O=Google Inc/CN=Google Internet Authority
 1 s:/C=US/O=Google Inc/CN=Google Internet Authority
   i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIF/DCCBWWgAwIBAgIKUCYyawAAAAB1rzANBgkqhkiG9w0BAQUFADBGMQswCQYD
VQQGEwJVUzETMBEGA1UEChMKR29vZ2xlIEluYzEiMCAGA1UEAxMZR29vZ2xlIElu
dGVybmV0IEF1dGhvcml0eTAeFw0xMjEyMDYwODU1MTVaFw0xMzA2MDcxOTQzMjda
MGYxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1N
b3VudGFpbiBWaWV3MRMwEQYDVQQKEwpHb29nbGUgSW5jMRUwEwYDVQQDFAwqLmdv
b2dsZS5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBANVbVfzED5wsJwvB
MthJd0La0SgmKm0arWCccSo/xqrBRkaFln+fU6J+exYCJSAxI/7z1bvEEqUhfhL5
YfTleYcz1YkOPKl8G+sfmoD+gtRJEX8i7+864E2PMhMhelat+49QOgt4XpGzwFyJ
SgMacD8/jlsIz+JR/BrE9ozTsc/tAgMBAAGjggPPMIIDyzAdBgNVHSUEFjAUBggr
BgEFBQcDAQYIKwYBBQUHAwIwHQYDVR0OBBYEFJar3XGZYEBWccT9IgbO7f1KyBPA
MB8GA1UdIwQYMBaAFL/AMOv1QxE+Z7qekfv8atrjaxIkMFsGA1UdHwRUMFIwUKBO
oEyGSmh0dHA6Ly93d3cuZ3N0YXRpYy5jb20vR29vZ2xlSW50ZXJuZXRBdXRob3Jp
dHkvR29vZ2xlSW50ZXJuZXRBdXRob3JpdHkuY3JsMGYGCCsGAQUFBwEBBFowWDBW
BggrBgEFBQcwAoZKaHR0cDovL3d3dy5nc3RhdGljLmNvbS9Hb29nbGVJbnRlcm5l
dEF1dGhvcml0eS9Hb29nbGVJbnRlcm5ldEF1dGhvcml0eS5jcnQwDAYDVR0TAQH/
BAIwADCCApUGA1UdEQSCAowwggKIggwqLmdvb2dsZS5jb22CDSouYW5kcm9pZC5j
b22CFiouYXBwZW5naW5lLmdvb2dsZS5jb22CEiouY2xvdWQuZ29vZ2xlLmNvbYIW
Ki5nb29nbGUtYW5hbHl0aWNzLmNvbYILKi5nb29nbGUuY2GCCyouZ29vZ2xlLmNs
gg4qLmdvb2dsZS5jby5pboIOKi5nb29nbGUuY28uanCCDiouZ29vZ2xlLmNvLnVr
gg8qLmdvb2dsZS5jb20uYXKCDyouZ29vZ2xlLmNvbS5hdYIPKi5nb29nbGUuY29t
LmJygg8qLmdvb2dsZS5jb20uY2+CDyouZ29vZ2xlLmNvbS5teIIPKi5nb29nbGUu
Y29tLnRygg8qLmdvb2dsZS5jb20udm6CCyouZ29vZ2xlLmRlggsqLmdvb2dsZS5l
c4ILKi5nb29nbGUuZnKCCyouZ29vZ2xlLmh1ggsqLmdvb2dsZS5pdIILKi5nb29n
bGUubmyCCyouZ29vZ2xlLnBsggsqLmdvb2dsZS5wdIIPKi5nb29nbGVhcGlzLmNu
ghQqLmdvb2dsZWNvbW1lcmNlLmNvbYINKi5nc3RhdGljLmNvbYIMKi51cmNoaW4u
Y29tghAqLnVybC5nb29nbGUuY29tghYqLnlvdXR1YmUtbm9jb29raWUuY29tgg0q
LnlvdXR1YmUuY29tggsqLnl0aW1nLmNvbYILYW5kcm9pZC5jb22CBGcuY2+CBmdv
by5nbIIUZ29vZ2xlLWFuYWx5dGljcy5jb22CCmdvb2dsZS5jb22CEmdvb2dsZWNv
bW1lcmNlLmNvbYIKdXJjaGluLmNvbYIIeW91dHUuYmWCC3lvdXR1YmUuY29tMA0G
CSqGSIb3DQEBBQUAA4GBALzbzbHGJ/Hh2TgY+u4QEPStP8wto66SN5moJCEAvcni
4crVEwayKhp6JprUTsiC8D3r4NPj+avoZ9UXN1axgctcuJxMWt0UcnrH71J0hWNM
v6UdT7iXlxVh2oslpIksSw2bf0H7PUQQjFCMLVbiEMAQbXal4+SsyYB+GsPVvlR+
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=*.google.com
issuer=/C=US/O=Google Inc/CN=Google Internet Authority
---
No client certificate CA names sent
---
SSL handshake has read 2388 bytes and written 307 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-SHA
    Session-ID: 5D3E200EA3FEC49C57B8C00BFA2318872530483E7A024E8AB2BFFD2BF490C52A
    Session-ID-ctx: 
    Master-Key: AF8100DAA07B166028F95957F4C588D2F7919CD6FC0816B28DB758C199C43CB3A1ECC91C1B54DD20EA63657EA6CC229E
    Key-Arg   : None
    Start Time: 1356134095
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---
GET / HTTP/1.1
Host: www.google.com

HTTP/1.1 302 Found
Location: https://www.google.com.mx/
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Set-Cookie: PREF=ID=6ca905cfb23de7b7:FF=0:TM=1356134107:LM=1356134107:S=6Bd4iW4iJ4mHJMgq; expires=Sun, 21-Dec-2014 23:55:07 GMT; path=/; domain=.google.com
Set-Cookie: NID=67=ddmjOi-luVmCq2P-yLxz26qU0zNg81B76DoZqL2vTzdHiIvDiHs7JrsBeUszlEhji_nbLANurOajxjUgd1RgFTEjM4aRU0F5P1dapCzRwzFqL3qGnNbjq0FmEuR9tQXW; expires=Sat, 22-Jun-2013 23:55:07 GMT; path=/; domain=.google.com; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Date: Fri, 21 Dec 2012 23:55:07 GMT
Server: gws
Content-Length: 223
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN

In the next post I will explain technical details about the TLS handshake and how the validation of the certificates works.

  1. HTTPS and the TLS handshake protocol. « Albertx blog - pingback on January 2, 2013 at 14:20
  2. With havin so much content do you ever run into any
    issues of plagorism or copyright infringement? My site has a lot of unique content I’ve either written myself or outsourced but it looks like a lot of it is popping it up all over the web without my authorization. Do you know any methods to help prevent content from being ripped off? I’d truly appreciate it.

  3. This program is intended to recover lost passwords for RAR/WinRAR archives of
    versions 2.xx and 3.xx. http://www.passwordtowinrar.tk The free professional solution for recovering lost passwords to RAR and WinRAR archives.

    It is actually a great and useful piece of info.
    I’m happy that you simply shared this useful info with us. Please stay us informed like this. Thanks for sharing.

Leave a Comment


eight × 4 =


NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Trackbacks and Pingbacks: