TLS/SSL Deep Dive: Protocol Internals, Vulnerabilities and Security Assessment

Transport Layer Security (TLS) is the cryptographic protocol that secures the majority of internet traffic — from HTTPS web browsing to email delivery, VPN tunnels, and API communication. Despite its critical role, TLS has suffered from a series of severe vulnerabilities over the past decade, several of which affected implementations used by billions of devices. This guide covers the TLS protocol in depth, including how the handshake works, what can go wrong, and how to assess TLS implementations during security assessments.

TLS Protocol Versions

TLS 1.2 Handshake Internals

Understanding the TLS handshake is critical for understanding where attacks occur:

Client                                          Server
  |                                               |
  |------ ClientHello ------------------------->  |
  |       (supported cipher suites, TLS versions,|
  |        random_C, session ID, extensions)      |
  |                                               |
  |<------ ServerHello -----------------------    |
  |       (selected cipher suite, TLS version,   |
  |        random_S, session ID)                  |
  |                                               |
  |<------ Certificate -----------------------    |
  |       (server's X.509 certificate chain)      |
  |                                               |
  |<------ ServerKeyExchange (optional) ------    |
  |       (DH/ECDH parameters if DHE/ECDHE)       |
  |                                               |
  |<------ ServerHelloDone -------------------    |
  |                                               |
  |------ ClientKeyExchange ----------------->    |
  |       (pre-master secret or DH key share)     |
  |                                               |
  |------ ChangeCipherSpec ------------------>    |
  |------ Finished (encrypted) -------------->    |
  |                                               |
  |<------ ChangeCipherSpec ------------------    |
  |<------ Finished (encrypted) --------------    |
  |                                               |
  |====== Application Data (encrypted) ======     |

TLS 1.3 Handshake — Simplified and More Secure

Client                                          Server
  |                                               |
  |------ ClientHello + KeyShare ------------->   |
  |       (only TLS 1.3 ciphers, ECDH key share)  |
  |                                               |
  |<------ ServerHello + KeyShare ------------    |
  |<------ {EncryptedExtensions} (encrypted)      |
  |<------ {Certificate} (encrypted)             |
  |<------ {CertificateVerify} (encrypted)        |
  |<------ {Finished} (encrypted)                 |
  |                                               |
  |------ {Finished} (encrypted) ------------>    |
  |====== Application Data (encrypted) ======     |

# Key improvements in TLS 1.3:
# - 1-RTT handshake (vs 2-RTT in TLS 1.2)
# - 0-RTT resumption (session resumption without extra round trip)
# - Forward secrecy is MANDATORY (ECDHE only)
# - Removed: RSA key exchange, DH, RC4, 3DES, MD5, SHA-1
# - Server certificate encrypted immediately (privacy improvement)
# - Simplified cipher suites (only AEAD)

Cipher Suite Analysis

A cipher suite specifies the cryptographic algorithms for key exchange, authentication, bulk encryption, and MAC in a single string.

TLS 1.2 Cipher Suite Format

# Format: TLS_[KeyExchange]_[Authentication]_WITH_[Cipher]_[KeySize]_[MAC]
# Example: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

# Breaking down: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
# TLS           = Protocol
# ECDHE         = Key exchange (Elliptic Curve Diffie-Hellman Ephemeral — provides FS)
# RSA           = Authentication (server certificate type)
# WITH          = separator
# AES_256       = Bulk cipher (AES with 256-bit key)
# GCM           = Mode (Galois/Counter Mode — AEAD, preferred)
# SHA384        = MAC/PRF algorithm

# Strong cipher suites (TLS 1.2):
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256

# Weak cipher suites to flag:
TLS_RSA_WITH_RC4_128_MD5          # RC4 (broken), MD5, no FS
TLS_RSA_WITH_3DES_EDE_CBC_SHA     # SWEET32 attack, no FS
TLS_RSA_WITH_AES_128_CBC_SHA      # No FS, CBC mode (BEAST)
TLS_ECDHE_RSA_WITH_RC4_128_SHA   # RC4 broken even with FS
TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA  # Export-grade — FREAK/Logjam

TLS 1.3 Cipher Suites

# TLS 1.3 simplified format (key exchange separated)
# Only AEAD ciphers allowed:
TLS_AES_256_GCM_SHA384      # Preferred
TLS_AES_128_GCM_SHA256
TLS_CHACHA20_POLY1305_SHA256
TLS_AES_128_CCM_SHA256
TLS_AES_128_CCM_8_SHA256    # Reduced tag (for IoT)

Major TLS/SSL Vulnerabilities

BEAST (CVE-2011-3389)

# Browser Exploit Against SSL/TLS
# Affects: TLS 1.0 and SSL 3.0 using CBC mode
# Attack: Man-in-the-middle injects chosen plaintext,
#         exploits predictable IV in CBC mode to decrypt cookies

# How: In TLS 1.0, the IV for each record is the last ciphertext block
# of the previous record (chained IVs) — making IV predictable
# Attacker can choose plaintext to target specific bytes (one byte at a time)

# Detection:
nmap --script ssl-enum-ciphers target.com | grep "TLSv1.0"
openssl s_client -connect target.com:443 -tls1

# Mitigation: Upgrade to TLS 1.2+, disable TLS 1.0
# Server-side RC4 was temporary workaround (now RC4 itself is broken)

CRIME (CVE-2012-4929)

# Compression Ratio Info-leak Made Easy
# Affects: TLS/SSL with DEFLATE compression + HTTPS
# Attack: Compresses plaintext before encryption — length leaks info
# If attacker can inject into a request alongside secret, compression ratio
# reveals bytes of the secret (guessing one byte at a time)

# This is why TLS compression and HTTP compression together are dangerous
# Variant: BREACH (against HTTP-level compression, harder to fix)

# Detection:
openssl s_client -connect target.com:443 2>&1 | grep "Compression"
# "Compression: NONE" = safe
# "Compression: zlib compression" = vulnerable

# Mitigation: Disable TLS compression entirely
# In nginx: ssl_comp_level 0;
# In Apache: SSLCompression off

POODLE (CVE-2014-3566)

# Padding Oracle On Downgraded Legacy Encryption
# Affects: SSL 3.0 (original POODLE), TLS CBC implementations (variant)
# Attack: MITM downgrades connection to SSL 3.0
#         Exploits CBC padding oracle in SSL 3.0

# How: SSL 3.0 padding is implementation-defined (any value valid)
# Unlike TLS where padding must be a specific repeated value
# Attacker modifies last block of CBC ciphertext and observes response
# Can recover session cookies by sending ~256 requests per byte

# Detection:
openssl s_client -ssl3 -connect target.com:443
testssl.sh --poodle target.com
nmap --script ssl-poodle target.com

# TLS-POODLE: Some TLS implementations also vulnerable to CBC padding oracle
# More narrowly exploitable but still serious

# Mitigation:
# Disable SSL 3.0 entirely
# TLS fallback SCSV (RFC 7507) prevents downgrade
# nginx: ssl_protocols TLSv1.2 TLSv1.3;
# Apache: SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1

Heartbleed (CVE-2014-0160)

# Heartbleed — memory leak in OpenSSL's heartbeat extension
# Affects: OpenSSL 1.0.1 through 1.0.1f
# Severity: Critical — allowed reading 64KB of server memory per request
# Exposed: Private keys, session tokens, passwords in memory

# The bug: Heartbeat request says "I'm sending X bytes"
# but sends fewer. Server reads X bytes from memory and echoes back
# — including unrelated memory contents

# Detection:
nmap --script ssl-heartbleed -p 443 target.com
testssl.sh --heartbleed target.com

# Manual test:
python heartbleed-poc.py target.com

# What attackers extracted:
# - SSL private keys (allowing decryption of all past traffic if no FS)
# - Session tokens (account takeover)
# - Plaintext usernames and passwords (from memory buffers)
# - RSA private keys

# Mitigation: Upgrade OpenSSL, regenerate all keys, revoke old certificates

DROWN (CVE-2016-0800)

# Decrypting RSA with Obsolete and Weakened eNcryption
# Affects: Servers supporting SSL 2.0 that share keys with TLS servers
# Attack: Uses SSL 2.0 as an oracle to decrypt modern TLS sessions

# How: Even if your HTTPS server doesn't use SSLv2,
# if ANY server uses the same private key with SSLv2 enabled
# (e.g., an old mail server), your HTTPS is vulnerable

# Detection:
testssl.sh --drown target.com
openssl s_client -ssl2 -connect target.com:443

# Mitigation:
# Disable SSL 2.0 everywhere (all services using the same key)
# Don't share private keys between services
# Check smtp/smtps, pop3s, imaps for SSLv2 exposure

FREAK and Logjam

# FREAK (CVE-2015-0204): Factoring RSA Export Keys
# US export restrictions mandated 512-bit RSA keys in 1990s
# These "export" cipher suites were never removed from many servers
# 512-bit RSA factorable in hours with a modern computer

# Logjam (CVE-2015-4000): Discrete log on 512-bit DH
# Similar attack on DHE_EXPORT cipher suites
# 512-bit Diffie-Hellman can be factored — many sites use same 1024-bit DH params
# NSA likely already has precomputed tables for common 1024-bit primes

# Detection:
testssl.sh --freak --logjam target.com
nmap --script ssl-dh-params target.com

# For custom DH parameters (avoiding common primes):
openssl dhparam -out dhparam.pem 4096
# In nginx: ssl_dhparam /path/to/dhparam.pem;
# In Apache: SSLOpenSSLConfCmd DHParameters "/path/to/dhparam.pem"

testssl.sh — Comprehensive TLS Assessment

# testssl.sh is the most comprehensive open-source TLS testing tool
# GitHub: https://github.com/drwetter/testssl.sh

# Installation
git clone --depth 1 https://github.com/drwetter/testssl.sh.git
cd testssl.sh

# Full assessment
./testssl.sh target.com
./testssl.sh https://target.com:443

# Specific checks
./testssl.sh --heartbleed target.com
./testssl.sh --poodle target.com
./testssl.sh --drown target.com
./testssl.sh --freak target.com
./testssl.sh --logjam target.com
./testssl.sh --beast target.com
./testssl.sh --breach target.com
./testssl.sh --crime target.com
./testssl.sh --sweet32 target.com
./testssl.sh --robot target.com      # ROBOT attack on RSA decryption

# Cipher suite enumeration
./testssl.sh --cipher-per-proto target.com  # ciphers grouped by protocol
./testssl.sh --ciphers target.com            # all cipher suites

# Certificate analysis
./testssl.sh --certificate target.com

# Headers analysis (HSTS, HPKP, CSP, etc.)
./testssl.sh --headers target.com

# Complete output to JSON for reporting
./testssl.sh --jsonfile output.json target.com
./testssl.sh --htmlfile output.html target.com

# Multiple targets from file
./testssl.sh --file targets.txt

# Non-HTTPS services
./testssl.sh smtp:mail.target.com:587    # SMTP STARTTLS
./testssl.sh imap:mail.target.com:143    # IMAP STARTTLS
./testssl.sh ldap:ldap.target.com:389    # LDAP STARTTLS
./testssl.sh ftp:ftp.target.com:21       # FTP STARTTLS

sslyze — Python TLS Scanner

# sslyze is a fast TLS scanner with Python API
pip install sslyze

# CLI usage
sslyze target.com
sslyze target.com --json_out results.json

# Specific plugins
sslyze target.com --heartbleed
sslyze target.com --robot
sslyze target.com --certinfo
sslyze target.com --elliptic_curves
sslyze target.com --openssl_ccs      # CCS injection (CVE-2014-0224)

# Multiple targets
sslyze target1.com target2.com --json_out results.json

# Python API
from sslyze import Scanner, ServerNetworkLocation, ScanCommand, ServerScanRequest

scanner = Scanner()
request = ServerScanRequest(
    server_location=ServerNetworkLocation("target.com", 443),
    scan_commands={
        ScanCommand.CERTIFICATE_INFO,
        ScanCommand.SSL_2_0_CIPHER_SUITES,
        ScanCommand.SSL_3_0_CIPHER_SUITES,
        ScanCommand.TLS_1_0_CIPHER_SUITES,
        ScanCommand.HEARTBLEED,
        ScanCommand.ROBOT,
    }
)
scanner.queue_scans([request])
for result in scanner.get_results():
    print(result)

Nmap SSL Scripts

# Enumerate all supported cipher suites
nmap --script ssl-enum-ciphers -p 443 target.com

# Check for Heartbleed
nmap --script ssl-heartbleed -p 443 target.com

# Check certificate
nmap --script ssl-cert -p 443 target.com

# POODLE check
nmap --script ssl-poodle -p 443 target.com

# DH parameters
nmap --script ssl-dh-params -p 443 target.com

# ROBOT (Return Of Bleichenbacher's Oracle Threat)
nmap --script ssl-ccs-injection -p 443 target.com

# Comprehensive SSL check
nmap --script "ssl-*" -p 443 target.com

Certificate Validation and HSTS/HPKP

# Certificate chain verification
openssl s_client -connect target.com:443 -verify 5

# Check certificate details
openssl s_client -connect target.com:443 2>/dev/null | openssl x509 -noout -text

# Key size check
openssl s_client -connect target.com:443 2>/dev/null | openssl x509 -noout -text | grep "Public-Key"

# Certificate expiry
openssl s_client -connect target.com:443 2>/dev/null | openssl x509 -noout -dates

# Check SANs (Subject Alternative Names)
openssl s_client -connect target.com:443 2>/dev/null | openssl x509 -noout -text | grep -A5 "Subject Alternative"

# HSTS (HTTP Strict Transport Security)
# Response header: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
curl -I https://target.com | grep -i "strict-transport"

# Certificate Transparency (must be included per Chrome policy)
curl -I https://target.com | grep -i "expect-ct"
# Or via certificate: signed certificate timestamp (SCT) extension

# Common issues to flag:
# - Certificate expiry < 30 days
# - Self-signed certificate
# - Hostname mismatch
# - SHA-1 signature algorithm
# - RSA key < 2048 bits
# - Missing HSTS header
# - HSTS max-age < 1 year
# - Missing includeSubDomains
# - wildcard certificate (*) — broader attack surface
# - Multiple CN/SAN entries exposing internal hostnames