Who am I?

  • Currently part of Cloud Monitoring at Rackspace
  • Previously at SuSE on Kernel team
  • Saw a talk at Linux Plumbers Conference (Tunneling Monster)

What are Cloud Networks?

Cloud Networks are...

  • Transparent to the cloud machines
  • Multi-tenant on a single IP network
  • Tenant networks can't see/talk to each other

Networking Basics

Don't we already have VLANs?

  • An extension to ethernet frames
  • 12 bits of VLAN ID, only 4096 (tenant)
  • Requires physical switch participation
    • Lots of different hardware to configure
    • Hard to do quickly on-demand
  • Conclusion: Built for a different era

L2 in L3 Tunneling

L2 in L3 tunneling to the rescue

  • You want a private L2 network between VMs
  • You already have working IP between the members of the network
  • Use that "underlay" IP network and overlay a private ethernet on top

Three Contenders

  • STT


  • Network Virtualization using Generic Routing Encapsulation (nvgre)
  • Uses the GRE for encapsulation
  • 24 bits of tenant space (16 million)
  • Microsoft/HP/Intel/Broadcom


  • Virtual eXtensible Local Area Network (vxlan)
  • Encapsulates in a UDP packet
  • 24 bits of tenant space (16 million)
  • VMWare/Cisco/Broadcom/Red Hat


  • Stateless Transport Tunneling (stt)
  • Uses a "cute" hack to get performance from network gear
    • Use the TCP header "syntax"
    • Take advantage of hardware TCP offload on existing server hardware
  • Large 64 bit context id (16 million)

STT Example Explained

  • ACK is used as an identification field
  • SEQ upper is used for STT frame length
  • SEQ lower is used for STT frame offset of this packet

STT is a cool hack! But...

  • Firewalls inspecting L3 traffic will be grumpy
    • TCP state machine isn't used
  • Could confuse the network in the middle

What about broadcast/multicast

  • Essentially all standards rely on IP multicast
  • Mapping of virtual network to multicast group is left undefined
  • Out of scope for these tunneling protocols

What is the payoff?

  • Multicast between virtual machines
  • Less burden on 100% secure firewalls
  • Can be built in a secure manner
  • Much more sophisticated attacks required

Security & Privacy

Security of L2 in L3 Tunneling

  • VM ethernet can be spoofed and inspected by any attacker at the IP level
  • Virtual Network ID must be properly communicated automically without reuse
  • All three RFCs recommend considering IPSec for underlay network

Cases where a cloud network could increase security

  • Are you lazy? A Cloud Network would be a big step up
  • Keep backend servers off of public IPs
  • Running old insecure OSes for building packages
  • Creating isolated dev/qe machines environments

Bottom Line?

Pragmatic choice is to still secure your services with TLS/SSL/IPSec

Take aways

  • L2 in L3 is probably here to stay in the cloud
  • Three emerging standards: VXLAN/NVGRE/STT
  • In its raw form these protocols give a level of privacy but not security
  • You should still use SSL/IPSec/etc between backend services

<Thank You!>