Encrypted EVPN-MPLS over interconnection fabrics with Nokia ANYsec

We are pleased to share a design we recently verified in our lab: a fully encrypted Segment Routing / EVPN-MPLS network spanning three sites over a third-party interconnection fabric, using Nokia ANYsec on the 7750 SR FP5 platform. The fabric only ever sees MPLS frames with encrypted payload, while the services on top (EVPN-VPLS, EVPN-VPWS, EVPN-VPRN) run unchanged.

Overview

Background

ANYsec is Nokia’s extension of IEEE 802.1AE MACsec from the link to the tunnel: instead of encrypting everything on a physical port, it encrypts the payload of individual MPLS / Segment Routing tunnels, end to end over any underlying transport. Key exchange uses the familiar MACsec Key Agreement (MKA), carried over IP between the tunnel endpoints, so no intermediate device has to understand or terminate anything.

On the wire a frame carries the transport label (the node SID of the destination PE), an ANYsec encryption label, the MACsec security tag (SecTAG), and then the ciphertext. The EVPN service label and the complete customer frame are inside the ciphertext, followed by the integrity check value (ICV). An observer on the fabric learns which PE talks to which, and nothing else.

This post covers tunnel encryption: one encryption group protects every service that rides a given SR tunnel. ANYsec can also be bound per service, so that encrypted and unencrypted services share the same fabric port with their own keys and labels. That variant gets its own post.

Key Benefits

End-to-end encryption in your own hands: The keys live on your PEs, the ciphertext spans PE to PE, and nothing in between, whether a fabric operator, a carrier or a colocation provider, can read or has to be trusted with the traffic. Encryption is independent of the underlay.
Fabric agnostic: The interconnection fabric forwards ordinary Ethernet / MPLS frames. No MACsec support, no VLAN tricks, no coordination with the fabric operator beyond port, VLAN and MTU.
Full mesh over one broadcast domain: One ELAN, one IS-IS subnet, SR node SIDs, and ANYsec peers between every pair of PEs. Adding a site means adding one port and one peer, not n new circuits. The same design works unchanged over point-to-point circuits (EPL / EVPL) or a mix of both, because ANYsec follows the SR tunnel, not the underlying link type.
Services stay untouched: EVPN-VPLS, VPWS and VPRN are configured exactly as on a private backbone. Encryption is a property of the transport tunnel, not of the service.
Predictable overhead: 58 bytes per frame (outer Ethernet 14, two labels 8, SecTAG 16, encrypted service label 4, ICV 16). A 1514-byte customer frame becomes 1572 bytes on the fabric, so a fabric MTU of 1600 bytes keeps a clean 1500-byte customer payload.
Per-tunnel visibility: Every peer has its own state, counters and MKA session, all available through gNMI for streaming telemetry.

Technology Stack

This design is based on Nokia 7750 SR routers with FP5 forwarding (ANYsec requires FP5 silicon) running Nokia SR OS 26.10. We verified it end to end in containerlab with Nokia’s SR-SIM, including streaming telemetry (gNMI, Prometheus, Loki, Grafana) and live packet captures on every link, which gives us a reproducible model of the setup before any hardware is touched.

Use Case

Cloud exchange and interconnection fabrics, Internet exchange points and carrier-Ethernet ELAN products all offer a convenient Layer 2 domain between sites, but no confidentiality. With ANYsec the fabric becomes a transparent transport for an encrypted SR / EVPN-MPLS core: multi-site L2 and L3 VPN services, data centre interconnect, or a private backbone rented one port per site. All while keeping the operational model of a classic Nokia IP/MPLS network and without sacrificing security.

MACsec would be the obvious first choice for encrypting such a connection, and we have shown how to run it across an IXP fabric in an earlier post. The catch: not every fabric is MACsec transparent. Many interconnection platforms terminate or rewrite the Ethernet layer, drop frames with the MACsec EtherType, or simply do not forward MKA. ANYsec sidesteps this entirely, because to the fabric it is nothing but MPLS over Ethernet, and the key exchange runs over IP between the PEs.

Configurations

pe1

pe2

pe3

Observations

Lab dashboard

Live view of the lab: link rates, port states and all three ANYsec tunnels up

ANYsec peers

All six peer directions and MKA sessions up, zero receive errors, encrypted and decrypted packets per peer

tester-a <-> pe1 and pe1 <-> fabric

Unencrypted traffic between tester-a and pe1 on the left, the same traffic as 1572-byte ANYsec frames on the fabric port on the right

pe1 <-> pe2 peer disabled

ANYsec peer pe1 – pe2 administratively disabled on both sides: the SR tunnel stays up and the service keeps forwarding

pe1 <-> fabric with the peer disabled

Frames between pe1 and pe2 now cross the fabric as plain MPLS with a readable service label and customer frame

Encryption is a property of the tunnel and the tunnel does not depend on it, so monitor the per-peer ANYsec state as a first-class signal and decide deliberately whether a site should fail open or fail closed.

If an encrypted multi-site network over a shared Layer 2 fabric fits your requirements, feel free to reach out. We are happy to walk you through the design and the lab.

Zurück