Inter-Domain(OSPF) and Intra-Domain(BGP) Routing Security

Pre-requisite to understand this

IP Routing – Process of forwarding packets based on destination IP address.
OSI Model – Conceptual model explaining networking layers (L1–L7).
Autonomous System (AS) – A network under a single administrative control.
IGP vs EGP – Interior vs Exterior routing protocols.
TCP/IP Basics – Understanding of IP, TCP, and packet encapsulation.
Route Redistribution – Sharing routes between different routing protocols.
Basic Security Concepts – Authentication, integrity, spoofing, hijacking.

Introduction

OSPF (Open Shortest Path First) and BGP (Border Gateway Protocol) are routing protocols standardized by the Internet Engineering Task Force. OSPF is an Interior Gateway Protocol (IGP) used within a single Autonomous System, while BGP is an Exterior Gateway Protocol (EGP) used between Autonomous Systems, including across the global Internet. From a security perspective, these protocols are critical because they control how traffic flows through networks. If routing is compromised, attackers can intercept, redirect, or drop traffic. Therefore, securing OSPF and BGP is essential for maintaining network integrity, availability, and trust.

What problem can we solve with this?

Traditional routing security ensures that internal and external traffic paths are trustworthy and not maliciously manipulated. Without proper controls, attackers can inject false routes, cause traffic blackholing, create routing loops, or perform man-in-the-middle attacks. OSPF protects internal topology from rogue routers, while BGP security mechanisms prevent route hijacking and prefix spoofing on the Internet. Together, they protect both intra-domain and inter-domain routing environments. This layered protection helps maintain business continuity and prevents large-scale outages.

Problems addressed:

Prevents rogue router injection inside the enterprise (OSPF authentication).
Prevents route hijacking between ISPs (BGP filtering, RPKI).
Protects against route spoofing and falsified LSAs.
Reduces risk of traffic interception.
Maintains routing table integrity.
Ensures availability of critical services.

How to implement/use this?

To secure traditional routing, OSPF should use authentication mechanisms (MD5 or SHA-based authentication in modern implementations). OSPF areas should be designed hierarchically to limit attack scope. For BGP, use prefix filtering, AS-path filtering, TTL security, and TCP authentication. Additionally, implement RPKI (Resource Public Key Infrastructure) for origin validation. Edge routers should strictly control route redistribution between OSPF and BGP. Logging and monitoring must be enabled to detect anomalies. Security policies must define which routes are allowed to be advertised or accepted.

Implementation steps

Enable OSPF authentication (message-digest or key-chain).
Use passive interfaces where adjacency is not required.
Apply BGP prefix filtering.
Configure AS-path access lists.
Enable TCP MD5 or TCP-AO authentication for BGP sessions.
Implement RPKI origin validation.
Restrict route redistribution with route-maps.

Sequence Diagram (OSPF + BGP Secure Interaction)

This sequence shows secure routing communication. First, internal routers establish authenticated OSPF adjacency to prevent rogue devices. Link State Advertisements (LSAs) are exchanged securely. Then the edge router establishes a BGP session with the ISP router using TCP authentication. The ISP sends filtered and validated prefixes. Finally, the edge router redistributes only approved routes into OSPF. This ensures internal routers trust only validated external paths. Security mechanisms at each step prevent spoofing and unauthorized route injection.

Sequence Security Points:

OSPF Hello packets include authentication.
LSAs validated before acceptance.
BGP uses TCP-based session protection.
Prefix filtering prevents unwanted routes.
Redistribution is tightly controlled.

Component Diagram

This component diagram represents a traditional enterprise deployment. The internal network exchanges routes with the edge router using OSPF authentication. The edge router connects externally using BGP with filtering controls. The ISP provides routing updates, which are validated. The RPKI validator ensures that advertised prefixes belong to authorized AS owners. Only validated routes are redistributed internally. This layered architecture ensures both internal and external routing integrity.

Component Security Points

Internal routing isolated via OSPF.
Edge router acts as security boundary.
BGP sessions authenticated.
RPKI validates prefix ownership.
Controlled redistribution prevents leaks.

Deployment Diagram

This deployment diagram shows physical/logical placement. Internal routers operate within the LAN and run OSPF. The edge router in the DMZ connects internal and external networks. It runs both OSPF and BGP, acting as a policy enforcement point. The ISP router represents external connectivity. The RPKI server validates BGP announcements. Security is layered: internal trust domain (OSPF), boundary enforcement (edge router), and external validation (BGP + RPKI).

Deployment Security Points

OSPF limited to internal LAN.
Edge router enforces routing policies.
BGP only on perimeter devices.
External prefix validation via RPKI.
Clear separation of trust zones.

Advantages

Protects against internal rogue routing attacks.
Prevents Internet-scale prefix hijacking.
Maintains routing table integrity.
Improves traffic reliability and availability.
Enables controlled route redistribution.
Enhances trust between network domains.
Supports scalable and hierarchical security design.

Summary

In traditional routing architectures, OSPF secures internal topology while BGP secures inter-domain connectivity. Both protocols, standardized by the Internet Engineering Task Force, require explicit security configurations to prevent route manipulation. OSPF authentication protects against rogue routers inside an enterprise, while BGP filtering, TCP authentication, and RPKI mitigate route hijacking across the Internet. Their interaction at the edge router must be tightly controlled to prevent route leaks and policy violations. When properly secured, OSPF and BGP provide a resilient and trustworthy routing foundation for enterprise and ISP networks.