The network core

The network core is the mesh of packet switches and links that interconnects the Internet's end systems:

The network core

Packet switching

In a network application, end systems exchange messages which carry data.

Long messages are divided into smaller units called packets:

Store-and-forward transmission

Most packet switches use store-and-forward transmission because they must receive and process the entire packet (such as checking for errors and determining where to send it) before forwarding it to the next link.

This means that even if part of a packet has arrived, a packet switch waits until the full packet is received before sending it onward.

Due to the sequential transmission and forwarding process, packet switching with store-and-forward introduces delay.

Delays scale with the number of links and packet size.

Queuing delays and packet loss

Each router has output buffers for its outgoing links:

These delays vary based on network congestion.

If the buffer is full when a new packet arrives:

Forwarding tables and routing protocols

How routers determine which link to forward a packet to?

Forwarding tables are not manually configured:

Circuit switching

Circuit switching is one of two main approaches to data transmission in networks (the other being packet switching).

In circuit-switched networks (such as traditional telephone systems):

In contrast, packet-switched networks like the Internet do not reserve resources:

Industry trend:

A network of networks

End systems connect into the Internet via an access ISP.

But the access ISPs themselves must be interconnected.

This is done by creating a network of networks.

To understand today's Internet network structure, let's incrementally build a series of network structures to interconnect all access ISPs so that all end systems can send packets to each other:

Structure Description Key Points

Naive Solution

Every access ISP connects directly to every other access ISP forming a full mesh network
  • impractical and expensive
  • would require each access ISP to maintain separate links to hundreds of thousands of others worldwide

Network Structure 1

All access ISPs connect to one global transit ISP
  • this global ISP would have a vast infrastructure of routers and links, with a presence near every access ISP worldwide
  • very expensive to build, so the global ISP would charge access ISPs for connectivity
  • the pricing would reflect the volume of traffic exchanged
  • this creates a customer-provider relationship:
    • access ISPs = customers
    • global transit ISP = provider

Network Structure 2

Multiple competing global transit ISPs serve the many access ISPs
  • if one company builds a successful global transit ISP, others will enter the market to compete
  • competition improves price and service options
  • global transit ISPs must interconnect with each other to ensure full Internet connectivity across access ISPs using different providers
  • network Structure 2 is a two-tier hierarchy:
    • top tier (Tier-1): global transit ISPs
    • bottom tier: access ISPs
  • this structure assumes global transit ISPs can connect directly to all access ISPs:
    • but this is not practical due to cost and geography (no ISP has presence in each and every city in the world)
    • in reality, regional ISPs act as intermediaries between access ISPs and top-tier providers
  • Tier-1 ISPs (e.g., AT&T, NTT):
    • are similar to the previous single global transit ISPs but do not cover every location
    • interestingly:
      • Tier-1 ISPs are not officially designated
      • recognition is informal and based on influence and reach
  • there is a customer-provider relationship at each level of the hierarchy:
    • access ISP pays regional ISP
    • regional ISP pays Tier-1 ISP
    • Tier-1 ISPs pay nobody
    • access ISPs may connect directly to Tier-1 ISPs

Network Structure 3

Multi-tier ISP hierarchy
  • in some regions, there may be a larger regional ISP (possibly spanning an entire country) to which the smaller regional ISPs in that region connect
  • the larger regional ISP connects to a Tier-1 ISP
  • example: China
    • access ISPs in each city
    • connect to Provincial ISPs
    • which in turn connect to National ISPs
    • which finally connect to Tier-1 ISPs

Network Structure 4

Ecosystem closely resembling today's Internet
  • includes:
    • Access ISPs
    • Regional ISPs
    • Tier-1 ISPs
  • multi-homing:
    • ISPs (except Tier-1) may choose to connect to multiple provider ISPs
    • example: access ISP connects to two regional ISPs and one Tier-1 ISP
    • provides redundancy and fault tolerance (if one provider fails, the ISP can still access the Internet through others)
  • Points of Presence (PoPs) = where a network physically exists:
    • physical sites with routers, switches, etc.
    • connect customer ISPs to provider ISPs
    • exist at all levels except bottom access level
  • Internet Exchange Points (IXPs) = physical places where networks interconnect:
    • interconnect different networks (ISPs, CDNs, etc.)
    • use physical infrastructure to enable direct peering
    • operated by:
      • non-profit organizations
      • industry associations made up of participating ISPs, especially in Europe
      • for-profit, operator-neutral companies (often data center operators)
      • universities
      • government agencies
      • etc.
  • peering = the business and routing relationship between two networks:
    • customer ISPs pay provider ISPs for Internet connectivity depending on the volume of exchanged traffic
      • to reduce these costs, a pair of nearby ISPs at the same level of the hierarchy can peer
      • it means connecting their networks together
        • so that all the traffic between them passes over the direct connection
        • rather than through upstream intermediaries
    • peering is usually settlement-free:
      • neither party pays the other
      • Tier-1 ISPs also peer settlement-free with one another forming the backbone of global Internet connectivity
    • peering can occur:
      • publicly at IXPs
      • or privately through direct connections between just two networks
    • involves exchanging routing information using the Border Gateway Protocol (BGP)
    • requires compatible network infrastructure and assigned Autonomous System Numbers (ASNs)
  • relationship between PoPs, IXPs, and Peering:
    • a network first deploys equipment at PoPs
    • some PoPs connect to IXPs
    • at the IXP, networks establish peering relationships with one another
Relationship between PoPs, IXPs, and peering

Network Structure 5

Modern Internet
  • includes large content-provider networks like Google:
    • they operate massive global data centers connected via a private network
    • this private network
      • bypasses upper-tier ISPs by peering directly with lower-tier ISPs
      • often through IXPs
      • though it still pays Tier-1 ISPs when necessary.
    • by doing this, content providers reduce costs and gain more control over service delivery
Interconnection of ISPs

Today's Internet is a complex "network of networks":

Lower-tier ISPs:

Large content providers have built their own networks.

Users and large content providers are customers of lower-tier ISPs.

Previous The network edge All ⏎ Next Delay, loss, and throughput

A Kemar Joint