Notes from NORDUnet/Terena Gigabit Networking Conference

June 7-10, 1999

Lund, Sweden

Quality of Service and the Internet

The scale of the problem is millions of simultaneous connections through the backbone. This is too big to build in the lab, and too big to simulate. That leaves only one choice: use good design principles.

What is known about performance and the net is mostly anecdotal. The performance of the net swings dramatically over the course of a short time. Studies show that on average as much as 20% of the packets across international links are dropped. Most of this is due to congestion.

The problem is congestion, the solution is limiting traffic through differentiated services (diffserv, DiffServ).

Diff services must

• scale to the net as a whole (negative example of RSVP)
• support multicast
• offer a reasonable (not too big, not too small) number of services that can be understood
• be hardware independent
• be network independent - run on non-transparent networks
• be backward compatible (or it won't be deployed)

The basic diff serv model is

Source ---1--> Classifier ----2---> Network ------> Destination

packets are unmarked at 1
packets are marked by the classifier (2) with their DS Field

The Source and the Classifier may be one and the same.

The Classifier may due traffic shaping as well.

Forwarding

This is the job done by the routers, based on the DS Field of each packet.
This is known as Per Hop Behavior (PHB), and the overall qos result (source -> destination) is the concatenation of many PHBs along the way. That means

• realistically the overall qos experienced will be statistical, not purely deterministic
• you need standardized PHBs that various systems agree on, or you'll end up with no qos guarantees
• DS fields have to potentially be rewritten from AS to AS, so can't be part of checksum or encrypted

DS Field

6 bits, in IPv4 done in the TypeOfService field (8 bits)

Standardized in RFCs as of this week.
 

first 3 bits	second 3 bits	diffserv meaning
000	000	traditional best effort; no qos
ccc	000	class selector, 000 low, 111 high
xxx	xx1	reserved for local/experimental uses
101	110	EF (Expedited Forwarding); virtual leased line service
001	010
...		AF (Assured Forwarding) 4 classes of 3 drop precedences each (12 total combos)
100	110

Models

1. Use RSVP on campus, apply diffserv at the boundary router.
2. no RSVP, sources apply diffserv info directly.

 

 
 
 

In either case, the ISP's router will check for conformance to the service level agreement (SLS).

SLS

The service agreement between customer and ISP. A set of rules for how to forward various diffserv classes. The rules are stored in a database and must be delivered to the routers. There are quite a few proposals for doing this (SNMP, LDAP, etc).

Routers

Treat different classes of traffic differently. Conceptually more than 1 queue, in practice it's about queue handling. This isn't the hard part. Routers can take advantage of support for priority traffic if the underlying hardware/protocol of the outgoing link supports it (e.g ATM), but it isn't necessary (e.g. Ethernet).

You can view a "diffserv Internet" as an NxM network of queues (N routers, M classes). No one has analysed it this way. The mathematics are unkown, probably hard. Simulations are probably not feasible.

The basic approach is to allocate network forwarding resources (bandwidth, buffer space) using WFQ, priorities, etc, so that you can offer AF services end-to-end.

Random early drop is used. The threshold for dropping varies with the "drop precedence" for any given traffic class. He used "red", "green", "yellow" as examples.

Linux has a complete implementation that supports AP diffserv. I take it he meant something older than the stuff done at Karlsruhe and reported on at IWQoS?

The services he believes are important are:

Assured Bandwidth
Assured Delay

They use this service for their international connections, claim to have adequate intra-Finland bandwidth so that diffserv isn't necessary.

QBone is trying to bring the logjam of not having apps that need QoS because the networks don't provide QoS, and the networks won't provide it until there are apps to use it.

Claims there is a lot of overprovisioning in the US Internet2. He sees this changing, and doesn't think extra capacity does away with the QoS problem. Data showing east-west coast connection across Internet2 still shows congestion can be a problem, even with extra capacity.

The QoS working group

About 20 institutions
Luleå - Stephen Pink?
Susan Hares of Merit  (state of Michigan ISP) leads the "bandwidth broker" group

www.internet2.edu

Someone asked: Why not ATM, since ATM does qos already. Answer: IP goes everywhere, does everything. That's where the world has gone, the techniques for qos must follow.

The problem is hard: no one really knows how to put diffserv into the Internet, and no one can predict how any given technique will work.

There are huge variations in data rate across the net: modem to OC48 backbone - 3 orders of magnitude. This won't change soon.

Congestion happens at fast-slow connection points:

• The "last mile" is owned by the phone companies, improves slowly, highly regulated, major inertia, always will lag premise and backbone networks.
• Multiprovider interconnects - speeds of lines match, but traffic flow means you can get up to 8-9 speed X lines dumping into a single speed X line.

• Focus on the clouds, and their edges, not the path through the network. Large parts of the path can often be uncongested, and qos is unnecessary. The edges are where the slow-fast connections are so where congestion ocurs.
• Slow links happen at administrative boundaries, but you don't usually control the router that affects you. Example: data coming into your house hits the slow link in the last mile. The router there can congest in a way that you feel, assuming what comes to your house is more interesting to you than what leaves it. But you don't have any control over that router - your ISP does. This means there are important issues of trust and authentication, since differentiating services requires crossing administrative boundaries.

Scaling and Performance

graph of throughput R versus offered traffic T

R increases linearly with T until carrying capacity of network is reached, the wires are full of bits, then it flattens out

graph of delay D versus offered traffic T

flat until point where T flattened out, then increases linearly as buffers fill, then flat

The behavior of the network is quite different when multiple source/destination pairs are using it.

graph of throughput R versus offered traffic T

R increases linearly with T, but more slowly than above

graph of delay D versus offered traffic T

delay is flat

In such a situation, when the network is at or over capacity, then you can think about the multiple sources competing to get the empty slot on the "end" of the queue when a packet is transmitted. The chance of getting that empty slot is higher the faster you send data. The network will be congested, and everybody's performance will be worse than if congestion was avoided, but the individual station will get relatively more by being greedy and sending lots of data as fast as possible.

graph of throughput T versus average delay D

rises almost linearly, starts to flatten, then begins to decrease

On the far right side we have a congested network: low throughput and long delays.

The problem is a "tragedy of the commons" or "prisoners dilemma" problem. It is in the best interest of everybody to avoid congestion and put less traffic into the network, but if someone cheats, they will get a disproportionate share of the resource.

Some reasons for this behavior

Buffers which are full can not help with the normal small bursts that ocurr. As a result bursts cause dropping.

Analogy with funnels and pipes and waters.

Blocking of queues. Congested links can fill buffers and cause problems even for traffic which may be going to a non-congested destination.

Nice analogy with highways, exit ramps, congestion, and rude driver behavior.

The solution is to push the queues out to the edges of the network, to avoid congesting the queues on the interior. Two approaches:
1. Admission control (Expedited Forwarding)
2. Cooperation between end nodes and network (TCP congestion control)


The problem with the second technique is that it is doomed to failure because of the tragedy of the commons nature of the problem.

1. This is very hard to do with a highly heterogeneous network.

2. The difference in path lengths is a problem.

 

 
Traffice flows from R to R, and from B to B. Congestion ocurrs at G2, since the outgoing links are smaller than the incoming line. Since the line from G2 to R is much longer than from G2 to B, it takes many more packets to "fill the pipe" for that connection. Having more packets outstanding is necessary for R to have good performance. However, G2 will see more R packets, and the approach taken to reduce congestion might unfairly penalize R just because it happens to be a long path.

A single ISP might have an ecommerce company with only a few hosts, and a large campus with 1000s of hosts. Each customer might have a 45 Mbps line to the ISP. If diffserv is done by flows, then the campus would be favored. The router at the boundary of the ISP has a hard time distinguishing flows, and deciding how to treat them fairly. Move two hops in from the boundary and the distinction is completely lost.

Conclusion

Trust and incentive structures will be the key to solving congestion problems via diffserv.

Is RSVP dead?

VJ: RSVP signalling + Int Serv might actually be possible on a large scale, but it would incredibly expensive (economics of routers, numbers of routers). It's an economic issue. RSVP signalling is separate from Int Serve, and is a good and useful protocol for communicating what needs to be said about quality of service, network capacity, etc.

JH: The same arguments apply to the EP or "virtual leased line" that VJ is promoting.

VJ: No they don't because EP scales across many aggregation/de-aggregations of traffic. This is key. The information about customer must somehow be preserved as traffic is aggregated.

RSVP has not worked for IP telephony. The problem is that RSVP requires an IP/port and that info isn't available in IP telephony until the call is setup. Once done, RSVP might find that there isn't adequate resources: your phone rings, then you hear a busy signal.

RSVP has been used for diffserv signalling - MS and others have used it this way. It seems to work fine.

VJ on multicast

VJ on economics

Diffserv - what was the question?

VJ: Customers want both EF and AF services. EF doesn't have fixed endpoints like real leased lines. The major similarity is the temporal guarantee. The EF/virtual leased line should appear by all temporal performance measures to be a fixed leased line.

When you aggregate traffic you get burstiness. This is what caused early ATM switches to have too small buffers: they didn't take into account the bursty problems of coincidentally synchronized incoming lines. When you aggregate all the way up to the Internet backbone, you can no longer cope with the burstiness. It just can't be done. So something must be done to decrease burstiness, or to de-synchronize the aggregated streams. EF/VLL retimes signals to do just that. So it scales better.

BC: Confirms that there is strong customer demand for EF/VLL service for ecommerce.