Data Packets, Light and Routing

DOTMAGAZINE: Good morning. I'd like to talk to you today about the journey a network packet it takes, when it goes from one computer to the other. What happens in between? 

WOLFGANG TREMMEL: Okay, let's start with the person before the computer. A network packet is transmitted information. The first thing you need to do is, of course, to put that information into the computer. Usually, you sit at the keyboard, so you type letters which are then transmitted into a computer, that means that your operating system somehow takes whatever you have typed and converts it into a data format, and this then you want to transmit. So you now have a sentence or a few letters – let's just imagine you have typed "hello world". Your operating system now needs to know what you want to do with it. You have said you want to transmit it. So it puts it into a packet. On the Internet, the usual common maximum packet size is 1,500 bytes or 1,500 characters. You have to add some information about error checks, and headers, and so on, so let's say this "hello world" easily fits into one package. The first thing your computer then does is put a checksum behind the packet, and it puts some destination and source information in front of the packets. Now you have a data packet with the source (this is your computer), the destination (where it has to go), the sentence "hello world" and the checksum. 

Next, the computer transmits it. Usually, nowadays, it's all wireless. So it has to be transmitted to the base station, sitting somewhere in your house. To do that, it needs more information. It needs the name of the base station, so it puts another packet around that packet, and it encrypts it because, of course, you have your network password secured. It then transmits from your computer to the base station. The base station removes the outer packet, it decrypts it again, and now you have the data packet with source, destination, "hello world", and checksum. It then puts it onto a wire to your DSL router, cable modem and so on. The cable modem or DSL router now adds another outer layer with some DSL information, and transmits it through fibers or copper through the street to some central location of your telco. There again, they see the destination address and forward it on, most likely now it's fiber.

So how does it get into the fiber? The data packet, again, gets another layer added, another checksum, and the electrical information is converted into pulses of light. It goes down to the street through a fiber and then transmitted to some central location, where again it goes up. There the light is converted to electricity again; the electricity is converted to information. It looks at the destination address or maybe at the checksum, and it calculates if the packet is still intact. If it is intact, it is forwarded to the next point, most likely again, via fiber. So again, it goes down, is converted into pulses of light again, transmitted below the street, across the country or the ocean to the next hub. Usually, we talk about hubs, if we say that the packet is transmitted. Again, converted and the checksum is checked, and if something happens to the packet – if a bit is flipped, as we call it – so, if anything changes, the checksum no longer matches. If the checksum no longer matches, the packet is discarded. It is thrown away and your computer needs to make sure it is re-transmitted. However, let's assume everything is still intact. So it goes to the next hub, where the destination is checked. It goes on and on, until, at one point, it ends up on another computer. Its information is extracted, all the headers and checksum are thrown away, assuming everything is still okay. Then, it is displayed somewhere on a screen. That's the journey. 

DOT: What are the touch points that the network packet meets through its journey? What are the steps or places it goes through? 

TREMMEL: The very first step, of course, is the operating system of your computer. Then, it goes through a ceiling or a wireless station. Then, it further goes through your DSL box, then to a central box at a central location and then through a couple of – if you have a look at nowadays the Internet usually has 10 to 12 – routers where it basically is looked at and a decision is made where to send it on. But even in between these two routers, there are touchpoints which you might not be aware of, for example, a fiber. Fiber cannot be of an indefinite length. So every so and so many kilometers the light in the fiber needs to be regenerated. Nobody looks at the packets there. We just have a look at the incoming light, at the quality of the light regenerated and send it on. But this happens also in the ocean. So, every several hundred kilometers this happens. In each router, the light is converted into electricity, checked and converted back into light, and sent on. These are the touchpoints. 

DOT: Who are the people involved in this journey? Who are the people who could affect that journey, through decisions they make or by interfering in some way? 

TREMMEL: There are so many there are so many packets going around, no one is making any live decisions, no people are making live decisions. The people are the ones setting up the network. If they work at an ISP, the routers – they configure the routers to know what to do if certain destination addresses are seen. And the routers then communicate with each other and propagate that information. Of course, accidents happen, so people driving a backhoe or a tractor are the ones who might interfere by cutting fibers by accident during construction work. That happens. 

DOT: Are there other points where accidents like that can happen? That can have a big impact on a network in a city or an area? 

TREMMEL: Yeah, impacts, like I said, in the city are most likely during construction work. So if you dig up a road and there is fiber beside the road, there is a certain probability that the fiber gets disrupted. And then things get interesting because - has your Internet provider set up enough redundancy in his network so that this accident doesn't impact too much? And, of course, if you talk about trans-ocean fibers, trans-Atlantic fibers, fishing nets, anchors, and so on can also cut through transatlantic fibers. Then it gets even more interesting, because you have to send out a ship to find where the cut is, and then have to fix the cut. That can take weeks. 

DOT: Is it possible for a person or an institution, say secret service, to follow the journey a network packets takes? Can it be tracked? Can you hide where information is coming from and going to? 

TREMMEL: You can track it to a certain level. As I said, the network has layers. You have the information layer. That's what you put in. You have a network layer. That's what you see if you use the command "trace route". But that's not all. If you do a "trace route", some people listening to this will know what I mean, you see certain hubs. But, is that really all there is? Between these hubs, there may be things hidden, because you only see the hubs on an IP layer, but below the IP layer, there may be another layer, another network layer, which is hidden, for example, an MPLS layer. MPLS is a transport protocol, which you do not see. So you might think if you send, for example, a packet from South America to Europe, you go from North America, South America, Great Britain, Netherlands, to Germany. But, in fact, it might have a hub in Spain which you do not see; that's the MPLS layer, and even below that, there's fiber there, and you absolutely cannot see that. You cannot see which fiber point, through which fibers, packets travel. To check out all these layers, you have to look very carefully: Who is involved, where are their fibers? Which points do they touch? Where are they going? And that's a huge effort, if you really want to make sure that you know every point and usually nobody does that. 

DOT: Who does? 

TREMMEL: Network engineers do it for their part of their work. So if you're working on an IP layer, you're only interested in the IP level. If you are a transport carrier, you are interested in the lower MPLS network and make sure that it is error-free and running. And if something is broken, you route it around. You do not even tell the people responsible for the layers above that you have done so. It just happens within milliseconds. If you're a fiber operator, also fibers have redundancy. And if one fiber breaks, you reroute traffic through another fiber. That also can happen within milliseconds and nobody using the layers above you might even notice. 

DOT: What are other factors that affect the journey of the network packet, so speed, for example? What can be optimized about the current technical set-up for data routing? 

TREMMEL: You cannot basically beat the speed of light. The speed of light is the absolute limit in optimization. So you can make sure you take the shortest route. So, for example, getting from South Africa to Brazil, you might want to have a direct fiber. You do not want to route from South Africa to Europe, to North America, and then down to South America. That will just increase latency. 

The speed of light through a vacuum is very well known (you can look it up on Wikipedia). But what you might not know is that the speed of light in a fiber is slower because, first, of the medium, and second... look it up. Look it up on Wikipedia: the speed of light in fiber, and it is lower than in vacuum. 

DOT: So how much lower is it? 

TREMMEL: It's about two thirds of the speed of light in vacuum. 

DOT: Is there anything that can be done to speed this up? 

TREMMEL: No. 

DOT: So that's it. We've reached our technical limit, in terms of how quickly we can transmit data, or is there something else that can be done to increase speed? 

TREMMEL: Yeah, of course, you can work on the upper layers. You can make sure that the networking equipment you use has a very low latency. Every box in between network endpoints increases latency just a little bit. And you need to make sure that this little bit is very, very small, because you cannot work with the speed of light, but you can work on making your network equipment as fast as possible. 

DOT: Who are the economic stakeholders involved in data routing? Who makes money on your data being sent? 

TREMMEL: The economics of the Internet always starts with redundancy. The more you increase redundancy, the more you increase cost. So you need to have a reasonable level of redundancy where breakage affects your network so it still operates. But you do not want have to keep too much redundancy around because of the higher cost involved. For example, something disrupts the first fiber. And then you start selling capacity. If you sell redundant capacity, you reserve the capacity on both fibers. But, of course, you can sell non-redundant capacity for half the price if it only uses one fiber. So if the fiber is disrupted, everybody with redundant capacity is switched to the second still operating fiber. Everybody who was unlucky, and everybody who has purchased non-redundant capacity and is on the broken fiber is out of network, probably until the fibers fixed. And this goes through all levels also for terrestrial circuits. 

For example, if you have a large network in Germany you might build that as a ring. So if the ring gets disrupted on one side, the packets can still go around to the other side. But if you operate your ring at 90 percent capacity, and usually packets go around both sides, you might notice if it is broken in one place, everything has to go the other way around. And the ring gets overloaded. So at what capacity do you operate your ring? If you operate it at 50 percent, it can break wherever it wants and nothing gets overloaded. But then it's more expensive. The idea is here to find the right balance between cost and redundancy. 

DOT: So where can you find out more information if you want to find out more about what routing, peering, blackholing? What's a good place to start? 

TREMMEL: What I do here at DE-CIX is run the DE-CIX academy. We give trainings and webinars to DE-CIX customers and other interested parties. So if you're interested, have a look at our website to see is currently offered. At the moment I do a training about peering tools, which shows you what tools there are outside in the world wide web which you can use to monitor your network, to detect faults, and to operate your network. These webinars happen twice a week. The list is always on our website and you are very welcome to register and to listen. 

DOT: Thank you very much for talking to us, Wolfgang. 

TREMMEL: Thank you, you are welcome! 

The Head of the DE-CIX Academy is Wolfgang Tremmel, DE-CIX's long-time Head of Customer Service. He has handed over the responsibility for the Customer Service team to Tobias Neumann to concentrate on sharing his knowledge of more than 20 years of experience in the network operation and peering field.