Introduction to the Network Subsystem

This chapter includes:

Overview of io-net
Starting io-net
The life cycle of a packet

Overview of `io-net`

The QNX 6 network subsystem consists of a process, called io-net, that loads a number of shared objects. These shared objects are arranged in a hierarchy, with the end user on the top, and hardware on the bottom.

io-net and friends

The io-net component can load one or more protocol interfaces and drivers.

This document focuses on writing new network drivers, although most of the information applies to writing any module for io-net.

As indicated in the diagram, the shared objects that io-net loads don't communicate directly. Instead, each shared object registers a number of functions that io-net calls, and io-net provides functions that the shared object calls.

Each shared object provides one or more of the following types of service:

Up producer: Produces data for a higher level (e.g. an Ethernet driver provides data from the network card to a TCP/IP stack).
Down producer: Produces data for a lower level (e.g. the TCP/IP stack produces data for an Ethernet driver).
Up filter: A filter that sits between an up producer and the bottom end of a converter (e.g. a protocol sniffer).
Down filter: A filter that sits between a down producer and the top end of a converter (e.g. Network Address Translation, or NAT).
Converter: Converts data from one format to another (e.g. between IP and Ethernet)

Note that these terms are relative to io-net and don't encompass any non-io-net interactions. For example, a network card driver (while forming an integral part of the communications flow) is viewed only as an up producer as far as io-net is concerned -- it doesn't produce anything that io-net interacts with in the downward direction, even though it actually transmits the data originated by an upper module to the hardware.

A producer can be an up producer, a down producer, or both. For example, the TCP/IP module produces both types (up and down) of packets.

When a module is an up producer, it may pass packets on to modules above it. Whether a packet originated at an up producer, or that producer received the packet from another up producer below it, from the next recipient's point of view, the packet came from the up producer directly below it.

Starting `io-net`

When you start io-net from the command line, you tell it which drivers and protocols to load:

$ io-net -del900 verbose -pttcpip if=en0:11.2 &

This causes io-net to load the devn-el900.so Ethernet driver and the tiny TCP/IP protocol stack. The verbose and if=en0:11.2 options are suboptions that are passed to the individual components.

Alternatively, you can use the mount and umount commands to start and stop modules dynamically. The previous example could be rewritten as:

$ io-net &
$ mount -Tio-net -overbose devn-el900.so
$ mount -Tio-net -oif=en0:11.2 npm-ttcpip.so

Regardless of the way that you've started it, here's the "big picture" that results:

Big picture of io-net.

Big picture of io-net.

In the diagram above, we've shown io-net as the "largest" entity. This was done simply to indicate that io-net is responsible for loading all the other modules (as shared objects), and that it's the one that "controls" the operation of the entire protocol stack.

Let's look at the hierarchy, from top to bottom:

TCP/IP stack: This is at the top of the hierarchy, as it presents a user-accessible interface. A user typically uses the socket library function calls to access the exposed functionality. (The mechanism used by the TCP/IP stack to present its interface isn't defined by io-net -- it's a private interface that io-net has no knowledge of or control over.)
IP-EN converter: In order to use the Ethernet interface, the TCP/IP stack needs the services of a converter module to add/remove the Ethernet header. As we'll see, this isolation of hardware specifics from the down producer allows for easy addition of future hardware types. It also allows for the insertion of filter modules between the down producer and the converter, or between the converter and the up producer. In this case, the IP-EN converter basically provides ARP (Address Resolution Protocol) services.
Ethernet driver: At the lowest level, there's an Ethernet driver that accepts Ethernet packets (generated by the IP module), and sends them out the hardware (and the reverse: it receives Ethernet packets from the hardware and gives them to the IP module).

As far as the QNX namespace is concerned, the following entries exist:

/dev/io-net: The main device created by io-net itself.
/dev/io-net/enN: The Ethernet device corresponding to LAN N (where N is 0 in our example).

At this point, you could open() /dev/io-net/en0, for example, and perform devctl() operations on it -- this is how the nicinfo command gets the Ethernet statistics from the driver.

Here's another view of io-net, this time with two different protocols at the bottom:

Cells and endpoints.

Cells and endpoints.

As you can see, there are three levels in this hierarchy. At the topmost level, we have the TCP/IP stack. As described earlier, it's considered to be a down producer (it doesn't produce or pass on anything for modules above it.)

In reality, the stack probably registers as both an up and down producer. This is permitted by io-net to facilitate the stacking of protocols.

When the TCP/IP stack registered, it told io-net that it produces packets in the downward direction of type IP -- there's no other binding between the stack and its drivers. When a module registers, io-net assigns it a cell number, 2 in this case.

Joining the stack (down producer) to the drivers (up producers), we have two converter modules. Take the converter module labeled IP-EN as an example. When this module registered as type _REG_CONVERTOR, it told io-net that it takes packets of type IP on top and packets of type EN on the bottom.

Again, this is the only binding between the IP stack and its lower level drivers. The IP-EN portion, along with its Ethernet drivers, is called cell 0 and the IP-Z portion, along with its Z-protocol drivers is called cell 1 as far as io-net is concerned.

The purpose of the intermediate converters is twofold:

It allows for increased flexibility when adding future protocols or drivers (simply write a new converter module to connect the two).
It allows for filter modules to be inserted either above or below the converter.

Finally, on the bottom level of the hierarchy, we have two different Ethernet drivers and two different Z-protocol drivers. These are up producers from io-net's perspective, because they generate data only in the upward direction. These drivers are responsible for the low-level hardware details. As with the other components mentioned above, these components advertise themselves to io-net indicating the name of the service that they're providing, and that's what's used by io-net to "hook" all the pieces together.

Since all seven pieces are independent shared objects that are loaded by io-net when it starts up (or later, via the mount command), it's important to realize that the keys to the interconnection of all the pieces are:

the module type
the type of packet they produce or accept on the way up and down.

The loading order isn't important -- io-net figures all this out at runtime.

The life cycle of a packet

The next thing we need to look at is the life cycle of a packet -- how data gets from the hardware to the end user, and back to the hardware.

The main data structure that holds all packet data is the npkt_t data type. (For more information about the data structures described in this section, see the Network DDK API chapter.) The npkt_t structure maintains a tail queue of buffers that contain the packet's data.

A tail queue uses a pair of pointers, one to the head of the queue and the other to the tail. The elements are doubly linked; an arbitrary element can be removed without traversing the queue. New elements can be added before or after an existing element, or at the head or tail of the queue. The queue may be traversed only in the forward direction.

The buffers form a doubly-linked list, and are managed via the TAILQ macros from <sys/queue.h>:

TAILQ_EMPTY()
TAILQ_FIRST()
TAILQ_INSERT_AFTER()
TAILQ_INSERT_BEFORE()
TAILQ_INSERT_HEAD()
TAILQ_INSERT_TAIL()
TAILQ_LAST()
TAILQ_NEXT()
TAILQ_PREV()
TAILQ_REMOVE()

Buffer data is stored in a net_buf_t data type. This data type consists of a list of net_iov_t structures, each containing a virtual (or base) address, physical address, and length, that are used to indicate one or more buffers:

net_iov_t relationship.

Data structures associated with a packet.

The TAILQ macros let you step through the list of elements. The following code snippet illustrates:

net_buf_t *buf;
net_iov_t *iov;
int       i;

// walk all buffers
for (buf = TAILQ_FIRST (&npkt -> buffers); buf;
                        buf = TAILQ_NEXT (buf, ptrs)) {
    for (i = 0, iov = buf -> net_iov; i < buf ->
         niov; i++, iov++) {
        // buffer is        :  iov -> iov_base
        // length is        :  iov -> iov_len
        // physical addr is :  iov -> iov_phys
    }
}

Going down

We'll start with the downward direction (from the end user to the hardware). A message is sent from the end user (via the socket library), and arrives at the TCP/IP stack. The TCP/IP stack does whatever error checking and formatting it needs to do on the data. At some point, the TCP/IP stack sends a fully formed IP packet down io-net's hierarchy. No provision is made for any link-level headers, as this is the job of the converter module.

Since the TCP/IP stack and the other modules aren't bound to each other, it's up to io-net to do the work of accepting the packet from the TCP/IP stack and giving it to the converter module. The TCP/IP stack informs io-net that it has a packet that should be sent to a lower level by calling the tx_down() function within io-net. The io-net manager looks at the various fields in the packet and the arguments passed to the function, and calls the rx_down() function in the IP-EN converter module.

The contents of the packet aren't copied -- since all these modules (e.g. the TCP/IP stack and the IP module) are loaded as shared objects into io-net's address space, all that needs to be transfered between modules is pointers to the data (and not the data itself).

Once the packet arrives in the IP-EN converter module, a similar set of events occurs as described above: the IP-EN converter module converts the packet to an Ethernet packet, and sends it to the Ethernet module to be sent out to the hardware. Note that the IP-EN converter module needs to add data in front of the packet in order to encapsulate the IP packet within an Ethernet packet.

Again, to avoid copying the packet data in order to insert the Ethernet encapsulation header in front of it, only the data pointers are moved. By inserting a net_buf_t at the start of the packet's queue, the Ethernet header can be prepended to the data buffer without actually copying the IP portion of the packet that originated at the TCP/IP stack.

Going up

In the upward direction, a similar chain of events occurs:

The Ethernet driver receives data from its hardware, and allocates one or more packets into which it places the data. (For efficiency, it may use memory-mapping tricks to cause the hardware to directly place the packet into a preallocated area.)
The Ethernet driver then calls io-net's tx_up() function, telling it that it has a packet that's ready to be given to a higher level.
The io-net manager figures out which module should get the packet and calls that module's rx_up() function. In our example, this is the IP-EN converter module, as it now needs to look at the packet and get at just the IP portion (the packet arrived from the hardware with Ethernet encapsulation).

Note that in an upward-headed packet, data is never added to the packet as it travels up to the various modules, so the list of net_buf_t structures isn't manipulated. For efficiency, the arguments to io-net's tx_up() function (and correspondingly to a registered module's rx_up() function) include off and framelen_sub. These are used to indicate how much of the data within the buffer is of interest to the level to which it's being delivered.

For example, when an IP packet arrives over the Ethernet, there are 14 bytes of Ethernet header at the beginning of the buffer. This Ethernet header is of no interest to the IP module -- it's relevant only to the Ethernet and IP-EN converter modules. Therefore, the off argument is set to 14 to indicate to the next higher layer that it should ignore the first 14 bytes of the buffer. This saves the various levels in io-net from continually having to copy buffer data from one format to another.

The framelen_sub argument operates in a similar manner, except that it refers to the tail end of the buffer -- it specifies how many bytes should be ignored at the end of the buffer, and is used with protocols that place a tail-end encapsulation on the data.