Qnet Networking

QNX distributed
Name resolution and lookup
Quality of Service and multiple paths
Examples

QNX distributed

Earlier in this manual, we described message passing in the context of a single node (see the section "Neutrino IPC" in the Neutrino Microkernel chapter). But the true power of QNX lies in its ability to take the message-passing paradigm and extend it transparently over a network of Neutrino microkernels.

This chapter describes QNX native networking (via the Qnet protocol). For information on TCP/IP networking, please refer to the next chapter.

To understand how network-wide IPC works, consider two processes that wish to communicate with each other: a client process and a server process. In the single-node case, the client simply creates a connection (via ConnectAttach()) to the server, and then sends a message (perhaps via MsgSend()).

Now consider the case of a simple network with two machines -- one contains the client process, the other contains the server process. The code required for client-server communication is identical to the code in the single-node case. The client creates a connection to the server and sends the server a message. The only difference in the network case is that the client specifies a different node descriptor for the ConnectAttach() function call in order to indicate the server's node.

While this discussion emphasizes the Neutrino kernel calls (ConnectAttach(), MsgSend()), keep in mind that the higher-level POSIX calls, such as open(), read(), write(), etc., all use those underlying kernel calls as well.

Name resolution and lookup

How does the client know what node descriptor to use for the server?

The client uses the pathname space to "look up" the server's address. In the single-machine case, the result of that lookup will be a node descriptor, a process ID, and a channel ID. In the networked case, the results are the same -- the only difference will be the value of the node descriptor.

If node descriptor is:	Then the server is:
0 (or ND_LOCAL_NODE)	Local (i.e. "this node")
Nonzero	Remote

Connection ID

The practical result in both the local case and the networked case is that when the client connects to the server, the client gets a connection ID (returned from ConnectAttach()). This connection ID is then used for all subsequent message-passing operations.

Behind a simple open()

Suppose a client on one node (magenta) wishes to use the serial port (/dev/ser1) on another node (wintermute). The client will effectively perform an open() on the pathname /net/wintermute/dev/ser1.

Network nodes generally have the prefix /net, but other prefixes are possible as we'll see shortly.

The following diagram shows the steps involved when the client open()'s /net/wintermute/dev/ser1:

local and remote message pass

A client-server message pass across the network.

Here are the interactions:

A message is sent from the client to its local process manager, effectively asking who should be contacted to resolve the pathname /net/wintermute/dev/ser1.
Since the native network manager (npm-qnet) has taken over the entire /net namespace, the process manager returns a redirect message, saying that the client should contact the local network manager for more information.
The client then sends a message to the local network manager, again asking who should be contacted to resolve the pathname.
The local network manager then replies with another redirect message, giving the node descriptor, process ID, and channel ID of the process manager on node wintermute -- effectively deferring the resolution of the request to node wintermute.
The client then creates a connection to the process manager on node wintermute, once again asking who should be contacted to resolve the pathname.
The process manager on node wintermute returns another redirect, this time with the node descriptor, channel ID, and process ID of the serial driver on its own node.
The client creates a connection to the serial driver on node wintermute, and finally gets a connection ID that it can then use for subsequent message-passing operations.
After this point, from the client's perspective, message passing to the connection ID is identical to the local case. Note that all further message operations are now direct between the client and server.

The key thing to keep in mind here is that the client isn't aware of the operations taking place; these are all handled by the POSIX open() call. As far as the client is concerned, it performs an open() and gets back a file descriptor (or an error indication).

In each subsequent name-resolution step, the request from the client is stripped of already-resolved name components; this occurs automagically within the resource manager framework. This means that in step 2 above, the relevant part of the request is wintermute/dev/ser1 from the perspective of the local network manager. In step 3, the relevant part of the request has been stripped to just dev/ser1, because that's all that wintermute's process manager needs to know. Finally, in step 4, the relevant part of the request is simply ser1, because that's all the serial driver needs to know.

Network naming

As mentioned earlier, the pathname prefix /net is the most common name that npm-qnet uses; other prefixes can be specified on the command line or in buildfiles.

In resolving names in a network-wide pathname space, the following terms come into play:

node name: A character string that identifies the node you're talking to. Note that a node name can't contain slashes or dots. In the example above, we used wintermute as one of our node names. The default is fetched via confstr() with the _CS_HOSTNAME parameter.
node domain: A character string that's "tacked" onto the node name by npm-qnet. Together the node name and node domain must form a string that's unique for all nodes that are talking to each other. The default is fetched via confstr() with the _CS_DOMAIN parameter.
fully qualified node name (FQNN): The string formed by tacking the node name and node domain together. For example, if the node name is wintermute and the node domain name is qnx.com, the resulting FQNN would be: wintermute.qnx.com.
network directory: A directory in the pathname space implemented by npm-qnet. Each network directory (there can be more than one on a node) has an associated node domain. The default is /net, as used in the examples in this chapter.
name resolution: The process by which npm-qnet converts an FQNN to a list of destination addresses that the transport layer knows how to get to.
name resolver: A piece of code that implements one method of converting an FQNN to a list of destination addresses. Each network directory has a list of name resolvers that are applied in turn to attempt to resolve the FQNN. The default is ndp (see the next section).
Quality of Service (QoS): A definition of connectivity between two nodes. The default QoS is sequential (see the section on QoS later in this chapter.)

Resolvers

The following resolvers are built into the network manager:

ndp -- Node Discovery Protocol for broadcasting name resolution requests on the LAN (similar to the TCP/IP ARP protocol). This is the default.
dns -- Take the node name, add a dot (.) followed by the node domain, and send the result to the TCP/IP gethostbyname() function.

Quality of Service (QoS) and multiple paths

Quality of Service (QoS) is an issue that often arises in high-availability networks as well as realtime control systems. QoS really boils down to dependable transmission. If you're sending alarm data across the network in a process control system, you'll want to guarantee transmission rates, error rates, etc. in advance.

In QNX, the default QoS policy (called sequential) works like this. The first network link is used until it fails, then the next link is used, and so on. The user specifies the preferred order of the links: the most desired link is specified first, followed by the second choice, and so on.

If a link fails, Qnet will try periodic maintenance packets on that link to detect recovery. When the link recovers, it's placed back into the pool of available links.

If a less-desired link is currently in use, and a more-desired link comes back into service, then communications over the more-desired link will be automatically reinstated.

You specify the QoS policy as part of the pathname. For example, to access /net/wintermute/dev/ser1 with a QoS of sequential, you could use the following pathname:

/net/wintermute~sequential/dev/ser1

The QoS parameter always begins with a tilde (~) character.

Symbolic links

You can set up symbolic links to the various "QoS-qualified" pathnames:

ln -sP /net/wintermute~sequential /net/sql_server

This assigns an "abstracted" name of /net/sql_server to the node wintermute with a sequential QoS.

Abstracting the pathnames by one level of indirection gives you multiple servers available in a network, all providing the same service. When one server fails, the abstract pathname can be "remapped" to point to the pathname of a different server. For example, if wintermute failed, then a monitoring program could detect this and effectively issue:

rm /net/sql_server
ln -sP /net/magenta~sequential /net/sql_server

This would remove wintermute and reassign the service to magenta. The real advantage here is that applications can be coded based on the abstract "service name" rather than be bound to a specific node name.

Examples

Let's look at a few examples of how you'd use the network manager.

The QNX native network manager npm-qnet is actually a shared object that installs into the executable io-net.

Local networks

If you're using Neutrino on a small LAN, you can use just the default ndp resolver. When a node name that's currently unknown is being resolved, the resolver will broadcast the name request over the LAN, and the node that has the name will respond with an identification message. Once the name's been resolved, it's cached for future reference.

Since ndp is the default resolver when you start npm-qnet.so, you can simply issue commands like:

ls /net/wintermute/

If you have a machine called "wintermute" on your LAN, you'll see the contents of its root directory.

Remote networks

For security reasons, you should have a firewall set up on your network before connecting to the Internet.

Qnet uses DNS (Domain Name System) when resolving remote names. To use npm-qnet.so with DNS, you specify this resolver on mount's command line:

mount -Tio-net -o"mount=:,resolve=dns,mount=.com:.net:.edu" /lib/dll/npm-qnet.so

In this example, Qnet will use both its native ndp resolver (indicated by the first mount= command) and DNS for resolving remote names.

Note that we've specified several types of domain names (mount=.com:.net:.edu) as mountpoints, simply to ensure better remote name resolution.

Now you could enter a command such as:

ls /net/qnet.qnx.com/repository

and you'd get a listing of the repository directory at the qnet.qnx.com site.