system toolbox » voip

What this is all about¶

Here you find a patch for kapejod's zaphfc driver from the bristuff package. The patch improves the driver in three ways (actually, meanwhile, it contains a few more improvements—but those explained below are still the most important ones):

• It adds support for the slave mode of the HFC chip, thus allowing the frame clock of one card to sync to that of an other.
• It changes the driver to (mostly) use only one interrupt, no matter how many cards there are in the system, thereby greatly reducing the cpu load caused by the driver when handling more than one card (I call this feature "interrupt bundling" in the following).
• It makes the b channel handling (and thus the whole driver) much more tolerant to lost interrupts.

Please be aware that the latter two basically are addressed by bristuff versions 0.2.0-RC7h and later, too—in a way that introduces new problems, though. See below for what these problems are like. In the following, I will describe each of the three improvements in some more detail. The sections on "interrupt bundling" and "interrupt loss tolerant b channel handling" are somewhat outdated with regard to bristuff 0.2.0-RC7h and later—the behaviour of the patched driver is unchanged, though. Again, see below for the description of the changes introduced by bristuff 0.2.0-RC7h.

Slave Mode¶

What is it good for?

Before I describe in detail how it works and how to make use of it, I will try to explain what the "slave mode" is good for, so you can decide whether you need to understand the rest of it.

Every HFC-S PCI A card has a 12.288 MHz crystal on it from which, amongst other things, the 8 kHz frame clock is derived. This is the clock which clocks the sending out of frames on the ISDN interface as well as the generation of the interrupt which is used by the zaphfc driver for zaptel timing. Now when you have two cards in an asterisk box, each clocked by an independent crystal, the slight difference in the two crystals' frequency over time will cause the data buffers in between them (that is, the cards' FIFOs, the driver's buffers, and, probably the biggest chunk, asterisk's jitter buffer) to shrink or to grow and thus to finally run empty or to overrun, respectively, depending on whether transmission goes from the slower to the faster card or the other way around. Either way, the transmission is disturbed, be it by dropped data or by insertion of invented data. On voice connections you will hear a little crackling when that happens, which usually won't hurt. Fax machines and modems, though, generally don't like jumps in the time domain (which kindof is what is happening), so if you want to use multiple HFC-S PCI A cards for switching modem or fax connections through them reliably, this might be a problem. A problem which can be solved, though, using this patch and a soldering iron. If you think this doesn't apply to you, you might want to skip the following section. Also, you don't have to worry if all your lines are PSTN connections: Only HFC-S PCI A cards in NT mode can be used in slave mode. Cards in TE mode always synchronize to the clock of the connected ISDN line—and any number of PSTN ISDN lines should be in sync anyway, causing any cards in TE mode connected to them to be in sync, too.
How does it work?

When you look at the datasheet of cologne chip's HFC-S PCI A, you will find that it has a so-called "PCM bus interface" (or "GCI/IOM2 bus interface" as it is called in the datasheet). This is basically a serial synchronous bus that can be used for directly transferring b channel data between components connected to the bus without any cpu involved. One of the side effects of its functioning as a synchronous bus is that the frame clock of all connected devices runs completely synchronously—which is exactly what we need. This is achieved by having one master on the bus that drives the clock lines (there are two clock lines, called C4 and F0, see below in the pinout diagram) with its own local clock and a number of slaves that synchronize their local clock to these clock lines by the means of a PLL. The patch just makes use of this capability of the chip to synchronize its own frame clock to an external source (usually another chip of the same kind)—no data is transferred over the PCM bus, though; it's only the clock lines that are used for syncing!

Now, if you want to make use of this capability, apart from applying the patch, you have to do some soldering. Below you see the pinout diagram of the chip as it can be found on page eight of the chip's datasheet. You have to connect the pins #55, #54 and #52 of all slaves and the master in parallel. Usually, you will use one of the cards running in TE mode and connected to the PSTN as the master and connect all the cards running in NT mode to it as slaves, so that the PSTN clock is propagated to the NT side of your asterisk box. I successfully used 0.1 mm diameter magnet wire for connecting the chip's pins to a connector glued to the card that allows the cards to be connected easily using a short piece of ribbon cable with matching connectors. Anyway, never forget that whatever you do, you do it at your own risk! If you don't have any soldering experience, you might easily damage the card and maybe even more! pinout of the HFC-S PCI A chip

Interrupt Bundling¶

The original driver activates the 8 kHz timing interrupt of every card that is found in the system, causing quite some cpu load if there is more than one card. The patched version only uses the timing interrupt of one card, no matter how many cards there are in the system, thus greatly reducing the idle cpu load if there are several cards in the system. The interrupt loss tolerant b channel handling described below should make this work reliably even when cards are not running synchronously. I haven't tried this myself, but theoretically, there should be no problem.
Interrupt Loss Tolerant B Channel Handling

The original zaphfc driver has serious problems with interrupt sharing and with any concurrent load on the system in general. When using the original driver, each card in the system generates 8000 interrupts per second at least. Whenever one interrupt is not serviced by the driver before the next one occurs, one interrupt will be lost—and that happens easily, especially when you have more than one card, a not-so-new computer, some more things for the computer to do, or some combination of it. Except that the patch makes only one card generate the 8000 interrupts per second and thereby already greatly reduces the problems with several cards in one computer (see above), interrupts, of course, still can get lost. What really makes the difference here is how this is handled by the driver. With the original driver, lost interrupts "accumulate": If only a single interrupt is lost every five minutes, after one hour the send FIFO is shrunk by 12 frames and the receive FIFO is grown by 12 frames, sooner or later causing over- and underruns. The patched driver uses the card's hardware counters to re-synchronize in such a case, so that a problem only can occur if more than eight interrupts are lost in series, which normally should not happen.

The only price you have to pay for this increased immunity against high system load, except for having to apply an additional patch, is a little higher latency: The patch increases the maximum latency of the outbound channels by 1 ms to a total of 2 ms and the maximum latency of the inbound channels by 2 ms to a total of 3 ms. I strongly doubt, though, that this will cause any problems in any case whatsoever.
What about RTAI?

Bristuff 0.2.0-RC7h introduced the use of RTAI for b channel timing purposes. This also greatly reduces the CPU IRQ load since an RTAI timer of only 1 kHz is used, instead of the card's 8 kHz clock, plus only a single such timer is used for all cards, much like this patch uses only the clock of a single card for servicing all cards in the system.

However, the way the RTAI code in the original bristuff zaphfc driver is constructed causes some problems: It potentially leads to buffer over- and underruns. RTAI is clocked by some computer-internal clock source that runs completely unsynchronized to the card's clock and that thus probably will deviate from it, depending on the clock source's manufacturing tolerance as well as its temperature and other environmental factors. Now, when this unsynchronized clock causes the card's FIFOs to be filled and emptied by the driver at the computer's side at a rate that is different from the rate at which the card's logic is filling and emptying them at the line interface side, that will cause the FIFOs to run empty and full, respectively, at times.

This problem in principle could be worked around—however, I don't see any major advantages inherent in the use of RTAI over the way the patch currently works, as CPU utilisation is pretty low with recent versions anyway. Considering that the use of RTAI also makes the installation more complicated (needs an RTAI enabled kernel as opposed to the driver patched with this patch which should work with any vanilla 2.4 or 2.6 series kernel), that's why I'll stay away from RTAI for now—this patch removes all RTAI code from the zaphfc driver!

Experience¶

Now, after I have explained in detail how things should work in theory, let me tell you how well things prove in reality ;-)

I'm currently using the patched driver on a Pentium 3 with 500 MHz and four HFC-S PCI A cards, one of them sharing its interrupt with the on-board 3Com NIC. Two of the cards are connected in TE mode to the PSTN, two are connected in NT mode to a legacy Auerswald PBX. One of the PSTN-side cards provides the frame clock that the cards interfacing the PBX synchronize to. The machine is running Linux kernel version 2.4.27.

Flood pinging the machine through the 100 Mb/s NIC such that ssh responsiveness is rather bad while at the same time running

   1. while :;do sync;done (optionally with a sleep 1 in between)
   2. cat /dev/zero > foobar
   3. find /

for a few minutes did not cause any noticable problems. Neither were there any buffer over- or underruns reported in the system log nor was there any distortion audible on the telephone connection going on at the same time. With the original driver the connection was crackling all the time when doing this and you had quite a bit to read in the system log afterwards.

Also, idle cpu usage went down from 25 % with the original driver to 7 % with the patched version (update: version 3 is below 1 % now, though that's probably not a matter of the zaphfc driver or this patch). So, all in all, theory seems to work acceptably in practice.

As far as stability is concerned: So far, I had no crashes caused by this driver in about one week of continuous operation nor any other problems that are likely to be caused by the patch (only asterisk segfaulting in libpri and such, which happened with the original driver, too). However, this is only one test environment, so I don't know how it will behave on other computers, other kernels, with a different number of cards, …—your feedback therefore is very welcome.

Installation/Configuration¶

Installation is simple: Install as usual, just apply the patch before compiling the zaphfc driver. Something along this line should do the trick when you are in the source directory:

$ zcat /path/to/zaphfc_0.3.0-PRE-1f_florz-11.diff.gz | patch -p1

When it comes to loading the driver, you now have a few more choices. Namely the module options sync_slave and timer_card.
Module Option sync_slave

sync_slave works the same way modes does: It's a bitmask in which a 1 bit means the card should be configured for slave mode while 0 means that it should be in master mode. The default is for all cards to be in master mode, which is backward compatible with the original driver. If you don't understand what this is for, just leave it unspecified.

Module Option timer_card¶

The timer_card option specifies which of the cards in the system is to generate the 8 kHz interrupt that is used for timing. The cards are numbered starting at zero—you have to specify the number of the card, this is not a bitmask like the other options! By default, the first card is used. It's probably best to use a card which does not share interrupts with any other devices. In order to find out which card uses which interrupt, just load the driver once and have a look at the kernel log messages, then unload it and load it again with the timer_card setting of your choice.

For reliability reasons, I'd also choose a card in master mode since it does not depend on any external connections in order to be able to generate the interrupt, so in the event that a wire breaks or something, the smallest number of cards possible stops working.
Module Loading Example

In my setup described above, the driver is loaded like this:

    # insmod zaphfc.o modes=12 sync_slave=12 timer_card=2

That means that the first and the second card are running in TE and master mode while the third and the fourth card are running in NT and slave mode. The third card generates the 8 kHz interrupt.
Download

The original driver can be found in kapejod's bristuff package at http://www.junghanns.net/downloads/. In the following table, each version of the patch is listed together with the bristuff versions it has been tested with.

Please note that the zaphfc driver is not changed in every release of the bristuff package, which means that the patch doesn't need to be changed either—and I won't usually release a new version when the only change would be in the version number ;-) Thus, a version of the patch that was originally created for some older version of bristuff might still be applicable to newer bristuff versions. If the patch program doesn't complain, that's usually a good sign; if you want to be sure, just do a diff between the version of zaphfc the patch was originally created for and the version you intend to patch and see whether there are any differences. If not, just go ahead. Should any differences show up, though, and there is no newer patch available yet, you should consider dropping me a line.

vzaphfc¶

vzaphfc is a zaptel driver for Cologne Chips HFC-s - based single port ISDN PCI adapters. To build it you also need a bristuffed zaptel and you need a bristuffed asterisk to use it.

The maintainer of vzaphfc is Jens Wilke. See the README file for the details.

For some reason, there is no homepage for vzaphfc and it is only available from the Debian pkg-voip subversion repository. Thus I automted here the procedure of dumping it from the the the subversion. Note that no automated dumps are made. I may or may not generate newer tarballs here when there are newer commits to the vzaphfc code.

http://lists.alioth.debian.org/pipermail/pkg-voip-maintainers/2007-August/009069.html

Tzafrir Cohen wrote:

>> Basically we need to get a list 
>> of all ISDN adapters (I guess the files in upstream's kernel package are 
>> going to be helpful there), and ask whether they want to use them with 
>> mISDN or I4L, then set up udev rules for that. 

> When exactly will this be done? modules are loaded at boot.

At install time.

> Note that qozap, zaphfc and vzaphfc from zaptel-modules also combat with 
> misdn in the fight to be the default for the device IDs.

Okay, so it may be a good idea to have some sort of framework where 
these could be integrated as well.

I think it would be a common configuration to have multiple HFC based      <<<<<<<<<
cards in one box and run the TE side with zaphfc and the NT side with      <<<<<<<<<
mISDN (that way, you get a zaptel timer for MoH and Meetme).

>> 4. Work with/against upstream to get a real init system working, which 
>> means teaching the kernel to defer loading the l1-3 drivers (this will, 
>> guess what, change the kernel<->userspace API/ABI again, but we can 
>> probably catch that in libmISDN). That was originally my plan for the 
>> CCCamp, but I was sick in bed during that time.

> What do you mean here?

The kernel immediately instantiates a device stack when the hardware 
module is loaded. This needs to be deferred until the device is actually 
used for the first time, because there is no hard dependency from the 
hardware modules to the l1-3 modules (they do not need any modules), and 
the stack setup even succeeds if these modules are not loaded, but we 
end up with a few NULL pointers in kernelspace that are dereferenced on 
behalf of asterisk later.

> Do you mean that every non-blacklisted module will get the hardware
> before your module?

That is undefined, but pretty likely, given that we have to load 5 
modules while i4l only loads two or three AFAIK. Some testing without 
the additional dependencies showed that it was a matter of luck 
actually, mISDN won 1/3 of the times, i4l 2/3; I expect that to shift in 
favour of I4L.