RTCM Client

The Radio Thin Client Module (RTCM) is commercially available hardware for interfacing radios to an AllStarLink computer.

The Micro-Node RTCM and VOTER interfaces are typically used with AllStar in voting/simulcast applications. They MAY be used for ANY repeater interface application, through the chan_voter channel driver. The VOTER is the original through-hole board designed by Jim Dixon for this application. It is open-source, and the relevant Gerber files and BoM to build it are available. The Micro-Node Radio Thin Client Module (RTCM) is the commercial version of the VOTER. It uses surface mount parts (SMT), but is functionally equivalent to the original VOTER. In general, the two terms (RTCM/VOTER) are used interchangeably, as they operate the same, and use the same firmware (mostly, see below).

This page will highlight some of the operational concerns, quirks, bugs, and other items of interest that relate to these interfaces. Much of the information has been gleaned off the AllStar mail list, comes from personal experience, or comes from notes in the firmware source code.

Factory Reset

A factory reset can be performed by setting SW2-1 DOWN and then applying power to the RTCM. Wait for the green LED to stop blinking, power off and set SW2-1 back UP.

Motorola Quantar

Some things to consider:

• Install JP1 on the Quantar's RTCM. The squelch should calibrate at around 4 blinks rather than the 12 blinks or so without JP1.
• Be sure you've done the diode and squelch calibration with the actual attached radio (no antenna).
• The Quantar firmware should be 20.14.48 as later versions have better noise output.
• Try the "Chuck Squelch" RTCM version.
• Each RTCM should have 3 to 5 turns past threshold to prevent the squelch form being too loose. Somewhere around the 350 level seems about right. Chuck Squelch seems to need fewer turns but YMMV.
• Don't "and" CTCSS with squelch. That may override the RTCM's squelch detection. Compare with CTCSS on and off to see effect if any.

Also check the Quantar RTCM page for detailed interfacing and configuration information.

RTCM/VOTER LED's

RX LED on the RTCM/VOTER will flash (same rate as ACT LED) if you have External CTCSS enabled, and the received signal has the wrong (or no) valid PL.

Network Information

RTCM/VOTER Bandwidth

Assuming ulaw... The RTCM will require approximately 80kbps of bandwidth for continual usage.

Network Quality

Your IP Network quality is important. You may wish to add a rule in your routers to prioritize traffic on UDP port 667 so that RTCM audio is given priority over other traffic. Also see the #Debug Options below for notes on how to tag your packets with ToS.

RX/TX Buffers are NOT Both Millisecond Values

You might assume an RX buffer (in voter.conf) of 120ms would be equivalent of a TX buffer (in the RTCM) of 120ms. That is not the case.

The TX buffer is a number of 125 microsecond intervals, where the RX buffer is in milliseconds.

If you follow the buffer setting instructions, you should be fine, in most cases.

No GPS/Mixed Mode

You do not need a timing source to use an RTCM if you don’t want to vote. That’s called Mix Mode.

Having the PPS Polarity set to 2 tells the RTCM you do not have a GPS. That forces the RTCM to become a mix (non-voting) client. Turn it on, save it, and restart. A simple voter.conf would look like this:

[general]
port = 667
password = BLAH

[1000]
Site1 = pswrd1,master,transmit
Site2 = pswrd2,transmit

Mixed Client Error

"I am getting this error in Asterisk":

WARNING[2368]: chan_voter.c:4511 voter_reader: Voter client master timing source mobile1 attempting to authenticate as mix client!!

A mixed client error means the voter.conf file is expecting an RTCM to try and connect with a GPS IP packet (ie. has master in it in voter.conf), but the RTCM isn't sending a GPS IP packet. So its a mismatch between voter.conf and Option 10 in the RTCM.

If you want to use a mix client (non-voted), make sure that receiver's configuration line in voter.conf does not have the master option set.

Mix Clients with Voted Client Issues

Situation... "I have a private node with 6 voted receivers using RTCMs. I'd like to add a 7th RTCM to this node that is always mixed in rather than voted. I'm able to make this RTCM work as a 7th voted receiver with no problem. Everything I've read seems to indicate that if I change GPS PPS polarity to "none" this will achieve my desired results, however I am unable to get audio out of my transmit RTCM from this 7th site. It does change color to cyan in Allmon indicating it is non voted input but I do not get any indication or audio when that units COR goes active."

"Setting voter debug level 3 in Asterisk and I'm seeing the following message repeatedly scrolling by in a blur when the mix client detects COR (COR is active). Sequence numbers are continually incrementing by 1. I'm running software version 1.47 on all my clients:"

mix client (Mulaw) my_client index:0 their seq:629 our seq:629
mix client my_client outa bounds, resetting!!

If you have a similar situation to the above... check your buflen in voter.conf. Make sure it is at >=160 and see if that fixes it.

Ubiquity ToS

See https://help.ubnt.com/hc/en-us/articles/205231750-airMAX-How-is-QoS-and-prioritization-handled-by-airMAX-

So, if we set ToS/DSCP in the header to C0, then Ubiquity (and other gear watching ToS) should prioritize the packets. This sets the DSCP to 110 000 aka 48. UBNT shows this as 802.1p Class 6 (Voice <10mS latency). Other sources show this as a Network Control TOS.

There is an option in voter.conf to turn this on (utos=y). However, this only controls packets being sent from Asterisk TO the RTCM.

If you want to tag packets from the RTCM TO Asterisk, you need to set the RTCM debug option level to 16 (see #Debug Options for how this works).

At some point we should probably change the default behaviour of the RTCM firmware to mark the packets and use the debug setting to disable ToS. That change (if you wanted to compile your own firmware) would be:

Change line 90 in IP.c

From:
#define IP_SERVICE         ((AppConfig.DebugLevel & 16) ? 0xc0 : (IP_SERVICE_ROUTINE | IP_SERVICE_N_DELAY))

To:
#define IP_SERVICE         ((AppConfig.DebugLevel & 16) ? (IP_SERVICE_ROUTINE | IP_SERVICE_N_DELAY) : 0xc0)

This changes the conditional expression from if the debug level is 16 to mark the packets with DSCP 48 to if the debug level is 16, mark the packets routine (DSCP 0).

RTCM Simulcasting

The RTCM/VOTER boards do support simulcasting, however, there are a bunch of quirks that one needs to be aware of.

Radio Hardware

For best results, you should use all identical RF equipment between your voting RX sites and simulcast TX sites. If you don't, you can end up with strange audio artifacts when different receivers are used, and other strange audio issues when different transmitters are used. The most important thing when setting your levels is that ALL the RX and TX levels match from radio to radio. The best way this is achieved is using a "master" radio and reference from that - all radios might be identical models, but may not have identical audio characteristics so check those levels! Also check CTCSS and audio levels separately to make sure they all match.

9.6MHz Oscillator

If you want reliable simulcast with an acceptable level of overlap warble, you MUST inject a GPS-disciplined or OXCO 9.6MHz signal to the clock of the RTCM's, in lieu of their stock internal crystal.

Otherwise, the inherent internal clock jitter of the RTCM's will cause CTCSS warble, etc. in areas where your transmitters overlap. The 1 PPS feed that you normally feed the RTCM's is acceptable for voting, but naturally, a typical crystal-driven clock will slightly drift in between the pulses every second due to inherent jitter, etc., and that ain't to good for situations like simulcast which demand sample-accurate performance.

Here are some comments from Joe, KC2IRV, on the subject. He has also created his own wiki, specific to simulcast issues, you can find it at: http://rtcmsimulcast.wikifoundry.com/

I felt I needed to let those on this mailing list know my findings with simulcasting with the RTCM units since I have received a great deal of help and information from people on this list.

For the past week I have had a two (2) site simulcast system up and running on UHF using the RTCM units. I have done a great deal of testing and proving on the bench and it is so far working beautifully.

The one issue that was made apparent to me early on was that in order for the RTCM's to be suitable for simulcast, the processor inside it needed to be clocked to a more accurate source. This source is supposed to be the Programmable Clock Generator Module in order to produce a GPS locked 9.6 MHz reference for the processor. Unfortunately this unit is not currently available.

Knowing this, I started to go down the road of possibly designing my own, but stumbled upon a cheap, plentiful supply of Symmetricom 9.6 MHz sinewave OCXO's. I took a chance and ordered a few to experiment with to see if the RTCM would be able to use this in place of the supplied crystal for a clock. I found they did, with that I decided to modify the RTCM with an SMA connector just above the audio adjustment pots and use this OCXO. I trimmed the frequency on the units to match each other and used them to produce the processor clock.

After a week of testing, I have found the results in the overlap areas to be exactly as I would expect them on a public safety/commercial simulcast system. While in an overlap region where both transmitters are within 3dB of one another, to the ear you hear no audio phase wandering. Of course , this has only been a week of testing so I wouldn't call it case closed yet.

I will say that if this system remains this way after testing for 6 months to a year than using a 9.6 MHz OXCO is a viable alternative to achieve the needed precision and accuracy needed for the processor clock to make it suitable for simulcast.

Comments from Jim Dixon on the issue:

You have to use something that will take the precise 10MHz (or whatever it is) from the GPSDO and PLL (phase-locked) convert it to 9.6MHz (square wave at 3.3V for the CPU clock).

We had to use 9.6MHz mainly because of the DAC in the dsPIC instead of 10MHz. The options for various divide/clock radios in the part are RATHER limited. As I recall, there wasn't even a way of getting the necessary 16kb/s sample rate on the ADC from 10MHZ, either.

Micro-Node RTCM Clock Issue

James, KI0KN, had some strange-ness with some of his RTCM's when used for voting. As soon as he changed menu item 10 on the RTCM to either a (1) or a (0) instead of (2), he instantly got this on the RTCM console:

04/05/2016 18:44:13.660  Lost GPS Time synchronization
04/05/2016 18:44:13.660  Host Connection Lost (Pri) (10.16.1.240)

And it sits there forever and never re-establishes connection to the host. The issue was finally traced to a bad batch of 9.6MHz crystals that affected a small run of the RTCM's. The issue was eventually resolved by Micro-Node, but we'll document it here just for record keeping.

James comments to the list:

Well, after many of you offered your time and thoughts on my voter problem, Jim, WB6NIL, graciously donated a couple of hours of his time to remotely help me and he uncovered the problem.

All 5 of my RTCMs were purchased within the last 6 months, and I suspect all 5 have the same problem (will verify that today). The problem is that the microprocessor crystal is running too fast! There is 9.6Mhz crystal that drives the MPU, on the particular unit that Jim helped me diagnose, it's running 2.5 khz too fast.

There is a sanity check in the firmware that makes sure the correct number of samples were taken in the last second (it's supposed to be 8000) and fails if the sample is incorrectly sized (mine is taking 8003 samples). Jim helped adjust the code to be more tolerant of the sampling error and my whole system instantly worked.

I have since tested it with ALL my GPSs (they all work great!). So I now have everything working, including the voting.

I haven't contact Micro-Node about it yet. Jim told me that voting should work fine but this clock error would probably not be acceptable for Simulcast (something we have no plans on doing as of now). Since mine are deployed at mountain top sites, I'll probably pursue a crystal that is running at the correct frequency and use "standard" firmware so that if the decision is ever made to play with simulcast, I won't be right back in this same boat.

Thanks to all that helped out, and a HUGE thanks to Jim for taking the time to troubleshoot this.

BTW- This clock error will NOT effect the RTCM in mix client mode, it only affects it in voting or simulcast mode!

In voter.c, the stock line is:

if ((samplecnt >= 7999) && (samplecnt <= 8001))

it needs to be changed to something like:

if ((samplecnt >= 7999) && (samplecnt <= 8003 ))

That was just enough to make my first RTCM work. I've had a chance now to check a few more of my RTCMs on the bench this morning and the crystal frequencies are kind of all over. The next one I had on the bench was 5.2Khz fast. The above code was still not enough to fix it (as would be expected) so it looks like the quality control on whatever manufacturer that crystal is, isn't very good (or a low tolerance crystal is being used). I am going to pursue crystal replacement to get a highly accurate, stable, on-frequency crystal in there as my first choice. Changing the firmware 5 different times doesn't seem like the right answer.

Jim put that sanity check in there for a reason. Working around it may allow the device to work, but to me, it seems getting the hardware operating the way it was supposed to originally is the better answer. I will follow up with Micro-node and let you all know how it goes.

After following up with Micro-Node:

They were all off, some were off a LOT. I talked to Mark @ Micronode. He told me there were 25 units that made it out the door with a crystal from an unapproved source and that was quite likely the reason for the problem. He gladly sent me 5 new crystals to replace (he offered to replace them himself if I sent the units in, but I am comfortable doing it myself to save the time and postage).

I haven't received them in the mail yet to show that the problem is fixed, but I will post here when that happens. He was great to work with and had no issue getting things set straight. If you are having the same issue and your RTCMs were bought about the same time, I'd suggest you contact him and have him help you resolve the issue!

Duplex Mode 3

Does delayed repeat audio bug you? Tired of hearing a bit of yourself after you unkey? Wish you could talk full duplex? Hate the echo chamber effect? Then we have the answer for you.

Duplex Mode 3 in app_rpt allows for "in-cabinet repeat" (where the radio hardware provides repeat audio) and app_rpt adds the hang time, courtesy tones, linking - all the things apt_rpt does sans repeat audio. Therefore no repeat audio delay. Cool, eh? This duplex mode has been in app_rpt for a while. Problem has been how to implement it in the RTCM environment.

Duplex Mode 3 support in the RTCM provides in-cabinet repeat functionality. Repeat audio loops through the RTCM and has almost zero delay because it does not have to traverse the network. The delay is not quite zero but it's plenty short enough to eliminate all of the above mentioned annoyances.

Of course Duplex Mode 3 support can't be used with voting or simulcast. You also loose Touch Tone muting, Time Out Timer and Repeater Disable functions because the repeat path is not through app_rpt.

Setting Voter Buffers

Voter ping is useful for end-to-end network evaluation when ICMP ping is turned off and/or the RTCM is behind a firewall and is not ICMP reachable. It can also help with finding the correct RTCM and voter.conf buffer settings.

Voter Ping Usage

The voter ping asterisk CLI syntax is:

*CLI>voter ping nameOfClient [packetCount]

If packetCount is not specified 8 pings will be sent. Use packetCount of 0 to stop an in progress voter ping. Set packetCount to at least 100 when evaluating link quality.

The result will be similar to:

...
PING (nameOfClient): Packets tx: 100, rx: 100, oos: 0, Avg.: 26.710 ms
PING (nameOfClient):  Worst: 38 ms, Best: 22 ms, 100.0% Packets successfully received (0.0% loss)

The output above is self evident except for oos which is a count of out of sequence packets. Voter ping requires RTCM firmware 1.23 or newer and chan_voter 2013-08-04 or newer.

RX Buffer Size

Ping all the receiver sites and look for the worst response of the worst client. As a rough rule of thumb the buflen setting in voter.conf should be set to the worst response + 40ms or 120 whichever is greater. Using the above case the buflen should be set to 120 (38+40

TX Buffer Size

Ping all the transmitter sites and look for the worst response of the worst client. The RTCM TX Buffer Length should be set to (worst response + 40ms) * 8 or 480 whichever is greater. Using the above case the RTCM TX Buffer Length should be set to 624 ((38+40)*8).

Assumptions

• The minimum TX buffer size is 480 (60ms) and the minimum RX buffer is 120ms. These were derived by testing on a LAN segment with chan_voter 2013-08-04 and RTCM 1.26.
• The ping times are not round trip times but they are in fact round trip times. Therefore the worst response could (should?) be divided by 2. Ie RX buffer = 38/2+40=59 and TX buffer = (38/2 + 40) * 8 = 472. Minimums still apply.
• The internet path to and from the RTCM under test is symmetrical.
• The added 40ms pad is an estimate of buffer ingress and egress.

As always your milage may vary. Some trial and error may be required to find the optimum settings.

Setup

The RX buffer is set with buflen=120 in /etc/asterisk/voter.conf. The value is in milliseconds. The TX buffer is set in the RTCM with menu item 7. The value is in 125 microsecond increments. To match the size of the TX to the RX buffer, use the RX buffer * 8 to get the TX buffer size.