Improving Open AMBE for D-Star

 

Open AMBE | DroidStar

 

Overview:

The AMBE vocoder (speech coder) problem started when D-Star was developed and introduced to ham radio in the late 90's early 2000's. It's since been compounded by DMR, Yaesu Fusion, and other digital radios. The proposed solution has been to replace this with an open source codec.   However, none of these digital modes have a means to specify or differentiate the audio codec should they ever want to move away from AMBE.   And there still isn't much of a working implementation of Codec 2 for VHF/UHF radios.  There is a working implementation for HF. Codec2/FreeDV is a good conception, but it hasn't come to fruition in the advent of the AMBE patents expiring and other digital radios also employing AMBE and flooding the market.  Besides we still don't have radios with open firmware where we could load our own apps or alterative codec.

Keen observers have noticed a number of good radio initiatives that sadly haven't come to be.  The; Connect Systems CS7000, Wireless Holdings DV4 Mobile, the HT of the Future (aka: Algoram, Katena, Whitebox), the Newradio initiative in Germany/Austria, RU3ANQ SDR, and so on.  There is a new (2019) initiative, the M17 Project, but its too early to tell if that will reach production.  

Manufacturers are a fairly conservative bunch. They don't want to invest in anything unless they know it's going to sell.  We all understand.  Fortunately, much of the innovation now is purely in the form of software, which is much easier to mass-produce than hardware. So all you need the manufacturers for is to make general purpose SDR hardware, which is an easier sell than some new special mode.  It's also important for leaders to recognize the works of the dedicated software developers.  And when the hardware manufacturers see the potential from their work, to work out an agreement so the software folks receive more than just a thank you.

For many hams these days the AMBE concern is not well understood.  First you have to take a larger look at the point of amateur radio.  To learn about radio, propagation, and the electromagnetic spectrum in general.  To understand how it works, and maybe even build or modify your own equipment.  In order to learn we must be able to inspect; to tinker, or at the very least have access to a specification we can build from.  Closed source applications such as; Dplus, Hamvoip, Peanut, and others don't contribute to the advancement of the radio art because they retard people from learning the technical operations.

So while D-Star was interesting as it was a new thing to play and learn about when it came about, there was this "black box" aspect where you just buy and are a passive consumer.  This didn't sit well with many. Why even bother with a hobby of sharing and learning, if there’s a big wall saying  “this far, no further!”, that you are not permitted to create your own radio for. Or permitted to understand.

The "HT of the Future" talk that Bruce Perens also gave some time ago, did also stress the need to have open radio firmware.  Let's take a quick look at why this would be good for the hobby.

The Linksys WRT54G WiFi router of the early 2000's was a good example of the good that can come from open firmware/open source.  The history here was the original factory firmware was discovered to be based on Linux components, which are covered by the GPL.  This required the manufacturers to release the source code.  With the code in hand, developers learned exactly how to talk to the hardware inside and how to code any features the hardware could support. It has spawned a handful of open source firmware projects for the WRT54G that extend its capabilities, and reliability, far beyond what is expected from a cheap consumer-grade router. In short, due to open source, one can load a third party firmware on the router and give a $60 consumer home-grade router all the functionality of a $600 Cisco professional router. 

Pretty much Every other consumer electronics item is updateable to some extent. Ham radio has a heck of a lot of talented software coders as evident by the 1000's of projects on github .. It's a shame, and down right stupid that we cannot have hardware to be able load applications on,
rather than the antique approach of shelling out hundreds of dollars for a new piece of hardware.

There have only been a few radio firmware reverse engineering efforts.  The most well known is the MD380 project (2015) by Travis, KK4VCZ and his group.  The hobby can use a lot more of this and a lot more people like Travis.   We haven't yet figured out how to re-write a radio's firmware to create that elusive digital radio that can do more than one digital mode.  But that day may still come.  Software Defined Radio was likely a foreign concept to many 20 plus years ago when this problem was first brought to our awareness. The USRP, HackRF, HamShield, RTL-SDR, are known to many now, and having to have a hardware dongle to do the speech coding with those is illogical.  

The second most well known firmware reverse engineering effort is the OpenGD77 (2019) is opensource firmware for Radioddity GD-77, Baofeng RD5R, DM1801 and DM860 radio, developed by Roger VK3KYY, in Melbourne.  This has since inspired some Italian radio amateurs to take it further with their project called OpenRTX.  

(Interesting to note that while AMBE software implementations are often times rejected by arm chair lawyer types, oddly enough don't hear as much push back on the underlying radio firmware reverse engineering projects that have lead to new firmware developments such as the M17 OpenRTX version.  In terms of a "clean room" approach open AMBE was figured out from printed specifications, the radio firmware projects relied on reverse engineering the proprietary and encrypted radio firmware.)

The Need:

At this point, the AMBE used in D-Star (3600x2400 bit/s) is out of patent. Since there was never a published specification for it like its other variants, hams have thus far not been able to create a software version with similar performance.  (Good sounding software AMBE does exist for other modes)

Software AMBE is extremely handy for folks running cloud based conference bridges like DV Switch, or XLX Reflectors that allow hams to cross communicate between digital modes, promoting interoperability and cutting down on digital fragmentation .  It also can be used for internet connected apps (DroidStar) that connect to the RF gateways without need for hardware (MMDVM hotspots & HT's).

The original reverse engineering thread: https://forums.radioreference.com/threads/decoding-d-star-any-success.215282/

Its been said elsewhere that the suspected key to getting it better output is with the quantization tables:

// decode V/UV parameters
// load b1 from ambe_d 
//TODO: use correct table (i.e. 0x0000 0x0005 0x0050 0x0055 etc)

This likely can't happen at the very least unless the coefficients for encoding the specific D-Star variant of AMBE  are exposed in a software implementation somewhere like the one in the MD380 firmware.  The good news is it appears Travis and his team have begun working on that.  (reach out to BitBangingBytes, and or Abalamahalamatandra if you'd like to help with this)

In an ideal situation, the D-Star spec would have required publication of this like the APCO specifications did.  After the fact, I don't foresee any reason for the patent holder to release such details.  Another ideal scenario is if the FCC rules actually emphasized or required digital transparency, then not only would this enable those who want to learn and code/build, it would likely force working source code for AMBE to be published tomorrow.

It seems sad that various national amateur radio organizations + have not stepped up to support open source efforts.  So until then, please get a hold of me if you have the skills and desire to work on this for the benefit of the ham radio community and or join the OpenDV group.  I will ensure you are recognized for your efforts. - Steve, KB9MWR


Arm Chair Lawyer Notes:  

There are a number of people in the ham community that feel the need to play the part of an "arm chair lawyer."  I see this need to discourage from everything from tower climbing, FCC rules interpretation, to areas like this, concerning software patents.  Bruce Perens' who brought the potential patent issues years back to the ham communities awareness has actually created a negative stigma.  People (will) think these patents are still in effect well beyond their expiration.  Bruce's comments were well intended and more targeted for actual/potential software developers.  The fact is there is intellectual property cases that is in the courts for many end consumer electronic devices, mostly thanks to China, and patent trolling/hoarding. Outside of the ham word, end users likely don't concern themselves or give this any thought.  

--From Bruce Perens's AMBE DCC talk--

You've Already Paid Once...

Multi Band Excitation (MBE) was initially conceived in the mid-1980’s, at the Massachusetts Institute of Technology (M.I.T.)  Those speech models were developed by Griffin and Lim. The AMBE technology was developed under a United States Government grant though the Rome Air Development Center at Griffiss Air Force Base of the United States Air Force.  You paid for this technology with your taxes.  Publicly funded research should be open.

Doctrine of Laches: If you have a patent, and you know of an infringement but you wait for the market to get larger before you bring suit, that is reason your lawsuit to be denied. The usual duration for a Laches case is 5 years, although Laches cases have been win with as little a 1- year delay

Key patents were filed in April 2003, and could potentially be invalidated by the prior art of David Rowe, who wrote his thesis in 1997, and some of his work is from earlier than that.

Also, DVSI made use of the AMBE codec in commerce before some of their patent applications, potentially invalidating their own patents. There's one ham and intellectual property expert who feels that due to the date of first use in commerce, the important patent claims will expire in 2016.

Bruce Perens supposedly has a report from a patent analyst. It would be nice if he could share that.  There were about more than a dozen patents related to this stuff. However all but a one are expired, and that remaining runs till 2028. It's nearly impossible to tell what patents pertain to what, unless you are some sort of technical-codec lawyer with a high IQ.  But the short version is when it comes to patents, the rule of thumb is; Expiry: the longer of 17 years from issue date or 20 years from filing date.  All the patents for the version of AMBE used in D-Star expired back in 2017 per Bruce Perens (source).  The remaining patent concerns AMBE+2 used in DMR, Fusion, and NXDN. It was applied for in 2003, but was tardily granted in 2013, thus making that remaining patent for those modes expire in 2028.

Google has a patent search function, the remaining patent can be read here: https://patents.google.com/patent/US8359197B2/  Indeed the USPTO adjusted the validity of this patent until 2028, because they failed to examine it quickly enough. The adjustment not possible in Europe or Canada, where the patent will expire in 2024. Apparently, the process was not patented in China and the patent was refused in Japan.  There is still even the question if this patent applies as its a question if so-called "AMBE+2" is the same as the halfrate vocoder that is documented in the p25 specs (the former is apparently used in dmr, ysf, and nxdn, and p25p2)... There are "features" in AMBE+2 that are not publicly documented...

The open source version of AMBE for D-Star, DMR, Fusion have existed since May 2010 as mbelib on github with no DMCA takedowns. All of them with the exception of D-Star were figured out from public TIA documents. (The APCO spec required a technical publication of the vocoder details). Some smart folks did there best a while later in 2013 to prove a point through a reverse engineering and guess-and-test, as the D-Star codec is completely undocumented.  But sadly it still sounds like it could use work when compared against the audio processed by the chip.

"If anybody wants to continue the research / work, I suggest you look at the osmocom GMR code that Sylvain  Munaut worked on. Those phones use a similar codec - I believe with longer frames for the satellite latency. Initially he used the mbelib code, enhanced it for things like tone support, but he later rewrote the synthesis code completely. See OsmocomGMR for his  presentations and source code." (per a author)

DSD+ or dsdplus - Is a closed source Windows fork of DSD that came about in December 2013. It caused the original DSD author to give up his development. (These DSD+ guys have gall, as you can pay for fast lane access to new features.)

If you really have to be arm chair lawyer like, here is the AMBE talk: K6BP "AMBE Exposed" from the 2014 DCC

I am not a lawyer, but I don't see anyone bothering you (haven't heard of it thus far) unless you are making a profit selling something that has makeshift AMBE under the hood. I even think Bruce eludes to this when he mentions doctrine of latches, etc.  However making use of mbelib /open source AMBE for anything other than casual home receiving is problematic until all the applicable patents are expired.  Obviously for most U.N. member states; non-commercial/research usage of patented technology is covered by exceptions on the definition of "patent infringement." Actual practice may differ. Ref

And on that note; In May 2018 Chinese AMBE (Team6160) started to appear on the market.  It's a WT3080 AMBE 3000 Compatible Device (it does use the 460800 baud rate). Apparently, this chip is loaded with the bootleg code that is used in some Chinese DMR radios.   (see Nanjing Wutong Microelectronics Center aka: http://www.indusic.com)


For Developers:

https://github.com/szechyjs/mbelib/commit/2656ec2202e15468f92dca0c226919166928c52e - This is the initial DSTAR AMBE implementation.  
https://github.com/nostar/dudestar
- DudeStar / DroidStar
https://github.com/nostar/DroidStar - DudeStar / DroidStar
https://github.com/f4exb/dsdcc - dsd rewrite - C++ library with a single decoder object
https://github.com/LX3JL/xlxd
https://github.com/szechyjs/dsd
https://github.com/szechyjs/mbelib (it's been up on GitHub for 10+ years with no take downs)
https://github.com/DSheirer/sdrtrunk/tree/master/src/main/java/io/github/dsheirer/jmbe

https://github.com/LouisErigHerve/dsd
https://github.com/lwvmobile/dsd-fme

http://git.osmocom.org/op25/tree/op25/gr-op25_repeater/lib - Op25 has encode and decode support for AMBE (D-Star, DMR and YSF) and IMBE (P25)
https://github.com/balint256/op25/tree/master/op25/gr-op25_repeater/lib/imbe_vocoder - Pavel's IMBE Encoder/Decoder Fixed-Point implementation
https://github.com/pizzabeerllc/vocoder_plugin
http://git.osmocom.org/osmo-gmr/tree/src/codec 
https://github.com/on1arf/voice-ann
https://github.com/dl1bff/ircDDB-mheard/blob/master/ircDDB-mheard.c
- Line 383 - handles the bit interleaving and FEC processing
https://github.com/travisgoodspeed/md380tools/  - The MD380 Emulator (capable of AMBE encoding and decoding)
https://github.com/kawks/dstar-n800
https://github.com/g4klx/AMBETools

TIA-102.BABA - TIA Standard - Project 25 Vocoder Description
TIA-102.BABA-1 - APCO Project 25 Half-Rate Vocoder Addendum
TIA-102.BABC - TIA/EIA Standard - Project 25 - Vocoder Reference Test

GMR-1 Speech Codec
DV Dongle Technical Reference Manual
Decoding AMBE+2 in MD380 Firmware in Linux
AMBE_Exposed

Vocoder plugins

http://dudestar.gw8szl.co.uk/

md380tools - support group

Radio Firmware Reverse Engineering:

MD-380
OpenGD77
Kenwood TH-D74
Yaesu FT-2DR
Yaesu_FT3DR

Yaesu FT-70D

Xiegu X6100 
Vero VR-N7500
Quansheng UV K5 (8)/K6 

 

Comparison Samples:

The following are created from the supplied DSDCCX discriminator output samples. The samples were taken at the output of the discriminator of the radio. (S16LE 48 kS/s single channel (mono) samples.)  The sample for each mode was passed thru mbelib, and a separate pass thru AMBE dongle to create the following side by side comparison files.

Syntax used:
sox -t s16 -r 48k -c 1 $INPUTSAMPLE -t s16 -r 48k -c 1 - | dsdccx -T3 -i - -fa -o - | sox -q -t s16 -r 8k -c 1 - $OUTPUT_mbelib.wav
sox -t s16 -r 48k -c 1 $INPUTSAMPLE -t s16 -r 48k -c 1 - | dsdccx -D /dev/ttyUSB0 -T3 -i - -fa -o - | sox -q -t s16 -r 8k -c 1 - $OUTPUT_dongle.wav
#-fa (DMR), -fy (YSF), -fd (D-Star)

dmr_it_8_dongle.wav
dmr_it_8_mbelib.wav

dstar_f1zil_1_dongle.wav
dstar_f1zil_1_mbelib.wav
dstar_f1zil_2_dongle.wav
dstar_f1zil_2_mbelib.wav

ysf_f5zoo_dongle.wav
ysf_f5zoo_mbelib.wav


Notes and Misc:

Also worth mentioning is the XLX multiprotocol gateway reflector that was started in 2016 by Luc, LX1IQ.  This was the first (2016) bridged multiprotocol reflector conception.   This brought about some issues of needing physical access to to the server for the AMBE dongle. As well as the single stream limitation using the available hardware AMBE dongles. In the summer of 2017 Florian DF2ET designed a homebrew AMBE-3003 PCB to provide 3 channels. https://github.com/phl0/AMBE3003USB   An option for a USB-3006 device was made by LX1IQ and LX3JL by Winter 2017. https://github.com/LX3JL/usb-3006  This prompted a kickstarter by NW Digital (XC-3006): https://www.kickstarter.com/projects/985312845/6-channel-usb-ambe-transcoder-for-digital-voice-sy   Which provided a small quantity of them by Spring 2018.

A condensed version of this AMBE call for help may be found in the Autumn 2020 issue (#147) of TAPR’s quarterly newsletter

In  Nov 2020, I had communications with the ARRL on the topic of bringing the ham populous up to date on the State of Amateur Digital Voice.  And that they should be pointing out there is ample experimentation and building in the open source software communities. And how thanks to these folks Amateur Radio is in fact vibrant and relevant. And that story just isn’t being told.  Lets see if they ever address this via their own poduim (QST, social media etc).. Also brought up was reinstating the "Future Systems Committee", akin to the RSGBs "Emerging Technology Coordination Committee," to help foster communications between manufactures and the community etc,

If you are not developer you can still help!

Hams come from all walks of life, each one has certain skills.  There is a great community in ham radio willing to help each other and that is what makes is great.   By being vocal, standing your ground by only patronizing those in line with your ideals, and committing yourself to learning, anyone in the hobby can help.  Perhaps a coordinated approach of community members taking turns to voice their concerns at Dayton, on the manufacturers social media platforms, as well as the so called leaders platforms, is what needs to be done to prompt change?