Sending and receiving webstreams with Spectrum Lab

This chapter describes how to send (transmit) or analyse (receive) web audio streams using Spectrum Lab.

Since 2011, this is possible without the AudioIO-plugin (DLL), because native support for Ogg / Vorbis is integrated in Spectrum Lab (client side, to receive or send audio).

Since 2014, the audio streaming client was complemented with a tiny built-in audio stream server which, at the time of this writing, could provide a compressed or non-compressed audio stream for a limited number of remote clients.

Contents

  1. Analysing web streams
    1. Notes on the URL format; Examples
  2. Sending streams to the web
    1. Configuration of the output stream
    2. Unattended operation
  3. Logging web streams (compressed audio data) as disk files
    1. Name templates to specify filenames with the current date and time
  4. VLF 'Natural Radio' audio streams
    1. Links to VLF Natural Radio audio streams
    2. Setting up a timestamped VLF 'Natural Radio' audio stream
  5. Operation as audio stream server
  6. Internals
    1. Supported stream formats
      1. Compressed
      2. Non-Compressed
    2. Supported protocols
    3. Stream Test Application

1. Analysing web streams (SL acting as client)

To open an internet audio stream for playback and analysis, select 'File' (in the main menu), Audio Files & Stream, Analyse and Play Stream / URL .

An input box opens with the previously visited stream URLs. Pick one from the list, or enter the complete URL (*) of the audio stream in the edit field.

To store the URL for the next session, set the checkmark 'save URL history'. If you are concerned about privacy, leave it unchecked.

Next, click "Ok". The program will try to establish a connection with the Internet (or, depending on the URL, with an audio stream server in your LAN).
This may take a second or two.

Optionally, the program can re-connect a stream when the connection broke down. To allow the network to recover after errors (DSL- or WLAN problems, etc), the program will wait for a few seconds before attempting a new connection.

1.2 Notes on the URL format

The URL must contain the complete address of the audio stream resource.

(*) Note on the URL:
In most cases, SL is able to resolve the stream URL if the given URL is not an Ogg resource, but an M3U redirector (aka "playlist").
But some websites respond with an error when the HTTP 'GET' request doesn't originate from a webbrowser, or doesn't carry a bunch of red tape (which SL doesn't support), or doesn't use HTTP Version 1.1 (SL only uses HTTP/1.0).
Examples :
  • rawtcp://78.46.38.217:4401    ( "VLF1" from Todmorden, on Paul Nicholson's Natural Radio server; with GPS-based timestamps)
  • http://78.46.38.217/vlf1.m3u    ( "VLF1" from Todmorden, on Paul Nicholson's Natural Radio server; HTTP stream without timestamps)
  • rawtcp://78.46.38.217:4406    ( "VLF6" from Spenge near Bielefeld, raw stream with GPS-based timestamps)
  • http://78.46.38.217/vlf6.m3u    ( "VLF6" from Spenge near Bielefeld, via Paul Nicholson's Natural Radio server; HTTP stream without timestamps)
  • http://78.46.38.217/vlf15.m3u    ( "VLF15" from Cumiana, NW Italy, via Paul Nicholson's Natural Radio server; HTTP stream without timestamps)
  • rawtcp://78.46.38.217:4416    ( stereo combination of VLF1 from Todmorden (L) and VLF6 from Spenge (R), with timestamps)
  • http://78.46.38.217/vlf16    ( same stereo combination without timestamps)
  • rawtcp://78.46.38.217:4427    ( stereo combination of VLF1 from Todmorden (L) and VLF6 from Marlton (R), with timestamps)
  • more live VLF streams at abelian.org/vlf/
  • http://dradio.ic.llnwd.net/stream/dradio_dkultur_m_a.ogg   ( Deutschlandradio Kultur from Germany .. moved to an unknown location )
  • http://127.0.0.1/_audiostream.ogg to connect to the built-in audio web server of another instance of SL
    running on the same machine, with its audio server configured for HTTP.
  • Use your imagination to connect to Spectrum Lab running on another PC in your local network. Only the IP address will be different (definitely not 127.0.0.1). You can see it on the other machine by looking at SL's HTTP server control panel, or on SL's Audio Stream Server control panel.

If it doesn't work, copy the URL into the address bar of your favourite web browser (if the web browser isn't able to play the audio, chances are low that this program will be able to play it). See next chapter for details about the supported audio stream formats, and the supported network protocols.

Often, the 'real' stream resource is hidden by an awful lot of Javascript or other stuff. In that case, playing around with the web browser can often reveal the 'real' stream URL. Spectrum Lab makes no attempt to execute Javascript, only to find out the real URL. In fact, it cannot execute browser scripts.

2. Sending audio streams to a remote server (SL acting as client)

This chapter describes how Spectrum Lab can be configured to send one audio stream to a remote server. In this case, SL acts as client, which means it initiates the connection, and then starts sending live audio to the remote server. This function was first used to feed live data from VLF receivers to the Live VLF Natural Radio server (with high-speed internet connection) operated by Paul Nicholson.
In contrast to the server, the streaming client as described below only requires a moderate DSL connection. For example, an monophone Ogg/Vorbis stream with 32000 samples/second, and Vorbis 'Quality' set between 0.4 and 0.5 required about 64 ... 75 kBit/second.

In SL's file menu, select Audio Stream Output. This opens a control panel for the audio stream output.

The control on this control panel are:

Configuration file for the outbound stream
That file contains all the details which the program must know to send the stream. It's a plain text file which you have to write yourself, and store in a safe place.
(you will need some info about the remote server for this - see next chapter)
The stream configuration file is not a part of the Spectrum Lab configuration (only the name of the file, but not the file itself). This eliminates the risk of passing passwords and other 'private' details along with a normal configuration file.

Automatically reconnect, Start output stream when Spectrum Lab starts:
Used for 'unattended operation'.

Send Channels:
Defines which of the input audio channels you want to send (left / right audio channel, or both), and whether to send GPS-based timestamps in an Ogg stream or not.

Save audio logfiles:
With this option, a "copy" of the outbound audio stream (Ogg/Vorbis) will be saved on your harddisk. The dotted button near the input field opens a file selector.


2.1 Configuration of the output stream

The configuration file may look like this:

; Configuration file for an OUTBOUND audio stream .
; Loaded by Spectrum Lab shortly before starting to send a stream.

; server: <protocol>://<host>:<port> .  protocol: rawtcp or http .
server = rawtcp://127.0.0.1:1234

; password: some audio streaming servers may ask for it
password =

; format: only required for non-compressed streams, default is Vorbis-compressed ogg
; format = ogg     ; default
; format = uncompressed,int8
; format = uncompressed,int16
; format = uncompressed,int32
; format = uncompressed,float32

; quality: see Vorbis documentation; 0.5 seems to be a good default value
quality = 0.5

Empty lines, or lines beginning with a semicolon, are ignored by the parser.
At the moment, the keywords shown in the example ('server', 'password', 'format', 'quality') are the only recognized keywords in the stream configuration file. Everything else is silently ignored.
A template for an output stream configuration file is contained in the installation archive, see configurations/output_stream_config_dummy.txt .

Non server-specific options are defined in the dialog box shown above. Only those data entered in the configuration dialog will be saved in a *.usr or *.ini file by Spectrum Lab.
The stream configuration file will never be modified by SL - it will only be read but never written (again, it's your task to write it with a plain text editor).

2.2 Notes on unattended operation

For long term operation, without regular supervision, you should..

  • Set the option 'Automatically reconnect when connection lost' on the control panel shown above.
  • To let SL start streaming without your intervention as soon as it is launched, set the option 'Start output stream when Spectrum Lab starts' on the same panel.
  • Turn off annoying screen savers, automatic hibernation, etc.
  • Stop windows from rebooting your PC without your permission !
    Don't forget the settings in the 'windows security center' or 'windows action center' (..what a name..).
    (in german: "Updates herunterladen, aber Installationszeitpunkt manuell festlegen"). If you turn off automatic updates in the 'security center' (or whatever MS decided to call that thing today), check for system updates yourself.
    Otherwise, you may find your audio stream broken down for a few days, just because windows decided to reboot your PC after installing a security update.
    Alternative (not recommended):
  • Leave automatic updates enabled (i.e. let windows update your system whenever it thinks necessary), and find out how to let windows launch Spectrum Lab automatically after each reboot.
    Under Windows XP, this was possible by placing a link to the program in the 'Autostart' folder.
    But you will also have to find out how to disable the login procedure at system boot, etc... especially under "Vista" and later versions you're on your own at this point.
  • If the purpose is to feed timestamped VLF streams to Paul Nicholson's Live VLF Natural Radio server, consider using Paul's VLF Receiver Toolkit under Linux.

3. Logging web streams

This is possible with received (inbound) as well as transmitted (outbound) streams. The stream data in Ogg/Vorbis audio format (plus the optional timestamp channel) can be logged as a file on the harddisk or similar.
Because the storage format of an Ogg file is exactly the same as an Ogg stream, the stream data are written without any conversion, so this causes no significant additional CPU load. Also, logging the audio data this way saves a lot of disk space compared to the wave file format.
At the moment, SL does *not* generate a different name for the logfile (with a sequence number, or date+time in the filename) but this may change in future versions.

3.1 Templates to specify filenames with date and time

On a number of occasions, the name of the logfile should contain the current date and time as part of the filename. This could be achieved by using a string expression evaluated by Spectrum Lab's interpreter, but the following formats (templates) with special placeholders may be even more intuitively to use, and faster to enter:

Instead of typing

  c:\vlf\20150115_2000.ogg

in the file selector dialog, and adjusting the name for the current date each time before starting the stream, you can enter

  c:\vlf\<YYYYMMDD_hhmm>.ogg

which works similar as the interpreter function str(), but it always uses the system's current date and time (in UTC), and replaces the 'time and date placeholders' as in the format string for various interpreter functions in Spectrum Lab.
The parser recognizes any template by the angle brackets (and removes them), because neither '<' nor '>' are valid characters in a normal filename.
The example shown above will produce a filename with the current year, month, day, hour, and minute in an ISO8601-like fashion. The placeholders for hour, minute and second are lower-case letters to tell 'mm' (two-digit minute) from 'MM' (two-digit month).
The following characters are automatically replaced by date- and time fields, when placed between angle brackets as shown in the example:

	  YY   = Year, two digits
	  YYYY = Year, four digits
	  MM   = Month, two digits
	  MMM  = Month-NAME (Jan, Feb, Mar, Apr, ...)
	  DD   = Day of month, two digits
	  hh   = hour of day, two digits
	  mm   = minute, two digits
	  ss   = second, two digits

Any other character placed between angle brackets will not be replaced.
Thus the underscore used in the example appears as a 'separator' between date (YYYYMMDD) and time (hhmm)in the filename.
The angle brackets themselves cannot appear in the filename (under windows, '<' and '>' are no valid characters in a filename anyway).
The same principle (filename templates with current date and time) can also be used in other places within the program, for example when starting to record the input from, or output to the soundcard as an audio file.

4. VLF radio streams

At the time of this writing, the following live VLF radio streams had been announced in the VLF group:

4.1 GPS-timestamped VLF radio streams

The following VLF live streams can only be played with the VLF Receiver Toolkit under Linux, or Spectrum Lab under Windows (since V2.77 b11). Note that 'rawtcp' is not an official protocol name, thus your browser will not know "what to do" with the above pseudo-URLs. But you can copy and paste them into URL input box shown in chapter 1.

More timestamped VLF streams will hopefully be available soon, since they can provide valuable information for research - see discussion and announcements about ATD (arrival time difference calculations), origins and spacial distribution of Whistler entry points, etc; in the VLF group.
As a motivation for prospective VLF receiver operators, the next chapter describes how to configure Spectrum Lab for sending such (GPS-) timestamped audio streams.

4.2 Setting up a timestamped VLF 'Natural Radio' audio stream

Prior to 2011, some of the VLF streams listed above used the combination of Spectrum Lab + Winamp + Oddcast as described here. Since the implementation of Ogg/Vorbis in Spectrum Lab, plus Paul Nicholson's extension to send GPS-based timestamps in his VLF receiver toolkit, the transition from the old (non-timestamped) MP3 based streams to the Ogg/Vorbis streams (with or without timestamps) has become fairly simple.

First of all, if possible at your receiver's location, consider sending a timestamped audio stream. All you need in addition to your hardware (VLF receiver, a soundcard with hopefully a stereo 'Line' input) is a suitable, despite cheap, GPS receiver. The author of Spectrum Lab had good success with a Garmin GPS 18 LVC (emphasis on 'LVC', since the Garmin GPS 18 "USB" doesn't deliver the important PPS output). If you are interested in the principle of GPS pulse synchronisation, also look here (later).

Here is a typical VLF receiver setup (hardware), with a GPS receiver delivering precise UTC time and date (through its serial NMEA data output) and the sync pulse ("PPS", typically one pulse per second):

Timestamped VLF Receiver Hardware

R1 and R3 form a voltage divider for the NMEA data, to reduce the amplitude to an acceptable level for the soundcard.

R2 and R3 forms a different divider for the 1-PPS (sync signal with one pulse per second). Important: The divided voltage of the 1-PPS voltage must be higher than the voltage of the NMEA (as seen by the soundcard), otherwise the software has difficulties to separate them. Also, the input of the soundcard must not be overloaded / clipped on any channel.

Capacitor "C" is not really required - it's only there to emphasize that a 'DC' coupling between the GPS receiver and the input to the soundcard is not necessary.

Fortunately, in all GPS receivers tested so far (Garmin GPS 18 LVC, and a Jupiter unit) had no overlap between PPS and NMEA.

A Garmin GPS 18 LVC with a relatively new firmware (anno 2011) produced this combined waveform on the 'R' audio input:

GPS waveforms at the soundcard input

A suitable configuration to send a timestamped VLF stream with the hardware shown above is contained in the Spectrum Lab installation.  

To load it, select File (in SL's menu), 'Load Settings From', and pick the file 'Timestamped_VLF_Stream_Sender.usr' from the configurations directory.

Further reading:

5. Operation as audio stream server

For ad-hoc tests, and only for a limited number of clients, SL can also operate as server (which can stream audio to remote clients).

Per definition, the server listens on a TCP port for incoming connections from remote clients. The server can only accept a limited number of connections, usually limited by the 'upstream' bandwidth of a typical ADSL connection.
To stream audio over the internet, it is recommended to use the Ogg/Vorbis compression.
In a local network (LAN), better performance (especially for very weak signals "buried in the noise") can be achieved by streaming non-compressed samples.

To configure SL's built-in audio streaming server, select
  'File' .. 'Audio Stream Server' from the main menu.

The audio stream server inside SL usually uses HTTP as the transport protocol. In that case the port of Spectrum Lab's integrated HTTP server is used (default: 80), as configured on SL's HTTP server control panel.

Hint:
To open the HTTP server control panel from the Audio Stream Server panel, first set 'Protocol:' to 'HTTP' as shown in the above screenshot.
The 'Listening port' on the Audio Stream Server panel will be grayed,
and shows -> HTTP to indicate that that the HTTP server's port number cannot be configured here, but on the HTTP server's own control panel.
Click into the grayed field showing '-> HTTP' to open the HTTP Server panel.

To embed the audio stream in a webpage (e.g. audiostream.html in SL's "server pages"), put an HTML5 <audio> element into it.
Note:
As often, "that certain browser" sucks.. it only supports the patent-encumbered mp3 format but not Ogg Vorbis.
As usual, the solution is try another browser... Firefox, Opera, Safari and most other browsers support Ogg.

To test the audio stream server without a browser, a second instance of Spectrum Lab can be started, for simplicity on the same PC. In that case ("local" test on the same PC), enter the URL http://127.0.0.1/_audiostream.ogg in the audio stream receiver (client), as explained in the chapter about analysing web streams.

6. Internals

6.1 Supported formats

6.1.1 Compressed formats

At the moment, only Ogg / Vorbis audio is supported. Optionally, the Ogg container may contain (sic!) an extra logic stream for the GPS-based timestamps. These streams are compatible with Paul Nicholson's VLF receiver toolkit, which is available as C sourcecode along with a detailed documentation at abelian.org/vlfrx-tools/.

MP3 streams, as well as MP3 files, are not supported to avoid hassle with Fraunhofer's software patents.

The "Content Type" (in the HTTP response from the remote server) must be application/ogg or audio/ogg.
Only for raw TCP connections the program will recognize ogg streams from the first four bytes ("OggS") .

6.1.2 Non-compressed formats

Supported data types (at the time of this writing, 2014-04-27) :
  8 bit signed integer (range +/-127, the pattern 0x80 is reserved to identify headers)
  16 bit signed integer (range +/-32767, 0x8000 is reserved to identify headers)
  32 bit signed integer (range +/-2147483648, 0x80000000 is reserved to identify headers)
  32 bit floating point (range +/- 1.0)
  64 bit floating point (range +/- 1.0)
As usual on Intel- and most ARM-CPUs, use little endian format (aka 'least significant byte first').

To allow automatic detection of the sample format, it is recommended to insert 'stream headers' at the sending end. Any header begins with the unique pattern 0x80008000 (in hexadecimal notation as a 'doubleword').
The same non-compressed formats can also be used when streaming audio over a serial port (aka "COM port" or "Virtual COM Port" under windows) into Spectrum Lab, with the intention to keep the microcontroller firmware (which may drive an external A/D converter) as simple as possible.

The header structures for non-compressed streams are specified below (as C sourcecode):


typedef struct VT_BLOCK // compatible with VT_BLOCK defined in vlfrx-tools-0.7c/vtlib.h { // DON'T MODIFY - keep compatible with Paul's VLF Tools ! int32_t magic; // should contain VT_MAGIC_BLK = 27859 = Data block header # define VT_MAGIC_BLK 27859 uint32_t flags; // See VT_FLAG_* below uint32_t bsize; // Number of frames per block uint32_t chans; // Number of channels per frame; uint32_t sample_rate; // Nominal sample rate uint32_t secs; // Timestamp, seconds part uint32_t nsec; // Timestamp, nanoseconds part int32_t valid; // Set to one if block is valid int32_t frames; // Number of frames actually contained int32_t spare; double srcal; // Actual rate is sample_rate * srcal (8 byte IEEE double precision float) } T_VT_Block; // Bit definitions for VT_BLOCK and VT_BUFFER flags field. Copied from vlfrx-tools/vtlib.h : #define VTFLAG_RELT (1<<0) // Timestamps are relative, not absolute #define VTFLAG_FLOAT4 (1<<1) // 4 byte floats (8 byte default) #define VTFLAG_FLOAT8 0 #define VTFLAG_INT1 (2<<1) // 1 byte signed integers #define VTFLAG_INT2 (3<<1) // 2 byte signed integers #define VTFLAG_INT4 (4<<1) // 4 byte signed integers #define VTFLAG_FMTMASK (VTFLAG_FLOAT4 | VTFLAG_INT1 | VTFLAG_INT2 | VTFLAG_INT4) // Mask for format bits typedef struct tStreamHeader { uint32_t dwPattern8000; // pattern 0x80008000 (0x00 0x80 0x00 0x80, little endian order), // never appears in the 16-bit signed integer samples because // THEIR range is limited to -32767 ... +32767 . // Also very unlikely to appear in a stream of 32-bit floats. # define STREAM_HDR_PATTERN 0x80008000 uint32_t nBytes; // total size of any header, required for stream reader // to skip unknown headers up to the next raw sample . // Must always be a multiple of FOUR, to keep things aligned. // Example: nBytes = 4*4 + sizeof(VT_BLOCK) = 16 + 10*4+8 = 64 bytes . // (assuming that 'int' in struct VT_BLOCK is, and will always be, 32 bit) uint32_t dwReserve; // 'reserved for future use' (and for 8-byte alignment) uint32_t dwStructID; // 0=dummy, // 1=the rest is a VT_BLOCK as defined in vlfrx-tools-0.7c/vtlib.h // (etc... more, especially headers with GPS lat/lon/masl will follow) # define STREAM_STRUCT_ID_NONE 0 # define STREAM_STRUCT_ID_VT_BLOCK 1 } T_StreamHeader; typedef struct tStreamHdrUnion { T_StreamHeader header; union { T_VT_Block vtblock; // used when header.dwStructID == STREAM_STRUCT_ID_VT_BLOCK }u; } T_StreamHdrUnion;

6.2 Supported protocols

At the moment, the reader for compressed audio streams only supports HTTP and 'raw' TCP/IP connections. These protocols are recognized by the begin of the URL (or pseudo-URL):

http://
Hypertext transfer protocol. This is the common protocol (also for audio streams) used in the internet.

rawtcp://
Uses a raw TCP/IP connection, not HTTP. Used for software tests; and to feed audio into VLF Natural Radio stream servers. The token "rawtcp" is not an officially recognized protocol; it's only used inside Spectrum Lab to leave no doubts about the protocol.

6.3 Stream Test Application

During program development, a simple test utility was written as replacement for a remote audio stream server (audio sink for Ogg/Vorbis encoded webstreams).
The sourcecode and executable (a windows console application) included in the 'Goodies' subdirectory after installing Spectrum Lab.
The "C" sourcecode contains a short description.
When launching the executable (TCP_Dummy_Sink_for_AudioStreams.exe) without parameters, it will open TCP port 12345 and listen for connections. Only one connection can be handled at a time. After being connected (for example, from Spectrum Lab), the utility will write all received data into a file, beginning with the file "test000.ogg". This name was chosen because in most cases, Ogg/Vorbis encoded streams will be sent by Spectrum Lab. For each new accepted connection (after the previous one has been disconnected), the file sequence number is increased, a new file will be written (test001.ogg, test002.ogg, etc etc). If the connection works as it should (and nothing gets lost "on the way"), the contents of the received file (written by the "TCP Dummy Sink" utility) will be exactly the same as the stream audio logfile (optionally written by Spectrum Lab, while sending the stream).


Last modified : 2015-03-23

Benötigen Sie eine deutsche Übersetzung ? Vielleicht hilft dieser Übersetzer - auch wenn das Resultat z.T. recht "drollig" ausfällt !