ARCHIVE AND STACKING FOR QrssPiG WITH A BASH SCRIPT
(2018)

VOOR VERTALING IN HET NEDERLANDS, KLIK DE RECHTER MUISTOETS EN KIES "Vertalen in het Nederlands"


Software for QRSS reception! Barefoot in our radio lab shows exactly what the right mentality for QRSS is!
Less instead of more! Milliwatts instead of kilowatts, barefoot instead of shoes!

Software
Software for a 24 hour archive and stacking of pictures for my grabber with RTLSDR dongle and QrssPiG grabber! Bob G4IOG had it on his website and was so kind to give me the software. It was a BASH script for Linux, the operating system of the Raspberry Pi. Only a few lines of script code, simple and minimalistic, exactly what we want!
Of course, you can only install and use such a BASH script, but it is much more interesting to try to understand the BASH script and to play with it. A BASH script, a totally new world! Is it possible to add an 8 hour grab? New challenges! So let's go barefoot in our radio lab! Not only to avoid Electrostatic Discharges, barefoot in our radio lab shows exactly what the right mentality for QRSS is! Less instead of more! Milliwatts instead of kilowatts!


QrssPiG grabber with links to the 24 hour archive pictures hf1-0.jpg ... hf1-71.jpg

What is a BASH script?
A BASH script is a readable ASCII text file. You can edit it with any text editor like Notepad of Leafpad. But I prefer "Geany Programmer's Editor" as you can see here below.
For the installation of Geany: Type in a Terminal screen: "sudo apt-get install geany".
Linux and BASH scripts were new to me. Where shall I start to learn something? Ryan's Tutorials (https://ryanstutorials.net/) did help me. Was it too difficult for me? The trick is simple, just go on, do not give up! And after a few evenings I could already understand the basics of a BASH script. It was even possible to add a few extra lines to make an 8 hour grab!
After two weeks I had a nice BASH script to make a 24 hour archive of all grabs and add stacking and a 8 hour grab with a QrssPiG grabber! And that BASH script can also do the FTP upload, so that the grabber does not have delays or buffer overloads or other problems due to long FTP uploads! And that BASH script can also delay the FTP upload a few minutes, so that you do not have the problem that all your grabbers are trying to upload at the same time! And one BASH script made the archive and stacking pictures for all the grabbers and did the FTP upload for all grabbers in one session instead of that all the grabbers had their own FTP session! Let's have a look at the first simple BASH script!


A BASH script opened with the programmer's editor Geany.
This is the first part of the script that makes the archive,

BASH script part one
A BASH script always has to start with the line "#!/bin/bash" to make clear that it is a BASH script. Only the part of the BASH script here above is already enough to make the 24 hour archive and to store the grabber pictures with a time and date stamp on your harddisk (or SD card).
It starts with the initialization of variables for the date and time information.
Then the variable "p" is calculated. It is a number from 0 to 71 and represents the start time of the grabber picture of the archive. The start time is p/3 (in hours).
"%72" is a modulo 72 division. 0%72=0 ... 1%72=1 ... 71%72=71 but 72%72=0 and 73%72=1 again. So the maximum number of files is 72 and that is exactly 24 hours of 20 minute grabs or 12 hours of 10 minute grabs. The "+71" is a substitution for "-1". The modulo division did not work correctly for negative values.
It seems so simple, but this simple formula did take quite some time! I am not such a smart guy...
The variables "sn1" and "an1" get the filename for the storage and the filename for the archive. QrsspiG saves the grabber picture to "QrssPiG-master/qrsspig.jpg". And all we have do is to copy the files to the right folders with the right names.


The second part of the BASH script.
Only 4 lines of text to make the average and peak stacking grabs!

BASH script part two
The archive files are stored in the folder "/home/pi/grabberresults/archive1/" so we start with changing the directory to that folder.
We make a list with file names "FILE_LIST1". It is a list of the most recent grabber pictures.
"ls -tr *jpg" lists files in directory, sorted jpg and reversed in time (-tr)
The symbol "|" is the piping symbol. What is that? It reroots the output of the "ls" command to the input of the "tail" command.
"tail -6" just takes the last 6 file names. If you want to stack more, change -6 to the desired value.
The symbol "|" is again the piping symbol to reroot the output of "tail" to the input of the "sed" command. "sed ':a;N;$!ba;s/\n/ /g'" removes the newlines between file names and replaces it with spaces or better, add spaces.
If you want to know how the commands work, type in a Terminal window "man ls" or "man tail" or "man sed" for the manual of a specific command!

And then only one line of code: "convert $FILE_LIST1 -average /home/pi/grabberresults/mf1.jpg"
It does the average stacking of the file list and saves it to a "file with the name "mf1.jpg"!

The next line is: "convert $FILE_LIST1 -background black -compose lighten -flatten /home/pi/grabberresults/lf1.jpg"
It does the peak stacking. In the original script, only one color (Blue) was taken and converted into a Black and White picture. Why Blue? Why not Red? However, not Red bud Green gave better results. After one evening of trials with errors, the best result was with all colors. So the result is not a Black and White picture, but the normal color picture.


The third part of the BASH script.
Just 5 lines of text to make the 8 hour grab.

BASH script part three
Ready for the next challenge, an 8 hour grab!
We start to make a list with file names "FILE_LIST1A", just as for the archive. However, not 6 but 24 files as 24 x 20 minutes = 8 hours. Then we have the next command line:

convert $FILE_LIST1A +repage -gravity West -chop 93x0 -gravity East -chop 195x0 -resize 4%x100% +append /home/pi/grabberresults/nhour1.png

With "-gravity West -chop 93x0 -gravity East -chop 195x0" we remove the left and right borders, so that we have only the picture part. Then the pictures are resized to 4% of the original width and then added with the "+append" instruction. The result is stored in "nhour1.png". It is important to have the "+repage" command at the beginning and the "+append" instruction at the end. It took me a few days to find that out!
The next line cuts the frequency scale of the picture "hf1.jpg", appends it to the 8 hour grab and removes the top (North) and bottom (South) borders from the result and stores the result as picture "hf1.jpg".
If you want to know more, type in a terminal window "man convert".


The fourth part of the BASH script.
The FTP upload routine for all the files. And "exit 0" to end.

BASH script part four
We are not ready yet, we have to do the FTP upload.
Some data is printed to the terminal screen with the "echo" command, just for information purposes.
Then there is a "sleep 1" command. It is a pause of 1 second. If you have more of these programs or more grabbers with FTP upload, you can change it to avoid that they all upload at the same time. Give all running programs a different value then, depending on how long an upload lasts.
The work directory is changed in the next line to the location of the files.
Then we have the file names and the FTP configuration HOST, USER, PASSWD and RDIR. RDIR is the remote directory, the directory of the FTP site where you want to upload the files. "./" is the current FTP directory.
The line "ftp -invp $HOST" opens the FTP session. Use "-inp" instead of "-invp" if you do not want that detailed information is printed to the Terminal screen during the FTP session.
The files are uploaded and the FTP session is closed.
Again a "sleep 1" pause of 1 second so that you can read the info messages of the FTP upload before the Terminal window is closed. If 1 second is too short, change the value to 10 seconds or so. And "exit 0" is the end of the program with error code "0" (no error).


The last part does nothing, it is only an explanation.

BASH script part five
The last part is just an explanation and notes about how to use this BASH script. This is a simple script, but you can make it much nicer or add your own routines. It would be nice to have a 24 hour grab and an archive of 3x8 hour grabs. And you can add text and markers to the grabs, it is all possible with "ImageMagick", that is the software package with all the commands used here. Do not try to understand all the commands. Just read the information and pick the commands that you find interesting to use.

Installation of the BASH script and other software
Save the BASH file on you Desktop and MAKE it EXECUTABLE (I always forget that...). There is an explanation at the end of the BASH script. Edit it with the text editor Leafpad or better Geany (installation: sudo apt-get install geany).
AND IMPORTANT: SET THE PROPERTIES OF THIS BASH FILE TO: Permissions --> Execute --> Anyone!
We have to install Imagemagick for the conversion and editing of the images and ftp for the ftp upload. Type in a Terminal screen:

sudo apt-get install imagemagick
sudo apt-get install ftp

Download the software here:

ZIP file with BASH scripts

New Version
In 2024, a new version is made. With a peak stacking and average stacking archive and a new directory structure.
The ftp upload did not work anymore, the Linux version is based upon tnftp nowadays. But "lftp" did the job perfectly! Install it, type in a Terminal screen:

sudo apt-get install lftp

However, he new BASH script is more or less similar to the old version.

How do we execute the BASH script?
But... How do we run that BASH script? Do we have to execute it manually every 20 minutes after the end of a grab? No, we can use a crontab for that!


We have to make a crontab so that the BASH script is executed every 20 minutes.

What is a crontab?
We need to execute the BASH script every 20 minutes (or 10 minutes if you have 10 minute grabs). And we can do that in Linux with a "crontab"!

Make a crontab. Type in a Terminal screen "crontab -e" and make it as follows:
m h dom mon dow command
*/20 * * * * /home/pi/Desktop/grabaid1.bash

This will activate the script "grabaid1.bash" every 20 minutes, just after that the grabber software has saved the picture.
The picture here above shows an example for "grabaid2.bash" and that the printed information is saved in the log file "/home/pi/Desktop/grabaid2.txt" instead of printed on the Terminal screen.

More BASH scripts
In the ZIP file with BASH scripts, you will find a "grabaid123.bash" BASH file. It is a BASH script for 3 grabbers and the last one is a special one! Instead of my own grabber, it downloads a picture from the internet from the grabber of WA5DJJ and uses his picture to create a webpage with 24 hour archive and stacking pictures and 8 hour grab! Downloading is done with "wget", type "man wget" in a terminal screen to find out how that works.
The BASH script "avpkstacking.bash" can be ued for stacking of ".jpg" files that you have. Store all the ".jpg" files and the script in one directory and it stacks all the ".jpg" files in that directory. The results will be in "aaa-av.png" and "aaa-pk.png". Indeed a ".png" file instead of ".jpg" file. When you execute the BASH script again, it will only select the ".jpg" file and not its previous ".png" results.
The BASH file "getgrab.bash" is an example that can be used to download pictures from a grabber and make an archive, hour grab and stacking. Then you can see later if your MEPT is received by that grabber.

Index PA2OHH