AO-40 power anomoly

We experienced some kind of power anomoly on AO-40 on 26 January 2004 which caused the satellite to do a safety shut down. Here are some plots showing a few potentially interesting sensor readings at that time, and at intervals previous to that. It appears that there might also have been a power anomoly in spring on 2003 I am still looking at.

Here are the last two orbits for which we have telemetry prior to system shutdown:
redI-Bat, Tot Bat Amps
greenI-28-U2, Main Bus
blueI-28-BCR, BCR amps
yellowVolts - Main Battery
purpleVolts - Aux. Battery
cyanTemp - SolPanl - 1
orangeTemp - SolPanl - 2
Note:   Colors may not match legend exactly, newer overlapping colors override rather than mix.
X-axis is MA, Y-axis is raw telemetry units, given this is mixed plot.

 
Looking back at one month intervals, we have:
2003-12-26_01449   #40
redI-Bat, Tot Bat Amps
greenI-28-U2, Main Bus
blueI-28-BCR, BCR amps
yellowVolts - Main Battery
purpleVolts - Aux. Battery
cyanTemp - SolPanl - 1
orangeTemp - SolPanl - 2
Note: Colors may not match legend exactly;
overlapping colors override rather than mix

 
2003-12-26_01448   #41
redI-Bat, Tot Bat Amps
greenI-28-U2, Main Bus
blueI-28-BCR, BCR amps
yellowVolts - Main Battery
purpleVolts - Aux. Battery
cyanTemp - SolPanl - 1
orangeTemp - SolPanl - 2
Note: Colors may not match legend exactly;
overlapping colors override rather than mix

 
2003-11-26_01411   #78
redI-Bat, Tot Bat Amps
greenI-28-U2, Main Bus
blueI-28-BCR, BCR amps
yellowVolts - Main Battery
purpleVolts - Aux. Battery
cyanTemp - SolPanl - 1
orangeTemp - SolPanl - 2
Note: Colors may not match legend exactly;
overlapping colors override rather than mix

 
2003-10-26_01372   #117
redI-Bat, Tot Bat Amps
greenI-28-U2, Main Bus
blueI-28-BCR, BCR amps
yellowVolts - Main Battery
purpleVolts - Aux. Battery
cyanTemp - SolPanl - 1
orangeTemp - SolPanl - 2
Note: Colors may not match legend exactly;
overlapping colors override rather than mix

 
2003-09-26_01334   #155
redI-Bat, Tot Bat Amps
greenI-28-U2, Main Bus
blueI-28-BCR, BCR amps
yellowVolts - Main Battery
purpleVolts - Aux. Battery
cyanTemp - SolPanl - 1
orangeTemp - SolPanl - 2
Note: Colors may not match legend exactly;
overlapping colors override rather than mix

 
2003-08-26_01295   #184
redI-Bat, Tot Bat Amps
greenI-28-U2, Main Bus
blueI-28-BCR, BCR amps
yellowVolts - Main Battery
purpleVolts - Aux. Battery
cyanTemp - SolPanl - 1
orangeTemp - SolPanl - 2
Note: Colors may not match legend exactly;
overlapping colors override rather than mix

 
2003-07-26_01256   #219
redI-Bat, Tot Bat Amps
greenI-28-U2, Main Bus
blueI-28-BCR, BCR amps
yellowVolts - Main Battery
purpleVolts - Aux. Battery
cyanTemp - SolPanl - 1
orangeTemp - SolPanl - 2
Note: Colors may not match legend exactly;
overlapping colors override rather than mix

 

This may not have been the first time this kind of thing has happened. Some kind of similar, less dramatic phenomena happened during the spring of 2003 that i am still studying, which appears to have resolved on 26 April 2003. Curiously enough, this was nine months to the day before the event on 26 January 2004 that caused a power shut down on AO-40, or i might not have spotted it.
 
Main battery voltage (yellow) went from approximately 25.5V back to nearly 28V after at least several weeks of fluctuating frequently in that range. For example, at one point, at seemed like at least one cell was dropping out part of the time. (Additional details here after further study.)

Some people have speculated that the propusion failure in December of 2000 may have damaged the main battery. That could well be the case, but telemetry doesn't show much change after that event and i didn't see much fluctuation until the latter part of 2001, which seemed later to become stable again. It seems to have done that several times before our most recent event.

Run your own query

I've saved the plots of these telemetry signals based on what's available 1in the on-line archive, from launch to the end of January 2004, and you can run your own query. If you want to look at the actual data, there is a link to it next to the plots but be aware that the unzip'ed archive lives on my laptop, and thus may not always be available to allow the decoded data to be examine.

Please don't publish the resultant URLs, as the machine which generates and stores these plots is not a public server.


This shows a sequence of plots by plot number, with #1 being the most recent event when AO-40 shut down due to very low batteries. For example, here's a query that shows the ten most recent plots (orbits) and another query which shows 10 plots at 20 orbit intervals starting with plot #2.

Starting plot (reverse chronological order, #1 is most recent):
Interval between plots: , number of plots to show:

Once you've run a few and know what the files look like, you can select plots by pattern. For example, the 'regular expression' '-26' shows the plots on the 26th of each month.

Query by pattern: , count:

Below is an attempt to print out changes in the status satellite register accoridng the published telemetry specification. Note that EXPFLAG disables the respective devices, and PSU Relay control connects both batteries to the power bus if both Main and Auxillary bits are turned off. (See AMSAT Phase 3D Telemetry.)
2004/01-26_01487/a01487.tlm:
 MA 9	#1	0x1ED: 0x00->01	 E-FLAGS Emergency status
		  ON: Battery voltage low
		0x1EE: 0x00->80	 EXPFLAG -lo 
		  ON: Passbands
		0x1EF: 0x00->10	 EXPFLAG -hi 
		  ON: Heater
2004/01-26_01488/a01488.tlm:
 MA 8	#1	0x18B: 0x01->00	 PSU Relay control
		  OFF: main battery
		0x1ED: 0x01->03	 E-FLAGS Emergency status
		  ON: Battery voltage very low
		0x1EE: 0x80->ff	 EXPFLAG -lo 
		  ON: Laser
		  ON: RF monitor
		  ON: A CAM
		  ON: B CAM
		  ON: GPS
		  ON: K Tx
		  ON: X Tx
		0x1EF: 0x10->1e	 EXPFLAG -hi 
		  ON: Rudak
		  ON: Cedex control
		  ON: Cedex power
 MA 32	#155	0x1ED: 0x03->00	 E-FLAGS Emergency status
		  OFF: Battery voltage low
		  OFF: Battery voltage very low
 MA 42	#243	0x1ED: 0x00->02	 E-FLAGS Emergency status
		  ON: Battery voltage very low
Here's my interpretation of the status bits from the last few orbits before the power anomoly occured.

Early in Orbit 1487, the satellite detected a low voltage condition, and disabled the passbands and the fuel heater (not in use), setting the corresponding bits on EXPFLAG. Soon thereafter, the main battery seemed to return to normal, but the BCR current continued to fluctuate (as it had in recent passes). At about the same time in Orbit 1488, the satellite detected a very low voltage condition, and disabled all non-essential functions by setting all bits in EXPFLAG (to 0xFF). PSU Relay control was also cleared connecting both batteries to the +28V bus (as before, only the main battery was connected to the +28V power bus).

At that point, the voltage on both batteries appeared to rise to about 27V (where 30V is the maximum value typically seen) by MA=20, fluctuated between 26V and 27V until MA=39, dropped to about 25.5V and collapsed between MA=40.6 and MA=40.9, dropping about 18V. It appeared to sit there until MA=40.9, and at MA=42.6, dropped to about 14V, where it stayed until we apparently lost telemetry at about MA=45.

So it appears that during that orbit, voltages reported for the main battery, auxillary battery and the +28V bus were within a few tenth of a volt, so one might conclude that the auxillary battery was at least partially recharged during that cycle. However, it seems that if the controllers can manage to separate the batteries and one of them recharges enough to permit partial operation, if it can't stay above the "very low voltage" point, it may be hard to keep the two batteries separated.


Good luck and if you find anything interesting, you might report it to the 'AMSAT-BB' mailing list.

-- KD6PAG