Mars Spirit Software Problem

I've noticed, over the last few weeks, that the anomaly suffered by Mars Spirit seems to be focusing on its software, specifically the apparent accumulation of "data files" in its flash memory. I've seen some references to problems related to a size mismatch between its flash memory and its RAM. Earlier, one of the services reported that the team said it had worked fine during the nine days of testing on earth, but that perhaps the problem was related to the 18 (I think that was the number) days of operation on Mars.

In other words, it sounds like a core leak (well, maybe a core/file system leak).

I wonder if anyone shares my interest in questions like the following:

This isn't much more info, but it runs VxWorks and I believe Nasa is rated a CMM 5. More in this article: http://www.space.com/businesstechnology/technology/mer_computer_040128.html.


See this thread: http://groups.google.com/groups?dq=&hl=en&lr=&ie=UTF-8&th=f1f58d2c78255ac6

The following excerpts that discussion: ...And since that will probably dissuade people from reading the thread, here's the best of the humor that appeared in those 40 articles:

>> > No. If I wanted to dissuade someone from using Lisp there are much more effective tactics, like suggesting they do a google search on "lisp jobs" and compare the results to "c++ jobs" and "perl jobs", or even "python jobs".

>> Googling on ``food service jobs gives you even more hits.''

> Ive never heard of the "food service" programming language. Is it new? Could you point me to a reference?

It is the latest thing. It has some of the features from all these languages: Java, Apple Script, Fromage, HotTea?, Jam, and Korn Shell.

Its best for writing spaghetti code .... if you can stomach it.


I can't believe they didn't think of this problem. There's no methodology that replaces thinking.

Don't underestimate NASA, the difficulty of what they do is enormous. To accuse them of not thinking is rather arrogant.

They ought to send the project lead out into the field to fix it... :)

Careful, there! Some NASA guys would want that, so you'd be actively encouraging them to insert bugs so that they could go on a field trip.

Wouldn't be too bad; after all NASA is reporting that there is a StarBucks? on Mars.

http://www.antimatternews.com/2004/01/us_rover_finds_.html


These guys are amazing: they were able to debug a problem at 120 bits per second using communication windows of half an hour at a time. But I bet even they couldn't have debugged the problem using standard tech-support protocols. ;-) Oh, and I hear they finally got the broadband guy out there to install a high speed connection...

Figures. Of course they'd support Mars long before my neighborhood!


From above: The flight software is all written in C running on vxWorks.

So now I want to ask some really provocative questions:

I hope that this very specific, pragmatic, and very real-world example might help to focus some of the various allegations and claims that have been made over the years on both sides of questions like this. Embedded systems running Smalltalk in real time were part of Tektronix oscilloscopes for a very long time (I don't know if they still are or not).

[They are not; the last TektronixInc scope to have SmalltalkLanguage in it was the 11k series; discontinued a while back. One reason Tek abandoned Smalltalk was that the low-end scope architecture (TDS1000, 2000, 3000 series) transitioned from the 68k architecture to the PowerPc, and a good Smalltalk implementation could not be found. High-end Tek scopes are, of course, PCs running Windows with special hardware inside. (This was a few years ago, before Squeak and others were widely available.) Lots of Smalltalkers still here at Tek though, reluctantly coding in OtherLanguages?...]

In retrospect, was CeeLanguage the best choice for the flight software of this mission?

It doesn't appear that CeeLanguage was the culprit; rather it's that the OS was misconfigured. If you use Smalltalk or Lisp on top of an RTOS (VxWorks or otherwise) you still have to configure the OS to your hardware; if you run Smalltalk or Lisp naked (can be done, though rarely done in EmbeddedSystems) likewise you still have to tailor your design to your hardware.

And the hardware design was severely constrained in many ways, for example:

These limit the choices you can make for your CPU and other components.

Given that they went with an embedded PowerPC environment, was a good Lisp or Smalltalk implementation available - one that has FLASH filesystem drivers and the like? Probably they could have gotten on; but it would be rather unlikely that such an environment would have significant use in production environments. While CeeLanguage and VxWorks have many shortcomings; there are zillions of deployments of such out there running along quite reliably. If you use vxWorks, you're stuck with C at the bottom layer for the most part - the language is implemented in C, its command shell is a sort-of-C interpreter, and its API is written entirely in C. You can run your app in other languages besides C, of course - most languages in widespread use have a vxWorks port of some pedigree - but it is doubtful that one would (or could, without rewriting vxWorks itself) rewrite the BoardSupportPackage? (the vxWorks version of a HardwareAbstractionLayer?) in anything but C. And the BSP is where the configuration code that was in error lies.

The short answer is - if the thing were done in Smalltalk, it probably wouldn't have made it to launch without tons of extra engineering effort. And that wouldn't have necessarily fixed the problem.


I know that lisp or smalltalk or your favorite language is the best thing ever, but how do you think flash is at all related to the language?

If they don't manage their storage, which I still find very hard to believe, they don't manage it. The language doesn't matter. I wonder if the file system check is taking too long with the extra data and it is watchdogging.


See Rover Navigation 101: Autonomous Rover Navigation: http://marsrovers.jpl.nasa.gov/gallery/video/animation.html

EditText of this page (last edited August 22, 2005) or FindPage with title or text search