A program frozen at the point of failure
detection may have already erased the evidence
that explains the failure.
trap each statement of the program and record
crucial information in a CircularBuffer
this buffer when the program halts to determine the
circumstances leading to the failure.
The concept is similar to the flight voice and
data recorders recovered from the wreckage of
airplane crashes. Information is easy to capture,
but, since failures are rare, all but the most recent
records are discarded.
I used a variation on this concept
to hunt bugs in multi-microprocessor
hardware and software.
After each statement in the program
I mechanically inserted ...
- A call on a trap handler
- the source line number
- a pointer to a table of in-scope variables
As inserted, the calls were to a subroutine
that would freeze the program, record and display the
line number, and wait for interactive requests
which included ...
- display in-scope variables
- disassemble code for the statement (to find compiler bugs)
- examine system message queues
- display the buffer of recorded line numbers
- modify the trap to call a more passive handler
In essence, the program started with a breakpoint on
every statement and this last interactive command
removed them. Except it didn't remove the breakpoints
completely. It changed them to breakpoints that silently
recorded the progress of the program in the circular buffer.
I would step through a loop once watching
each statement go by. The second time through the loop
I would remove the breakpoints as I went. Then, zip, I
would be out of the loop, at the next statement's breakpoint.
A quick check of the buffer would make sure the final
passes through the loop were as well behaved as the first.
This may all sound trivial compared to the heroics that
modern debuggers will perform on your behalf. Where it
saved me was when the systems crashed in bizarre ways. I
was never left without a clue. And, I could trust the
clues because of the absolute simplicity of the debugging
You're my hero, Ward. I have probably used plenty of stuff you touched over the years. -- MartySchrader, long time and extensive user of Tektronix instruments, development systems, and software
Hugh Blair-Smith Describes the same technique used to debug the ApolloGuidanceComputer?
in the 60's.
Another function served by these auxiliary erasable memories was called the "Coroner," which recorded every instruction executed, with its inputs and results, writing over the oldest such record when it filled up. When a program "crashed," you could punch out a full record of what it was doing in most of its last second and analyze the problem at your ease. I have often wished that PCs offered such an advanced feature.