Log entry time 17:23:14 on March 9,2010

Entry number 311236

15:20 Start tests of HAPPEX DAQ. 

Run 1856 at 240 Hz
Run 1857 at 120 Hz but did not change HAPPEX 
timing board (deliberately)
Run 1858 back at 240 Hz.

Restore the software (hapAdc18Lib and happex3_ts.crl) without
the "dirty trick" mentioned in halog 310993.

Run 1859.  See the usual "bad" data in 1st ADC.

Run 1860.  Added some logMsg diagnostics to CRL.
Bad data corresponds to csr = 0xa (output buffer empty)

Run 1863.  Make timeout loop 5000 (normally 50).
Observe icnt = 4275 +/- 10.
Note: 4275 is approx 4.2 msec = 1/240Hz, so we apparently
sometimes miss a helicity gate on the board.

Later in run I observe the "buffer full" condition csr = 0xd.

Run 1864.  Turn off logMsg.  This make things much cleaner,
I guess the logMsg can cause deadime leading to buffer full.

Then in 100K events I saw once this printout (what Rupesh saw):

-> interrupt: 
ADC18 ERROR: Buffer full ! Id 1 Csr = 0xffffffff
interrupt: 
You must power-cycle the crate
interrupt: 
ADC18 ERROR:  data ready yet buffer empty ! Id 1 Csr = 0xffffffff

This is for the 2nd ADC (id==1).
All bits 1 means the ADC was not addressable.
This is a different disease from the more frequent timeouts on
the 1st board, but may be related via bad things happening in 
the VME crate.   

Run 1865 ... put in more logMsg to dump everything
when timeouts occur
1st ADC getting buffer-full condition (csr = 0xd) sometimes.
I think these go away if no logMsg, so no real worry.
(And these really mean buffer-full, not non-addressable.)

Run 1872.  The csr=0xffffffff problem happened once.

Summary of problems:

1. 1st ADC in readout times-out 1% of time.  No solution yet.

2. Rarely, the 2nd ADC is not addressable, csr=0xffffffff.

3. If logMsg in interrupt routine, may get buffer-full (csr=0xd).
Solution: don't use logMsg.  (it's only for tests)

To be continued ...