User name J. Singh
Log entry time 06:06:08 on May28,2004
Entry number 123473
This entry is a followup to: 123452
keyword=BCM probs fixedthe whole story:
at 0252, we noticed that the BCM readouts on the medm screens went blank. the medm message window popped and displayed the message in fig 1. i tried rebooting the IOC (iocha5) from the remote reboot at http://hareboot5. this did not work. i called up RC Paul and MCC. MCC tried to reboot the IOC remotely. that didn't work. they called software on call.
in the meantime, we fiddled with stuff back at the counting house. we found the iocha5 in the "middle" room of the counting house in rack CH01B06 at the bottom. we tried a manual reset of the button with our hands. this didn't help.
we noticed that the HP3458A multimeters were not updating, so they were power cycled. afterwards they were updating every sec or so and displayed values in VDC. after reading the HALL A OPS manual and recalling that before the display showed VAC, we changed the displays to VAC. someone should check this.
we finally got a call back from MCC. they said that the software on call was working on the problem from home. the software on call seems to think that the problem in not the IOC but some issue with the "channel access" something something. note also in fig 1, that the message displayed in the medm window does talk about "channel access" something something. see below for a potentionally useful link regarding this "channel acess" something something business.
at about 0440, MCC calls back and says that it should be fixed. it is. at about 0450, the BCM screens go blank again. i call MCC and they say that this time it is probably a IOC problem, so they do a remote reboot. at 0452, the BCM screens are back up and everything seems okay.
during this time, we also tried to figure out if these problems were affecting the insertion of the BCM info into the CODA datastream. both Xiaochao and Peter pointed out that the BCM info was showing up in the end of run summaries for those runs that were taken while we couldn't see the BCM info on the medm screens. right now we're trying to see if we can extract the BCM info from the raw data files...worst case → runs 1331 to 1336 will not have BCM info.
relevant halogs and elogs:
0253 halog 123452: BCM medm screen is white
0329 elog 1215204: in coming phone call:Hall A
0338 elog 1215205: iocha5 is down update
0408 elog 1215207: CA Security Semaphore Investigation
0431 elog 1215208: Phone support: software on-call, iocha5
0517 elog 1215211: iocha5 prbms
0521 halog 123467: BCM ioc not responding to reboot
note that at the bottom of the "HALL A IOC SYSTEMS" medm screen (see fig 2), it says to "See the troubleshooting guides on the LLAPPS section of the controls_web home page." these links are listed below.
relevant sites + docs:
remote reboot for iocha5 (BCM): http://hareboot5
controls web trouble shooting guides: http://devweb.acc.jlab.org/controls_web/ONCALLSUPPORT/body.php
trouble shooting guide for channel access problems: http://devweb.acc.jlab.org/controls_web/LLAPPS/TroubleGuides/CAsecuritySemaphore.html
hall A IOC info: http://devweb.acc.jlab.org/controls_web/LLAPPS/TroubleGuides/hallA.html
FIGURE 1
FIGURE 2
