Summary from ELOG 1529122:
10:01 am Both PSS A/B of South Linac drop to Restricted Access on OM18 going in and out of alarm.
SSG Group Leader on-call in training, pager was off. 10:05 SSG Tech contacted 10:20 SSG Group Leader Called on cell phone by SSG Tech 10:50
Troubleshooting confirmed that there was a PSS remote I/O error on both systems A and B. Problem tracked to link between South Access (Master drop) and SL12 (NL remote I/O).
OM18 fails, in and out of a "trouble" alarm.
KM Informed ADSO. KM informed Engineering Dept Head. He will call reporting manager.
Established problem is one or more broken fiber trunks between South Access and South Linac. Also broke a beam sync fiber in the same conduit.
Affected devices: PSS remote I/O between South Access and South Linac (A/B) PSS camera signals to MCC from South Access PSS Modbus Plus (PSS Display (A/B)) CARM RS-485 communications network
NOT Affected: Public address - tested system in both South Linac Service Building and South Access Service Building.
Not known: Copper signals between SL and SA buildings. (late entry - suspect OM18 cable damaged.)
14:00 Multiple teams pursuing repair, investigation, reporting, test.
Operability Manager has piece of broken fiber optic cable.
PSS fiber optic cable is unsalvageable. No spare PSS fiber optic trunks between SA and SL available.
Only option at this time is to use spare fibers available on a controls network fiber optic cable between south access and South Linac. Planning activity to ensure config. control is maintained. Created ATLis for task. Using six spare channels from SA building to SL01. Running new PSS fiber trunk from SL01 to SL12 (PSS racks). This allows us to retain maximum configuration control and minimize impact on other systems.
16:30 met with SCMB, PD, and Arne Freyburger to review requirements for testing and restart.
18:00 PSS cable from remote I/O patched in. Remote I/O and South Access cameras are re-established. However, the Modbus + fiber is in another damaged trunk cable in same conduit. Now Using last four available controls spares to re-route the Modbus + connection.
19:31 Modbus + re-established. Can now see signals in the control room. Preparing for South Linac system test. Note: This is NOT recertification. This is a limited test to ensure signals are working properly and not corrupted. Established configuration control over I&C fiber optic breakout boxes containing PSS fibers. Also used PLC diagnostics to test South Linac A and B PLCs: All passed and communication is OK. Preformed "Compare" operation between on-line and back-up PLC programs for the South Linac. - ALL OK (there are not configuration or logic errors. The logic that is running is the same as that certified in March 2010.)
No connectivity to the CARMS.
OK - Performing limited I/O tests to verify that I/O connectivity is correct. Performed 100 % test of crash switches. 21:01 Performed full sweep. 21:10 South Linac to Power Permit 21:15 Performed A/B functional test of representative RF zone (SL13). Passed OK. 21:29 South Linac to Beam Permit 22:08 OK to resume beam operations from AD Operations Dept. Head.
Other issues: OM18 is off line - no power to the transmitter located in the south access building. This is likely due to cable damage in the same PSS conduit. Oxygen monitoring is fail-safe. However, we should be prepared to operate under ODH1/2 rules if multiple heads fail. SSG will pursue testing of south access ODH monitors and alarms during next available SL access. Note: South Linac ODH beacons and horns did work when OM18 went into alarm. Quick Lessons learned (very small things I do not want to lose): Need lots of spare fiber lines between buildings May need redundant signal path ? Hall D PSS design supports this. May consider separate fiber trunks in separate conduits. Cables entering and exiting major conduit runs should be labeled. (We had to verify each fiber optic cable). Years ago we abandoned the cable and wirelist database as too labor intensive. They would have been very helpful for this problem. Good - performing end to end QA of fiber terminations as we went averted problems due to bad fiber or terminations. - PSS documentation of fiber optic infrastructure helped to identify narrow in on specific cables quickly. - Help from I&C in identifying, running, and installing, and QA of fiber trunk lines in a few hours was amazing.