User name Armstrong
Log entry time 14:27:06 on August 01, 2005
Entry number 150319
Followups:
keyword=
Cryo-flow problems and our Target
Target operators please read this!
After considerable discussion with Mark Stapleton and
Pete Knudsen of the cryogroup, and data mining from the target
EPICS variables (which are recorded in our data stream), I have
come to the conclusion that the target venting events that happened
twice were nothing that we could have prevented.
The cryo-group has "retuned their loops" today to try to smooth
out the oscillations in CHL 4K flow which caused the events that
eventually lead to the target venting twice on the owl shift.
This is a similar retune that they had done last Wednesday, after
which things were very stable. They got worse again over the weekend,
leading to CHL flow excursions (unrelated to any JT valve changes
on otr part) as noted many times in the halog, e.g.
entry 150071. However, until last night the excursions were never
quite rapid or large enough that we couldn't recover, usually just
by removing beam before the pressure spiked the the venting point.
I should have had MCC contact the cryo on-call when the oscillations in
CHL re-appeared, but I did not, hoping that it would not cause
a problem over the weekend - my mistake.
The cryogroup's strip-chart for owl is given in the figure below, taken from
clog
1283895.
The black line is the flow of 4K coolant from CHL (CFI6061), the same one
we see on the target controls. In the first event at 1:30 AM, you can
see the flow gradually increase from 21 g/s up to about 22 g/s. Then
there is a rapid spike up to 23 g/s, followed by a rapid drip down
to 19 g/s. This is when the target vented. Checking our data stream,
I find that the Target JT valve was not adjusted during this - the
change was not caused by us. It seems (to my untrained eye) to be
correlated with the gray curve (CFI6712) which is a flow meter for the
flow to ESR from CHL, which the flow to our target tee's off from.
Note that at no point did our demand on the CHL flow exceed the 25 g/s
maximum we have been given (in fact it never exceeded 24 g/s).
The wild oscillations in the CHL flow after that until 4:00 AM
represent the cooling and re-filling of the target.
Then, from 4:00 AM to 5:40 AM or so, we see the same phenomenon: a gradual
rise in the CHL flow at 6061 (black line), unrelated to the JT valve
setpoint, followed by a sharp spike up (this time to > 25 g/s), again
followed by a rapid plunge, whereupon the target vented. This process
is again perfectly correlated with the flow in CFI6712 (gray line).
If the target operator had tried to toggle back the JT valve to
compensate for the slow rise in CHL flow, that would have only
made things worse, as there would have been less coolant available
when the plunge in CHL happened.
So, what should we do?
- The cryogroup have "retuned their loops" - things should be more stable now. Thanks!
- If similar "burps" in CHL flow happen in the future, please halog them. If more than one significant event occurs, contact the run coordinator, and I will contact the cryo-on call. Gentle fluctuations of +- 0.5 g/s (at fixed JT setting) are OK and normal, as well as slow drifts.
- MCC has set up a software "chime" to alert and turn off the beam if the CHL flow goes outside the range of 20 to 25 g/s; see halog entry 150293. We may have to adjust these limits...
- If you suspect a cryo-related issue, ask for no beam, as quickly as you can. That way we minimize the chance of a venting of the target.
- Make sure that the software alarms on the target GUI are set appropriately, especially after a target IOC reboot, so that you will be notified of a problem quickly.
Cryogenic Strip Chart (owl shift)
