This was touched upon already in the halog yesterday. Around noon yesterday the heater and loop thermometry starting going nuts. It was pretty clear that the problem was a flaky Cernox thermometer, the one at the input to the cell manifold, because the other thermometers were not swinging around nearly as wildly as that one, which (naturally!) was the one we were using in the heater PID loop. Damn that Murphy!
So we changed the thermometer used in the PID to the cell manifold out thermometer. That tamed the heater and solved the problem. We have 3 pairs of thermometers in the loop so there is plenty of redundancy.
There are two minor annoyances that are a result of switching the thermometer. First, the alarm handler looks at that thermometer and since it's fubar, we got alarms. So we disabled the alarms for that one thermometer in the alarm handler. I expect that if the target IOC is rebooted, then we'll have to disable it again.
The second annoyance is that the analog temperature readout we now have in the counting room, on the red LED display on a panel about a foot above the target computer screen, is useless now. It is hardwired to the thermometer that went bad. It can be changed to the new thermometer we are using but that has to be done in the hall when it's open. In the meantime we don't have a reliable analog (IOC & network independent) readout of the loop1 temperature.
Why did it fail? The 4 leads on each thermometer are such fine wire that it's almost impossible to see them with eyesight like mine. They're really fine wire, I mean 30-something gauge wire, to reduce heat transfer, and there are 4 of them per thermometer so the resistance of the leads can be eliminated. So they are hard to solder and one of the solder joints seems to have gone bad. There seems to be a kind of short in one of them, like a 1 Ohm resistance to another of the leads Chris said.
But after the fan problem this morning it could also be this thermometer got hit by something, or fell off by itself due to a cold solder joint or something and is flying around in the loop now.