Fact: Since midnight, our ep ion chamber has been tripping with increasing frequency. This brings down all 3 halls.
MCC thinks our dithering system in combination with the large raster is causing this. Although I would argue that these trips were occurring much more frequently than our dithering cycles.
MCC says that early in the owl shift they checked the orbit with tuned beam and it had not changed.
Our spot++ plots have not changed.
The dithering system is the next suspect...if the raster is very big and then we dither, it could cause part of beam to hit an aperture near the ep ion chamber.
MCC said they would like to do an aperture scan and an Fopt (tune mode) to check the orbit lock. Then they want to try to deliver static kicks to the dithering coils (one at a time) to try to reproduce the problem (Rupesh is coming in to help with this).
MCC operator Wang called a while ago to turn dithering back on (which we did, however they are not turning it on on their side and so no dithering magnets will be energized, but the system will go through its cycle...which involves turning FFB OFF--which is now the suspect) Since we have done this, everytime the dithering cycle is initiated and FFB is turned off the beam trips. It has been 4 cycles and the beam tripped during each of them (but we were not really dithering).
However, we have been running with this condition for several days without this problem. I asked MCC about raising the ep ion chamber trip limit, and they said it was last changed 2 weeks ago and they do not want to change it unless they know what is going on.
Wang called to ask if we can decrease the raster size. I said I would like to see more evidence that this is the problem before changing the size.
When Rupesh gets in we will do some vertical coil kick tests to see if we can reproduce the problem (when MCC is ready).
Arne called and said they would like to run this way for a while longer while they look into the FFB issue.
We are taking production data at 100uA (and have been since ~8:30am).
More to come later.
A copy of this log entry has been emailed to: paschke@jlab.org,souder@jlab.org