MINIBOONE DETECTOR: critical systems rapid response


AUTODIALER CALLS:


1. HVAC High Temperature: Check detector electronic rack temperatures (Rack Temperatures). All rack temperatures should be 35°C or less. If any temperature is more than 35°C, call DETECTOR / ELECTRONICS EXPERT immediately and discuss the situation. A decision might be made to call the MCR (630-840-3721) and tell them that the MiniBooNE HVAC has a problem, and ask them to send the duty mechanic to check it out immediately. If no action is taken and the temperatures continues to rise, racks will trip off at 40°C


2. Electronics Low Voltage: Check to see if DAQ is running (DAQ Monitor). Look for non-zero trigger rates and extremely short run times. If DAQ is still running normal, it was a momentary voltage spike (sometimes due to lightening storms). Keep checking the DAQ for normal operation. If DAQ is not running, then call the DETECTOR / ELECTRONICS EXPERT immediately.


3. Detector Main Bus Power Fault: See procedure in 2 above.


BEAM / DAQ / ACNET / COMPUTING PROBLEMS:


4. Check (ACNET Monitor Page). If ACNET beam rates are zero or inconsistent with DAQ rates, or file count >5000: if problem persists more than 5 minutes, call ACNET EXPERT immediately.


5. Check (DAQ Monitor Page). If TANK DAQ trigger rates zero and/or run times are extremely short (~minutes): if problem persists more than 5 minutes, call DETECTOR / ELECTRONICS EXPERT and DAQ EXPERT immediately.


6. Check Booster Neutrino Beamline Alarms and Autotune: If in alarm (red) for more than ~1 hour and the Beam (Ops/BNB) logs do not indicate a problem, call MCR and politely inquire.


7. Check Nearline/Enstore/UberDAQ: If persistent problems/errors, email nearline expert