Past news items, and archive:
Star Office DAQ talks since 10/2002.
run_monitor improvements - (12/16/2003): Improvements to run_monitor to reduce time between runs, and to impose the 8 hour run limit automatically.
Scaler inputs from trigger - (12/29/03): Chuck Alexander has been making final improvements to the scaler inputs so that the signals from the trigger board may be put to a scaler. The proposed 15 trigger inputs are the following: DET, VETO bits, SYNC, CLK, EOF (broadcast), FIFOW(broadcast), FIFOHF(trig memory), QT FIFO HF(QT status), TSA bits 0,1 (broadcast). Work done also by Richard, Nate, & Jackie (from LANL) is appreciated.
Trigger history data - (12/4/03): The trigger history data is now available via AnalysisFramework chunk, TriggerActivityChunk. This is follow up from the addition of that trigger history data during the Fall 2003 shutdown. It is available via CalibrateHits by turning on the RCP (logical) parameter for DAQbitFields.
Day 8 GPS bit - (11/25/03): "Day 8" GPS problem fixed. Occasionally, the GPS supplying the time stamp for the DAQ would show a time exactly 8 days in advance. This was found to be caused by the day-8 wire in a twisted pair with a 1kHz latch signal. Since these are TTL, not ECL signals, there is enough noise that it would cause this bit to go high on occasion. This is the only bit with that problem. Gordon has fixed it by shielding it with copper tape. I have fixed it in software so that even if it comes up again, it will be fixed on the fly. streamer has been altered to fix the bit if there is a big time jump. InputDAQFormat, with Steve Brice's help, in the AnalysisFramework has the offline version of the same fix. In addition, I have also added code to check the integrity of the GPS milliseconds value. BUT, the only bit that is current fixed (not just checked) is the day-8 bit. The day-8 problem has not recurred since Gordon's fix.
Fall shutdown - (week of 11/7/03): End of Fall shutdown, and end of the main set of improvements to the DAQ code. The main addition was of the trigger history data, along with lots of other improvements. We are now using the DAQ 6 format. The format was changed because we added more data to the trigger header. Here is the worklist for the Fall shutdown. All of this was accomplished, though the actual details differ of how the trigger history data was added.
More disk space on hal9002/4 - (11/03/03): New hard-drives for hal9002/hal9004 installed by me & Chris. After a short time, it was discovered that the data was being corrupted by the new disks. This problem was fixed by Chris Green, and Jon Link.
Scaler DAQ work - (10/29/03): Improvements to the Scaler DAQ code to do more quality checking of the data. Also, if there was a power outage, it would start putting garbage data into the data-base every second! This bug was fixed also.
Fall shutdown work - (9/12/03): We are currently performing planned shutdown improvements to the DAQ. Here is the current worklist, which goes into some detail of the code which will be changed. In addition, the scaler hardware will be installed in late September, or early October, which will be the final stage of the scaler project.
The scaler project is nearly complete. Here is Melanie Novak's talk about her work on this project. It will measure rates of 16 inputs directly from the trigger and 16 from the calibration systems.
Power outage: During the power outage last week (5/27 - 5/30), we did a few things with the DAQ: 1) Made new space in the DAQ rack for a NIM crate as well as the new patch panels to be used as inputs to the scaler. 2) Installed the calibration-to-scaler patch panel (16 signals).
Debuncher: The debuncher trigger signal is now installed and ready to be input into the trigger VME crate. Look in the CRL for my detailed entry on 6/2/03 4:47pm (#7021). This will be better documented into a memo soon. In summary, we are using the E2 input by varying the NIM pulse width to distinguish between strobe and debuncher trigger. For now, only the strobe assertes E2, since the trigger software is not yet ready to handle this new trigger type.
Other new work: In the next few days, I will add the horn sync pulse to channel 1546, currently occupied by the 90 degree monitor. Also in the upcoming weeks, I will add functionality to the Analysis Framework to give easy access to the DAQ bitfields. Here is the new bitfields spreadsheet used as input to create the new bitfields data-base, boodb_daq_bitmasks. This is essential for some of the data-quality analyses and plots that are planned to be used to better monitor the DAQ.
Network health: I ran down the documentation for the SMC router that is currently used for the DAQ private LAN (gets data from monoboards to hal9000). I was interested in seeing if there was a way to logon to the router to get statistics. After contating the vendor, it turns out that this is not available, since we have an "unmanaged switch." Presumably, the managed flavor allows this. This idea was brought on because I have been suspecting occasional network problems, and I would like to know if the switch is at capacity, or has having other problems. There is no direct way to know this. I was advised to look around for private software. I have not pursued it further.
Run restart problems, QT half-full: As of May 8th, the problems with frequent run-restarts due to the trigger detecting QT half-full have subsided. This problem seems to be intermittent, since it is still seen at a lower level, even though runs are lasting pretty much the maximum time (8 hours -- stopped by the shifters). If the QT half-full problem starts to be a problem again, Shawn and I have added some debug code (in early May) to try and trap the problem. This is not the only idiosyncracy, however. Another problem is that when runs need to restart (for whatever reason), one of the monoboards was "out to lunch," meaning that the controller program, halTalk, could not issue commands to it. QT6 was the biggest culprit, but by simply jiggling the board to ensure its connection to the VME backplane, and by jiggling the ethernet connections to it, QT6 now responds normally.
(4/11/03): There is a new proposal by Bonnie Fleming to add a debuncher trigger. Here is her memo, and the talk I gave on it at the last BooNE meeting.
Melanie Novak and Andrew Green are creating a VME Scalar counter which will count up to 16 NIM, and 16 TTL pulses. The scalar will count the number of beam pulses, calibration pulses of various types, trigger comparator thresholds, etc. A fuller list is in the the talk I gave on 4/11 at the FNAL BooNE meeting.
The new triggers that were created during the January shutdown have been operating well. See the new trigger document v2.0 for details on the new triggers. You can also refer to them directly: event_types.h, trigger_loop.c, trigger.h
There was an issue with latency about a week after the shutdown that was solved. Latency is now monitored in shared memory, and is available to the online GUI so the shifters can see if there is a problem with it. It was always there before, but we were not monitoring it. The average latency is 0.3% overall trigger types. Beam event latency is about 1/2000 events or 0.05%.
(4/21/03): The new monoboards that Nate ordered have been reconfigured for (almost) plug and play if any of the DAQ monoboards fail. Please note that only an electronics/DAQ expert is allowed to make a monoboard switch. See the list below for contacts.
(4/16/03): Morgan has slightly changed the channel assignments of the LSU tracking PMTs into the modified QT board in crate 13 on 4/14/03 due to a readout problem with 16 tracker channels. His change has somehow resulted in the DAQ runs lasting longer. It is not entirely clear why this is the case, but there are almost no QT HF signals reaching the trigger now. The trigger goes through far fewer PAUSE/RESUME cycles than before 4/14. The trigger has a ~1/20 chance of failing the RESUME broadcast, so the more RESUMEs we have, the more frequently the run restarts. So, this change to QT 13 has indirectly affected the DAQ in a favorable way. The CRL entry for that has more details.
(12/11/2002) Recent changes in the past month have reduced the amount of downtime between runs to ~40 sec (or frequently less). Plans for the January shutdown include reducing that time significantly. The system runs very well overall.
An new memo
has been posted which addresses the offset peak near the beginning of
the trigger window for E-bit event types.
Current problems (seen regularly by shifters):
Trigger hardware occasionally produces a bad broadcast. The system currently responds to this by restarting the run. The only solution to this is to wait for new trigger hardware.
QT FIFOs full in crate 13. Another shutdown task is to try and understand this in addition to the occasional QT FIFO sync problem in a few channels in crates 8, 12,13. The channels listed in PAUSE events contain
On run-restart, there is a delay every 10 runs or so caused by a monoboard not being able to establish a socket connection to halTalk, Shawn's run control program. The problem will be partially addressed during the January shutdown.
DAQ Plan for the January shutdown (1/13/2003 - 2/5/2003):
(this list is a subset of the DAQ To-do list, and is prioritized.)
Coordinate with electronics and calibration groups. Electronics repair has priority overall. Obviously, much of the work below may be done after-hours.
(.5 days) Study the SN rate more fully, and finalize the supernova trigger (if not already done).
(4 days) Complete remaining trigger software work (Rex):
Devise a gamma trigger.
Special signals from the calibration system to the trigger for signaling of different calibration event types (all using the E3 bit).
Test any new trigger hardware that could become available.
Further trigger studies as needed.
(2 days) Create running mode where assembler, QT & trigger keep alive between runs when a "New Run" command is issued. This will involve changes to trigger and assembler, but probably not the QT code. (Andrew).
(4 days) Commision Shawn's uberDAQ, which uses sockets to issue DAQ commands (Andrew & Shawn).
(5 days) QT driver, Crate 13, QT FIFO sync issues need to be addressed.
Also, as time allows, work on the assembler-to-QT socket problem (one of the contributors to DAQ downtime.)
Plan for Wednesday 12/12/2002 shutdown (6:00 - 12:30):
Do a supernova trigger study: change DET5 to 40, and use DET5 as the primary trigger on SNs, while leaving DET4 for michel triggers. No other triggers use DET5 currently (Andrew). (done)
Do a high rate study using laser + strobe events. Recall that strobes have the same laser holdoff of 1u-sec (Morgan & Ryan) (done).
Un-mask (in the software) the new crate 13 (ModQT) channels that Morgan has made available. Test them also (Morgan).
Make a CVS tag at the end of the day's work for the DAQ code.