Extensive testing by engineers at the MRO has found the cause of repeated command and control system failures on antenna 16's PAF - one of the internal low-voltage AC power regulator cables had been snagged during installation, eventually creating a short circuit. This has been rectified and the antenna is receiving a final set of spare parts before it is re-commissioned into the array.
Over the past two weeks, several CASS engineering and commissioning team members have been working closely together in an effort to understand the issues that have been impacting the stability and quality of correlator output data. These issues have several symptoms, including inconsistencies in the bandpass (blocks of several channels with different phase or amplitude than expected), the appearance of narrow spikes that don't seem to be caused by radio frequency interference and loss of packets flowing between the digital firmware and the data ingest servers.
Though many of these issues have been known for some time, the situation worsened earlier in the year when the loss of data packets between the correlator output and the software ingest pipeline became so frequent that we could not continue operating. Although this occurred around the same time as several firmware and software changes, the problem has proven highly variable and difficult to track down to any particular cause.
The recent effort involved bringing together several people working on different aspects of the system to discuss the common control interface and the differences between the main system and the tests applied in the laboratory when hardware is received back from the manufacturer. Although investigations are ongoing, CASS engineers now have several leads to follow and will make the resolution of these issues their top priority over the coming weeks.
There has been excellent progress on installation of the ROAR boards (part of the on-dish calibration system, or ODC) with 20 antennas now completed. The commissioning of these ODC systems is progressing well, with integration testing revealing some inconsistencies between the software drivers and the final hardware revision that have since been fixed. We are also commissioning a "calibration correlator" module in the beamformer firmware that sits alongside the array co-variance matrix (ACM) engine and computes a single row of correlations (using a reference port that can be chosen at runtime) with full duty cycle. This is a small fraction of the computational load of the full ACM and the data file it produces is also much smaller, which should enable regular monitoring of the complex gains of all PAF elements.
At present, we are trying to accelerate the commissioning of these ODC systems on the main array so that they can be used to help reduce beamforming overheads during early science observations when they can resume.
Preparation of the vertex optical fibre link during ODC installation
Our first production unit for the Upgrade Project; PAF #33; is undergoing functional testing at the EMC chamber following removal of PAF #32. Stability testing using the linear power regulators inside the EMC chamber will firstly be conducted followed by temperature ramp testing. We expect to have the balance of dominos for the remaining PAFs in hand by the end of May. Full requirements of power bulkheads are going through surface mount and hand assembly and may be ready for pick up next week. This will leave only the PAF controllers to be able to complete the electronic assembly of all PAFs.
Currently there are an additional three front-end control modules in working order and the plan is to ship a set of three completed PAFs for installation some time around September this year.