In order to keep the science community up-to-date with current events, we recently resumed publishing the ASKAP commissioning newsletter. This monthly summary is designed to give the science teams a better idea of the state of the telescope as we progress through the challenges associated with early science operations. It will be posted at least a week before the early science forum teleconference, to give everyone time to prepare questions and feedback. All issues can be read at the following URL:
The Pawsey centre has finished commissioning a new Lustre file system connected to the Galaxy supercomputer. 1 PB of space on this new system has been allocated to the real-time ingest of data from ASKAP, using a dedicated metadata server that should avoid load issues due to multi-user access patterns that may have impacted the performance of the real-time system in the past. For now, the intermediate products of the science data pipeline will still be written to the older /scratch2 filesystem, but we are negotiating how to best proceed when /scratch2 is decommissioned later in the year.
The seized azimuth gearbox on ak01 was recently swapped for a working unit from one of the other antennas that does not currently have a PAF installed. This process provided an opportunity for the MRO site mechanical engineering staff to gain direct experience with the gearbox mechanisms, a first since they were installed by the manufacturer several years ago. The failed unit will be sent for inspection and rework while CASS is in the process of sourcing spare parts for the future. In response to the need for more regular monitoring of drive train health, we have developed a standardised torque test program that drives all operational antennas through a predetermined sequence. Subsequent analysis of the electrical current required to drive each of the motors during this test can provide early warning of excessive binding in the drive train. As a result of these tests, some adjustments on the meshing of other gearboxes has been performed, but overall the remaining antennas seem to be in a good state.
During the last week, work progressed on the installation of the 1.6MW Solar photo-voltaic array that will expand the existing MRO power station's clean energy capabilities. It was a big week for the power station with the arrival of the containerised electrical systems, including the RFI-shielded battery containers. In addition, a high voltage cable was run between the Solar array and the Horizon Power station. During the mains outage, electrical work was carried out on antennas AK24 and AK27, progressing a gradual upgrade of all antenna switchboards to enable better remote power monitoring and circuit breaker control. The power outage also allowed work on several of the high voltage transformers located at each antenna pedestal and replacement of two high voltage RFI filters that allow the mains power to cross between the interior and exterior of the central control building without providing a path for RFI.
Extensive testing by engineers at the MRO has found the cause of repeated command and control system failures on antenna 16's PAF - one of the internal low-voltage AC power regulator cables had been snagged during installation, eventually creating a short circuit. This has been rectified and the antenna is receiving a final set of spare parts before it is re-commissioned into the array.
Over the past two weeks, several CASS engineering and commissioning team members have been working closely together in an effort to understand the issues that have been impacting the stability and quality of correlator output data. These issues have several symptoms, including inconsistencies in the bandpass (blocks of several channels with different phase or amplitude than expected), the appearance of narrow spikes that don't seem to be caused by radio frequency interference and loss of packets flowing between the digital firmware and the data ingest servers.
Though many of these issues have been known for some time, the situation worsened earlier in the year when the loss of data packets between the correlator output and the software ingest pipeline became so frequent that we could not continue operating. Although this occurred around the same time as several firmware and software changes, the problem has proven highly variable and difficult to track down to any particular cause.
The recent effort involved bringing together several people working on different aspects of the system to discuss the common control interface and the differences between the main system and the tests applied in the laboratory when hardware is received back from the manufacturer. Although investigations are ongoing, CASS engineers now have several leads to follow and will make the resolution of these issues their top priority over the coming weeks.
There has been excellent progress on installation of the ROAR boards (part of the on-dish calibration system, or ODC) with 20 antennas now completed. The commissioning of these ODC systems is progressing well, with integration testing revealing some inconsistencies between the software drivers and the final hardware revision that have since been fixed. We are also commissioning a "calibration correlator" module in the beamformer firmware that sits alongside the array co-variance matrix (ACM) engine and computes a single row of correlations (using a reference port that can be chosen at runtime) with full duty cycle. This is a small fraction of the computational load of the full ACM and the data file it produces is also much smaller, which should enable regular monitoring of the complex gains of all PAF elements.
At present, we are trying to accelerate the commissioning of these ODC systems on the main array so that they can be used to help reduce beamforming overheads during early science observations when they can resume.
Preparation of the vertex optical fibre link during ODC installation
Our first production unit for the Upgrade Project; PAF #33; is undergoing functional testing at the EMC chamber following removal of PAF #32. Stability testing using the linear power regulators inside the EMC chamber will firstly be conducted followed by temperature ramp testing. We expect to have the balance of dominos for the remaining PAFs in hand by the end of May. Full requirements of power bulkheads are going through surface mount and hand assembly and may be ready for pick up next week. This will leave only the PAF controllers to be able to complete the electronic assembly of all PAFs.
Currently there are an additional three front-end control modules in working order and the plan is to ship a set of three completed PAFs for installation some time around September this year.