Over the past two weeks, several CASS engineering and commissioning team members have been working closely together in an effort to understand the issues that have been impacting the stability and quality of correlator output data. These issues have several symptoms, including inconsistencies in the bandpass (blocks of several channels with different phase or amplitude than expected), the appearance of narrow spikes that don't seem to be caused by radio frequency interference and loss of packets flowing between the digital firmware and the data ingest servers.
Though many of these issues have been known for some time, the situation worsened earlier in the year when the loss of data packets between the correlator output and the software ingest pipeline became so frequent that we could not continue operating. Although this occurred around the same time as several firmware and software changes, the problem has proven highly variable and difficult to track down to any particular cause.
The recent effort involved bringing together several people working on different aspects of the system to discuss the common control interface and the differences between the main system and the tests applied in the laboratory when hardware is received back from the manufacturer. Although investigations are ongoing, CASS engineers now have several leads to follow and will make the resolution of these issues their top priority over the coming weeks.