CORRELATOR

Combining MWA data

The correlation scheme in the MWA is of the "FX" variety, in which the voltage samples from the antennas are first Fourier transformed (F) into the frequency domain, then are cross-multiplied (X) pairwise and integrated over time.  The sequence is:

  1. Each physical receiver digitises the signals from eight physical tiles in X and Y, at 655.36 Msamples/sec.
  2. A Polyphase Filter Bank (PFB) implemented using FPGAs inside each receiver splits the data into 256 'coarse channels', each 1.28 MHz wide.
  3. An arbitrary set of 24 of those coarse channels are transferred via fibre to the 'fine PFB' hardware - four FPGA cards, each of which handles the inputs from four receivers.
  4. The 'fine PFB' cards split each coarse channel into 128 'fine channels', each 10 kHz wide. Each fine PFB card has two output fibres, carrying all of these fine channels to sixteen 'media converter' boxes, that translate the custom fibre protocol (RocketIO) to UDP over ethernet.
  5. Each of the sixteen 'media converter' computers maintains TCP connections with each of the 24 'GPU boxes' that do 'X stage' of the correlator (cross-multiply and accumulate). Each GPU box handles one coarse channel, so the correlator 'corner turn' happens in the network switch between the media converter computers and the 'GPU box' computers.
  6. The 'GPU boxes' implement an XGPU based correlator that has no knowledge of the telescope geometry or current beam pointing direction, so it phases all observations to the zenith. Raw visibilities are archived to tape at the Pawsey Supercomputer Centre.

Because the correlator does not track the actual source location, the size of the array and the rotation of the Earth limits the amount of frequency and time averaging that can be done inside the correlator. In compact mode (all antenna within a ~600 m diameter), we are limited to 40 kHz or narrower channel averaging, and no more than 2.0 seconds of time averaging. In 'long baseline' mode, with all antenna inside a ~6 km diameter, we are limited to no more than 10 kHz and 0.5 second (sixteen times the output data rate).

Producing an image from these visibility files requires the data files themselves, plus a 'metafits file' describing the telescope configuration during that observation - including pointing direction, choice of coarse channels, tile locations, cable lengths, etc. This is done by the science teams - the data archive for the telescope consists of 'visibility files', not images.

A new correlator is currently being developed. The main differences are:

  • It will use sixteen 'media converter' computers to read data directly from the receivers in coarse 1.28 MHz channels, avoiding the fine PFB entirely.
  • It will use source position, tile locations, cable lengths, etc, to phase the visibility files to the actual source location. This will allow much more frequency and time integration inside the correlator, saving disk space in the telescope archive.
More information about the correlator is available here: MWA Correlator Paper