Reduction of flat-imaging

Warning

The following documentation has been prepared executing the commands in a terminal running the bash shell. If you are using the Z shell (that at the time of writing is the default in mac OS) it is important to remember that the wildcards employed in some filabres parameters need to be provided enclosed in double quotes.

Note

Filabres computes master calibration images for each night. Within each night, individual calibration exposures within a given time span (given by the keyword maxtimespan_hours in the file configuration_cafos.yaml) are combined. Note that in order to be included in a particular master calibration, the corresponding individual images should also have the same signature, i.e., the same values for the set of FITS keywords listed under signature in the file configuration_cafos.yaml for the considered image type (bias, flat-imaging,…).

The reduction of flat-imaging files is quite similar to the process followed for the bias images. Note that now the signature of the images also depends on the position of the grism wheel (keyword INSGRID; no grism for imaging mode), the filter employed during the observation (keyword INSFLID), and the absence of the wollaston prism required for imaging polarimetry (keywords INSPOFPI and INSPOROT).

$ filabres -rs flat-imaging
* Number of nights found: 58

* Working with night 170225_t2_CAFOS (1/58)
---
Working with signature SITE#1d_15__1650__1650__[251,221:1900,1870]__1__1__GRISM-11__FILT- 9__FREE__0
Creating flat-imaging/170225_t2_CAFOS/flat-imaging_caf-20170224-20:20:04-cal-krek_red.fits
Creating flat-imaging/170225_t2_CAFOS/flat-imaging_caf-20170224-20:20:04-cal-krek_mask.fits
---
Working with signature SITE#1d_15__1650__1650__[251,221:1900,1870]__1__1__GRISM-11__FILT- 9__FREE__0
Creating flat-imaging/170225_t2_CAFOS/flat-imaging_caf-20170226-06:24:27-cal-krek_red.fits
Creating flat-imaging/170225_t2_CAFOS/flat-imaging_caf-20170226-06:24:27-cal-krek_mask.fits
---
Working with signature SITE#1d_15__1650__1650__[251,221:1900,1870]__1__1__GRISM-11__FILT-10__FREE__0
Creating flat-imaging/170225_t2_CAFOS/flat-imaging_caf-20170224-20:49:51-cal-krek_red.fits
Creating flat-imaging/170225_t2_CAFOS/flat-imaging_caf-20170224-20:49:51-cal-krek_mask.fits
...
...
* Working with night 171228_t2_CAFOS (57/58)
---
Working with signature SITE#1d_15__1700__1700__[201,201:1900,1900]__1__1__GRISM-11__FILT- 5__FREE__0
Creating flat-imaging/171228_t2_CAFOS/flat-imaging_caf-20171228-13:14:11-cal-bard_red.fits
Creating flat-imaging/171228_t2_CAFOS/flat-imaging_caf-20171228-13:14:11-cal-bard_mask.fits

* Working with night 171230_t2_CAFOS (58/58)
---
Working with signature SITE#1d_15__800__800__[601,601:1400,1400]__1__1__GRISM-11__FILT-11__FREE__0
Creating flat-imaging/171230_t2_CAFOS/flat-imaging_caf-20171229-10:04:54-cal-lilj_red.fits
Creating flat-imaging/171230_t2_CAFOS/flat-imaging_caf-20171229-10:04:54-cal-lilj_mask.fits
---
Working with signature SITE#1d_15__800__800__[601,601:1400,1400]__1__1__GRISM-11__FILT-11__FREE__0
Creating flat-imaging/171230_t2_CAFOS/flat-imaging_caf-20171231-06:30:10-cal-lilj_red.fits
Creating flat-imaging/171230_t2_CAFOS/flat-imaging_caf-20171231-06:30:10-cal-lilj_mask.fits
* program STOP

Several warning messages may appear during the reduction of these images (ignore them).

Note that within each night one (or several) master flat-imaging frames (and their associated mask images) are created. The information on the terminal indicates the corresponding signature.

The master flat-imaging frames are stored in the subdirectory flat-imaging under the current directory:

$ tree flat-imaging
flat-imaging/
├── 170225_t2_CAFOS
│   ├── flat-imaging_caf-20170224-20:20:04-cal-krek_mask.fits
│   ├── flat-imaging_caf-20170224-20:20:04-cal-krek_red.fits
│   ├── flat-imaging_caf-20170224-20:20:04-cal-krek_red.log
│   ├── flat-imaging_caf-20170224-20:49:51-cal-krek_mask.fits
│   ├── flat-imaging_caf-20170224-20:49:51-cal-krek_red.fits
│   ├── flat-imaging_caf-20170224-20:49:51-cal-krek_red.log
│   ├── flat-imaging_caf-20170224-21:12:37-cal-krek_mask.fits
│   ├── flat-imaging_caf-20170224-21:12:37-cal-krek_red.fits
│   ├── flat-imaging_caf-20170224-21:12:37-cal-krek_red.log
│   ├── flat-imaging_caf-20170225-18:28:50-cal-krek_mask.fits
│   ├── flat-imaging_caf-20170225-18:28:50-cal-krek_red.fits
│   ├── flat-imaging_caf-20170225-18:28:50-cal-krek_red.log
│   ├── flat-imaging_caf-20170226-06:05:08-cal-krek_mask.fits
│   ├── flat-imaging_caf-20170226-06:05:08-cal-krek_red.fits
│   ├── flat-imaging_caf-20170226-06:05:08-cal-krek_red.log
│   ├── flat-imaging_caf-20170226-06:24:27-cal-krek_mask.fits
│   └── flat-imaging_caf-20170226-06:24:27-cal-krek_red.fits
│   └── flat-imaging_caf-20170226-06:24:27-cal-krek_red.log
├── 170403_t2_CAFOS
│   ├── flat-imaging_caf-20170403-17:19:21-cal-lilj_mask.fits
│   ├── flat-imaging_caf-20170403-17:19:21-cal-lilj_red.fits
│   ├── flat-imaging_caf-20170403-17:19:21-cal-lilj_red.log
│   ├── flat-imaging_caf-20170403-18:44:42-cal-lilj_mask.fits
│   └── flat-imaging_caf-20170403-18:44:42-cal-lilj_red.fits
│   └── flat-imaging_caf-20170403-18:44:42-cal-lilj_red.log
...
...
├── 171228_t2_CAFOS
│   ├── flat-imaging_caf-20171228-13:14:11-cal-bard_mask.fits
│   └── flat-imaging_caf-20171228-13:14:11-cal-bard_red.fits
│   └── flat-imaging_caf-20171228-13:14:11-cal-bard_red.log
└── 171230_t2_CAFOS
    ├── flat-imaging_caf-20171229-10:04:54-cal-lilj_mask.fits
    ├── flat-imaging_caf-20171229-10:04:54-cal-lilj_red.fits
    ├── flat-imaging_caf-20171229-10:04:54-cal-lilj_red.log
    ├── flat-imaging_caf-20171231-06:30:10-cal-lilj_mask.fits
    └── flat-imaging_caf-20171231-06:30:10-cal-lilj_red.fits
    └── flat-imaging_caf-20171231-06:30:10-cal-lilj_red.log

If you want to get more information concerning the reduction of these type of images, just add -v to increase the verbosity level. For example, we can try to repeat the reduction of the night 171228_t2_CAFOS:

$ filabres -rs flat-imaging -n 171228* -v
* instrument: cafos
* datadir: /Volumes/NicoPassport/CAHA/CAFOS2017
* ignored_images_file: ignored_images.yaml
* image_header_corrections_file: image_header_corrections.yaml
* forced_classifications_file: forced_classifications.yaml
* Loading instrument configuration
* Number of nights found: 1
* List of nights: ['171228_t2_CAFOS']

Results database set to filabres_db_cafos_flat-imaging.json

Subdirectory flat-imaging found
maxtimespan_hours: 1

* Working with night 171228_t2_CAFOS (1/1)
Reading file ./lists/171228_t2_CAFOS/imagedb_cafos.json
Number of flat-imaging images found 10
Subdirectory flat-imaging/171228_t2_CAFOS found
Number of different signatures found: 1
Signature (1/1):
 - CCDNAME: SITE#1d_15
 - NAXIS1: 1700
 - NAXIS2: 1700
 - DATASEC: [201,201:1900,1900]
 - CCDBINX: 1
 - CCDBINY: 1
 - INSGRID: GRISM-11
 - INSFLID: FILT- 5
 - INSPOFPI: FREE
 - INSPOROT: 0
Total number of images with this signature: 10
-> Number of images with expected signature and within time span: 10
File flat-imaging/171228_t2_CAFOS/flat-imaging_caf-20171228-13:14:11-cal-bard_red.fits already exists: skipping reduction.
* program STOP

Since filabres has detected that the output image already exists, the output file is not overwritten. You can force to overwrite the output file by using the additional argument --force in the command line:

$ filabres -rs flat-imaging -n 171228* -v --force
* instrument: cafos
* datadir: /Volumes/NicoPassport/CAHA/CAFOS2017
* ignored_images_file: ignored_images.yaml
* image_header_corrections_file: image_header_corrections.yaml
* forced_classifications_file: forced_classifications.yaml
* Loading instrument configuration
* Number of nights found: 1
* List of nights: ['171228_t2_CAFOS']

Results database set to filabres_db_cafos_flat-imaging.json

Subdirectory flat-imaging found
maxtimespan_hours: 1

* Working with night 171228_t2_CAFOS (1/1)
Reading file ./lists/171228_t2_CAFOS/imagedb_cafos.json
Number of flat-imaging images found 10
Subdirectory flat-imaging/171228_t2_CAFOS found
Number of different signatures found: 1

Signature (1/1):
 - CCDNAME: SITE#1d_15
 - NAXIS1: 1700
 - NAXIS2: 1700
 - DATASEC: [201,201:1900,1900]
 - CCDBINX: 1
 - CCDBINY: 1
 - INSGRID: GRISM-11
 - INSFLID: FILT- 5
 - INSPOFPI: FREE
 - INSPOROT: 0
Total number of images with this signature: 10
---
-> Reduction starts at.: 2020-03-26 16:56:23.128253
Working with signature SITE#1d_15__1700__1700__[201,201:1900,1900]__1__1__GRISM-11__FILT- 5__FREE__0
-> Number of images with expected signature and within time span: 10
 - /Volumes/NicoPassport/CAHA/CAFOS2017/171228_t2_CAFOS/caf-20171228-13:14:11-cal-bard.fits
 - /Volumes/NicoPassport/CAHA/CAFOS2017/171228_t2_CAFOS/caf-20171228-13:15:44-cal-bard.fits
 - /Volumes/NicoPassport/CAHA/CAFOS2017/171228_t2_CAFOS/caf-20171228-13:17:17-cal-bard.fits
 - /Volumes/NicoPassport/CAHA/CAFOS2017/171228_t2_CAFOS/caf-20171228-13:18:51-cal-bard.fits
 - /Volumes/NicoPassport/CAHA/CAFOS2017/171228_t2_CAFOS/caf-20171228-13:20:24-cal-bard.fits
 - /Volumes/NicoPassport/CAHA/CAFOS2017/171228_t2_CAFOS/caf-20171228-13:21:57-cal-bard.fits
 - /Volumes/NicoPassport/CAHA/CAFOS2017/171228_t2_CAFOS/caf-20171228-13:23:31-cal-bard.fits
 - /Volumes/NicoPassport/CAHA/CAFOS2017/171228_t2_CAFOS/caf-20171228-13:25:05-cal-bard.fits
 - /Volumes/NicoPassport/CAHA/CAFOS2017/171228_t2_CAFOS/caf-20171228-13:26:39-cal-bard.fits
 - /Volumes/NicoPassport/CAHA/CAFOS2017/171228_t2_CAFOS/caf-20171228-13:28:12-cal-bard.fits
-> Output fname will be: flat-imaging/171228_t2_CAFOS/flat-imaging_caf-20171228-13:14:11-cal-bard_red.fits
Deleting flat-imaging/171228_t2_CAFOS/flat-imaging_caf-20171228-13:14:11-cal-bard_red.fits
Deleting flat-imaging/171228_t2_CAFOS/flat-imaging_caf-20171228-13:14:11-cal-bard_mask.fits
WARNING: deleting previous database entry: flat-imaging --> SITE#1d_15__1700__1700__[201,201:1900,1900]__1__1__GRISM-11__FILT- 5__FREE__0 --> 58115.55635

Calibration database set to filabres_db_cafos_bias.json
-> looking for calibration bias with signature SITE#1d_15__1700__1700__[201,201:1900,1900]__1__1
->   mjdobsarray.......: [57905.6372  58102.60078 58105.56892 58108.57108 58111.0489  58112.72261
 58112.82979 58115.568  ]
->   looking for mjdobs: 58115.5515
->   nearest value is..: 58115.56800
->   delta_mjd (days)..: 0.016499999997904524
Median value in frame #1/10: 28403.5
Median value in frame #2/10: 28546.0
Median value in frame #3/10: 28278.5
Median value in frame #4/10: 28265.0
Median value in frame #5/10: 28226.0
Median value in frame #6/10: 28062.5
Median value in frame #7/10: 28058.0
Median value in frame #8/10: 28298.5
Median value in frame #9/10: 28038.5
Median value in frame #10/10: 28083.0
Creating flat-imaging/171228_t2_CAFOS/flat-imaging_caf-20171228-13:14:11-cal-bard_red.fits
Creating flat-imaging/171228_t2_CAFOS/flat-imaging_caf-20171228-13:14:11-cal-bard_mask.fits
Creating flat-imaging/171228_t2_CAFOS/flat-imaging_caf-20171228-13:14:11-cal-bard_red.log
-> Reduction ends at...: 2020-03-26 16:56:31.938600
-> Time span...........: 0:00:08.810347
* program STOP

For this particular night, all the flat-imaging files exhibit a single signature. The 10 available individual frames were obtained within one hour. For that reason all of them are selected to be combined in a single master flat-imaging frame. The name of the output file is taken from the first image in the sequence of 10 images, adding the prefix flat-imaging_ and the suffix _red (the latter prior to the extension .fits). A log file with the same name as the output file, but with the extension .log, is also generated.

An additional output file, containing a mask of useful pixels, is also generated, using the same file name but changing the suffix _red by _mask. In this mask a value of 0 is assigned to pixels without useful signal (probably due to vignetting), whereas a value of 1 is employed for the pixels in the useful image region.

Note that the reduction of the flat-imaging files requires the use of a master bias with a particular signature, in this case SITE#1d_15__1700__1700__[201,201:1900,1900]__1__1, which is compatible with the signature of the considered flat-imaging files: SITE#1d_15__1700__1700__[201,201:1900,1900]__1__1__GRISM-11__FILT- 5__FREE__0 (the bias signature does not depend on the use of grism and/or filters). Several master bias frames with this signature are found, each one with a different modified Julian Date. The selected one is the closest, in time, with the observing time of the flat images. In addition, the median signal in each individual exposure is also shown (note that this value is computed in the useful image region, where the computed mask is different from zero).

Database of master flat-imaging frames

The reduction of the flat-imaging files generates a file, placed in the current directory, called filabres_db_cafos_flat-imaging.json. This constitutes a database with the information of all the flat-imaging images, sorted by signature and, within each signature, sorted by the Modified Julian Date (FITS keyword MJD-OBS). In this way, when a master flat-imaging is needed in the reduction of a scientific image, filabres can determine the required calibration signature and then select the closest calibration to the corresponding observation time.

The structure of filabres_db_cafos_flat-imaging.json is similar to the one previously explained for filabres_db_cafos_bias.json in the section Database of master bias frames, and is not going to be repeated here.

Checking the flat-imaging reduction

In order to obtain a list with al the reduced flat-imaging frames just execute:

$ filabres -lr flat-imaging
                                                                                 file
1   flat-imaging/170225_t2_CAFOS/flat-imaging_caf-20170224-20:20:04-cal-krek_red.fits
2   flat-imaging/170225_t2_CAFOS/flat-imaging_caf-20170226-06:24:27-cal-krek_red.fits
3   flat-imaging/170225_t2_CAFOS/flat-imaging_caf-20170224-20:49:51-cal-krek_red.fits
...
...
84  flat-imaging/171225_t2_CAFOS/flat-imaging_caf-20171225-17:31:09-cal-bard_red.fits
85  flat-imaging/171225_t2_CAFOS/flat-imaging_caf-20171225-20:09:53-cal-bard_red.fits
86  flat-imaging/171228_t2_CAFOS/flat-imaging_caf-20171228-13:14:11-cal-bard_red.fits
Total: 86 files

The available keywords for this type of images are:

(filabfes) $ filabres -lr flat-imaging -k all
Valid keywords: ['NAXIS', 'NAXIS1', 'NAXIS2', 'OBJECT', 'RA', 'DEC',
'EQUINOX', 'DATE', 'MJD-OBS', 'AIRMASS', 'EXPTIME', 'INSTRUME', 'CCDNAME',
'ORIGSECX', 'ORIGSECY', 'CCDSEC', 'BIASSEC', 'DATASEC', 'CCDBINX',
'CCDBINY', 'IMAGETYP', 'INSTRMOD', 'INSAPID', 'INSTRSCL', 'INSTRPIX',
'INSTRPX0', 'INSTRPY0', 'INSFLID', 'INSFLNAM', 'INSGRID', 'INSGRNAM',
'INSGRROT', 'INSGRWL0', 'INSGRRES', 'INSPOFPI', 'INSPOROT', 'INSFPZ',
'INSFPWL', 'INSFPDWL', 'INSFPORD', 'INSCALST', 'INSCALID', 'INSCALNM',
'NPOINTS', 'FMINIMUM', 'QUANT025', 'QUANT159', 'QUANT250', 'QUANT500',
'QUANT750', 'QUANT841', 'QUANT975', 'FMAXIMUM', 'ROBUSTSTD', 'NORIGIN',
'IERR_BIAS', 'DELTA_MJD_BIAS', 'BIAS_FNAME', 'IERR_FLAT']

Note some new useful keywords:

  • IERR_BIAS: flag that indicates whether there was a problem when trying to retrieve the master bias frame corresponding to the signature of the flat images. The value 0 means that the master bias was found, whereas a value of 1 indicates that no master bias was found with the requested signature (in this case, the median value of the closest bias is chosen, independently of its signature).

  • DELTA_MJD_BIAS: time distance (days) between the master bias and the flat images being reduced.

  • BIAS_FNAME: path to the master bias image employed in the reduction of the flat images.

  • IERR_FLAT: flag that indicates a problem in the reduction of the flat images themselves (a negative median signal for example). These images should be revised.

For example, it is possible to quickly determine if IERR_BIAS or IERR_FLAT are different from zero in any of the reduced flat-imaging frames:

$ filabres -lr flat-imaging --filter 'k[ierr_bias] != 0'
Total: 0 files
$ filabres -lr flat-imaging  --filter 'k[ierr_flat] != 0'
Total: 0 files

None of the reduced flat-imaging frames has had any problem in the reduction process.

It is also useful to examine some statistical parameters of the reduced images:

$ filabres -lr flat-imaging -k quant250 -k quant500 -k quant750 -k robuststd -pxy
...
...
Reduced flat-imaging summary

We find that all the reduced flat-imaging frames exhibit the expected statistical behavior

Removing invalid reduced flat-imaging

In this case there is no apparent reason to remove any of the reduced flat-imaging frames. If that were the case, the method would be similar to that described in section Removing invalid reduced bias for the reduced master bias images.