MagE Pipeline Notes

Various notes to myself.

Reporting v1.01 (applies to earlier version of MIKE Pipeline)

In principle, when a pipeline is run, reports will be generated and optionally uploaded to the Plone Server on Shimura. In order to run just the report functions on an already reduced data set the pipeline makefiles need to be changed. So by running mikesetup again on your database file and specifying the targets and bands.

If you want your reports and *multi.fits file uploaded to the sever, add the option -owner USERNAME, or whatever you want your folder to be called on the Plone server. By default, running the pipeline will not upload to Shimura.

Note: By doing this if you need to rerun your data set stage-reporting will run again. Currently the default option runs the stage-reporting. This command is only needed if this is an old data set being run. 

<csh prompt> mikesetup -db march04MIKE.db -red -all -owner gwalth

To run just the stage-reporting, type: 

<csh prompt> make stage-reporting

When finished testing a data set and you wish to remove the stage-reporting files, type: 

<csh prompt> mikeRemove.csh

First run debugging for jfulb... (needs updating)

First off, to do separate extractions for each exposure essentially means that multiple exposures of same target need to be reduced individually, which means they need to be distinguished from one another in the database file.

Initially separate extractions were not supported, we run into problems when we run one exposure through the rebinning/extraction process. What happened is artifacts in the final spectra would appear.

More debugging...

An error produced by stage-wdist.

The python script mikeMatchLamps produces the following error output:

Traceback (most recent call last):
  File "/data1/mifune/kelson/Python.Linux/Basics/mikeMatchLamps", line 471, in ?
    tck = splrep(al,specs[jap],s=0,k=1)
  File "/data1/mifune/kelson/Python.Linux/lib/python2.3/site-packages/multipack/fitpack.py", line 214, in _curfit
    t,c,o = _fitpack._curfit(x,y,w,xb,xe,k,task,s,t,nest,wrk,iwrk,per)
SystemError: error return without exception set
make[1]: *** [stage-wdist] Error 1
make[1]: Leaving directory `/data1/hidari/gwalth/MIKE/PipelineData/jfulb/n2/hd104893blue'

Looking at the scatter of the output it appears to be all over the place. For the scatter the standard deviation of first value is in Angstroms and the second value is in pixels. Typically blue frames should be around 0.003 Angstroms and red frames should be 0.004 or 0.005 Angstroms.

Initially we tried playing with the value of mikeMatchLamps exhaustively without any solution in sight. Eventually we focused our efforts on mikeFindLines.

mikeFindLines hd104893blue_lamp2052fsspecs.fits -fwhm 2.500000 -th 10.000000

By changing the threshold value of mikeFindLines to 25 seemed to solve the problem.

mikeFindLines hd104893blue_lamp2052fsspecs.fits -fwhm 2.500000 -th 25.000000

If this becomes a consistent problem, we may change the default value from 10 to 25.

Noise Statistics (in progress...)

Within the reports the 25th, 50th, and 75th percentiles of the signal-to-noise are plotted versus orders. There was some question if estimated noise was correctly being calculated. So we employed another script called mikeStatistics.py which determines the ratio of actual noise to the estimated noise. 

Observed object = Intrinsic, noiseless spectrum + Actual noise
A rough estimate of the real noise can be determined by taking the observed object data and subtracting it by the median of the observed object. 
g = VTKMedian(obs_obj,w,1,numret=1)  where w is 31,17, and 7
w being the box size of the median smoothing.

checknoise and beyond...

Dan created a python script in order to plot the noise ratio vs. signal-to-noise. No patterns emerged until he assigned colors to each side of the detector. Dramatically the blue side was scattered by a factor of _blank_ higher than the red side.

Dan was able to determine the problem experienced was with the gain. The gain in the headers of the fits files were incorrect for the blue side and the red. In order to solve this problem, Dan wrote some code that computes the gain of the fits files.

Binning and kurt...

There will cases in which people will bin their data differently. In this particular case most of the data was binned 2x2 except for a few targets which were 1x1. In principle you can get away with not taking 1x1 binned quartz frames as long as you take 2x2 binned quartz frames. How we got around this problem since this is not directly addressed in the MIKE Pipeline is by adding the following lines to the slit*.pipeline Makefiles and fringe.pipeline Makefiles (obviously adjusting the names appropriately).

Under the stage-combine will be something like this... 

stage-combine: stage-overscan
# Combine the Slit Frames.
	echo Stage:  Combine the Slit Frames.
	efits -p slitblue_slit1025.fits slitblue_slit1026.fits slitblue_slit1027.fits slitblue_slit1028.fits slitblue_slit1029.fits slitblue_slit1030.fits slitblue_slit1031.fits slitblue_slit1032.fits slitblue_slit1033.fits slitblue_slit1034.fits 'median([i1,i2,i3,i4,i5,i6,i7,i8,i9,i10])' medianblue.fits

	touch stage-combine
You will need to change it to something like this... 
stage-combine: stage-overscan
# Combine the Slit Frames.
	echo Stage:  Combine the Slit Frames.
	efits -p slitblue_slit1025.fits slitblue_slit1026.fits slitblue_slit1027.fits slitblue_slit1028.fits slitblue_slit1029.fits slitblue_slit1030.fits slitblue_slit1031.fits slitblue_slit1032.fits slitblue_slit1033.fits slitblue_slit1034.fits 'median([i1,i2,i3,i4,i5,i6,i7,i8,i9,i10])' medianblue0.fits
	efits medianblue0.fits "VTKGrowN(i1,2,2,numret=1,interp=1,xshift=-0.5,yshift=-0.5)" medianblue.fits
	addkeys medianblue.fits "binning" "1x1"
	touch stage-combine
IMPORTANT!!! Remember when editing the Makefile, to use a tab instead of spaces!!!

Once you are done with that change, the following changes need to be done in order for the pipeline to run successful:

  1. In slit*.pipeline under stage-orders change findorders from "-len 20" to "-len 40"
  2. In lamp*.pipeline under stage-wdist change the mikeFindLines from "-fwhm 5.000000" to "-fwhm 10.000000"
  3. In science pipelines under stage-wdist change mikeFindLines from "-fwhm 5.000000" to "-fwhm 10.000000"

Really red reductions with mr

Michael's data was taken back Feb 03, and at the time nobody knew how red was red on MIKE, so roughly a third of the detector was wasted on the red end. This cause problems for us because the wavelength calibration gets messed up from missing so many orders. So one of the solutions was adding a parameter (orderspan) to mikesetup in order to only do a section of the data frame to reduce. 

mikesetup -db feb03_mikeMIKE.db -flatkey milkyquartz -orderkey quartz -lampkey thar -fringekey quartz -red -all -mk Makefile -orderspan 0,650
The way you use it is the following: 
orderspan   PIXSTART,PIXEND
So if you want to only do pixels 0-650, then you would add the parameter "-orderspan 0,650" to mikesetup.

Of course in order to solve one problem you end up creating more. When trying to fit to the position of the lines in the lamp the fit blows up on the smaller image size. A way to get around this is to change the orders of the fit from: 

mikeMatchLamps lampred_lampXXXXffspecs.fits -x 5 -o 4
to: 
mikeMatchLamps lampred_lampXXXXffspecs.fits -x 4 -o 3
It is possible that you may need to change the orders of the fit differently for different data and different ranges of data.

Double lamp lines

For MIKE there are different slits all the same length and varying in width. They are on a linear plate that slides from one slit to another. Occasionally the camera will be between two slits and you data will go between both. This is how double lamp and sky lines appear on the frames. How would one correct this? Does this affect the data of the object being observed.

Recent Additions...

Reduction Error Checking

Postscript Checks

gv lampblue/*histo.ps
gv lampred/*histo.ps
gv slitblue/*orders.ps
gv slitred/*orders.ps

*apspos.ps
  • Check for multiple frames for combining "Center Residual." They should be roughly normalized.
  • Scattered plots suggest no continuum or bad combine.
    • *histo.ps
  • Check for good solution to wavelength calibration.
  • Bad scatter due to lamps (maybe).
    • FITS File Checks

    dfits -nj {slitred/medianredf.fits,slitblue/medianbluef.fits}

    Extractions

    In stage-extractall 

    SHOW EXAMPLE OF TO HIGH HY IN LOW S/N
    SHOW EXAMPLE OF GOOD HY

    Andy's Water Vapor data...

    Makefile Nuances
    • If you run mikesetup with a fringe frame and then decide to regenerate without (after running), you should probably remove the directory before regenerating. The fringe frame gets added pretty early in the reduction and it will produce errors.
    • If you run mikesetup with the type set to LO and you want to regenerate with the type set to HI (or vice-versa). You will need to remove the stage-extractall*. Otherwise the Makefile will think it is already completed.
    • If you run mikesetup without the individual extractions and you want to run with them, then you will need to remove the stage-extractall*. Otherwise the Makefile will think it is already completed.

    Misc. documentation items:

    mikedb
    argumentdescriptions
    -d DATADIRdirectory of data
    -db DBNAMEname

    mikesetup
    argumentdescriptions
    -2D*generates 2D spectra, and simple (non-GHLB) 1D extractions
    -allgenerates pipelines for all targets
    -blueoption generates pipelines for blue frames
    -date DATEoption generates pipelines for specific date
    -db DATABASEdatabase file
    -dkx FLOATset non-standard knot spacing for skysub
    -flatblue FLATBLUElocation to a specific flatblue frame
    -flatkey FLATSTRINGstring to search for flat frames in database file
    -flatred FLATREDlocation to a specific flatred frame
    -fringe FRINGElocation to a specific fringe frame
    -fringekey FRINGESTRINGstring to search for fringe frames in database file
    -ignoresets ignore string
    -individualseparate extractions
    -lampblue LAMPBLUElocation to a specific lampblue frame
    -lampkey LAMPSTRINGstring to search for lamp frames in database file
    -lampred LAMPREDlocation to a specific lampred frame
    -logboption bins extracted spectra logarithmically in wavelength
    -mk MAKEFILEcreates a makefile for multiple targets
    -nofakedo not make model spectra (less time to run)
    -noobjects(a) Compute sky masks instead of sky apertures; construct wide/dummy extraction apertures, and in 2D mode do not extract 1D spectra
    -noskyapsDo not compute sky apertures or masks
    -noslitfnoption skips the generation and usage of a slit function
    -onlyflatOnly generate the flatfield pipelines
    -onlylampOnly generate the comparison lamp pipelines
    -orderkey ORDERSTRINGstring to search for order frames in database file
    -orderspanOptional comma-separated pair of column coordinates to specify portion of data frames to be reduced (e.g. To reduce the reddest 500 columns, use "-orderspan 0,500")
    -redoption generates pipelines for red frames
    -rnRead noise, in electrons, if the header values are not to be used
    -skykeyString to identify science frames to be used as sky frames
    -skyshiftUse sky lines to compute zero-point shifts in the red wavelength solutions [unstable]
    -slitblue SLITBLUElocation to a specific slitblue frame
    -slitred SLITREDlocation to a specific slitred frame

    * option is experimental
    science frames
    file extensions descriptions
    f divided bv flatfield
    ff divided by flatfield and fringe frame; i.e. red only, and optional
    s divided by slit-function
    m model of sky
    o (object) model of sky has been subtracted
    p fake frame, GHLB fit to o
    lamp frames
    file extensions descriptions
    f divided flatfielded
    s divided slitfunction
    b spacial filtered to isolated th ar lines
    _scat generated during spacial filtering (temp) not useful
    x-rect frame containing rect transforms (line curvature)
    specs 1d extracted th ar spectra for each order used for wavelength cal.
    diagnostic files
    files descriptions
    blaze{blue,red}.fits flatened quartz spectrum
    flat{blue,red}.fits pixel to pixel flat
    lamp{blue,red}.fits
    mask{blue,red}.fits mask image for excluding pixels for analysis
    slit{blue,red}.fits slit-function
    slit{blue,red}-y-dist.fits y-distortion (order curvature) from the quartz
    unknown files
    files descriptions
    actual.fits ???
    ccsum.fits ???
    fitted.fits ???
    flat1d.fits ???
    lc.fits ???
    lowmask.fits ???
    refer.fits ???
    refspecs.fits ???
    spatial.fits ???
    udata.fits ???
    uflatdata.fits ???
    files that get over-written
    files descriptions
    p.fits profile of the slit-function and profile of the lamp
    CT.fits cross-correlation of the profiles of the slit-function and the lamp
    found.dat generated wavelength cal.
    knots.dat generated sky-subtraction
    What are the parameters for:
    • getrect
    • GHLBExtract