MIKE Pipeline
- Introduction
- Requirements for Running the Pipeline
- Setting up the MIKE database file
- Generating Pipelines
- Normalizing the Flatfield
- Running the Science Pipelines
- Other Optional Parameters
- If All Goes Well
1. Introduction
Users of CarPy for spectroscopic data reduction should cite the following papers:
- Kelson, D.D., Illingworth, G.D., van Dokkum, P.G., & Franx, M. 2000, ApJ, 531, 159
- Kelson, D.D. 2003, PASP, 115, 688
These pages discuss the pipeline written by Dan Kelson Other helpful tips can be found here:
- MIKE Pipeline Command-Line Arguments
- MIKE Pipeline Makefile Help
- Greg's List of Arguments and Outputs
The pipeline is written in
Python http://www.python.org/, using an extensive set of C,
FORTRAN, and C++ libraries to do the heavy lifting. SBS users should contact Dan or Edward for help
on installing the pipeline on their own machines. The pipeline requires that you have CarPy installed.
Instructions on how to download CarPy can be found here.
Once you have CarPy installed, all you need to do is source the included Setup.csh or Setup.bash file, and you're good to go! For example:
<csh prompt%> source /where/CarPy/is/installed/Setup.csh
Top
2. Requirements for Running the Pipeline
Typically four types of frames are used during the reduction of MIKE data:
- Milky flat frames preferably taken using a hot blue star or the quartz lamp.
- Quartz or twilight frames (used to locate the order edges and map the order curvature)
- Target frames (use for science)
- Lamp frames (used to map the line curvature and for wavelength calibration)
Top
3. Setting up the MIKE Database File
The first step is to create a text file that lists the important properties of your MIKE FITS files. This file will be referred to as the MIKE Database File.
For a quick start, you can generate this database file by typing
<csh prompt%> mikedb -d DATADIR
where DATADIR is either a relative or absolute path to the directory in which you have your FITS files. The task "mikedb" actually reads the headers of the images and stores the important stuff into a new text file.
For example, suppose you kept your data for an entire observing run in a directory called "/data1/mifune/kelson/MIKE/sept06". You would type
<csh prompt%> cd /data1/mifune/kelson/MIKE
<csh prompt%> mikedb -d sept06
or if you have a particularly anal style of data management, you might prefer
<csh prompt%> cd /data1/mifune/kelson/MIKE
<csh prompt%> mkdir Reductions
<csh prompt%> cd Reductions
<csh prompt%> mikedb -d ../sept06
After running this step, you will find a file with "MIKE.db" in its name. In the above case we are
left with sept06MIKE.db in the current working directory. This file
contains a listing of the absolute paths of the FITS files so it need not sit in the directory in
which you will be doing your reductions (even though it can be very handy to have around). Recent updates to the pipeline codes may produce "db" files that look a little different than the above example.
You might typically generate a database file for your entire run, rather than for individual nights. However, if you do generate multiple database files, you can "cat" them together into one (longer) database file. You can combine the data from multiple observing runs in this way, if you essentially wanted your output data products (i.e. extracted spectra) to include data from multiple observing runs.
Important: the MIKE Database File pulls information from the headers and thus to use the resulting "MIKE.db" file one must trust the data stored in the FITS headers. Most of the information that is used has not been altered by the observer as those bits of information are generated by the instrument control systems. However, the "OBJECT" keyword is read and thus successful and pleasing use of the "MIKE.db" file supposes that you have fixed errors and/or inconsistencies in the "OBJECT" header data. I highly recommend that you view the "MIKE.db" file in your favorite editor and verify that the information in it is correct.
Top
4. Generating Pipelines
Upon completion of the "database file" describing the observations, one invokes "mikesetup" (replacing "sept06MIKE.db" with whatever your database file is called):
<csh prompt%> mikesetup -db sept06MIKE.db
This will run through the database file in its simplest mode and return something that looks like a list of all the targets in the database file.
#------------------------------------
# J Target
#------------------------------------
1 v23900_11
2 v32_11
3 v19309_11
4 cs22964-161_11
5 cs22888-014
6 cs29513-003
7 cs30493-071
8 cs22876-032
9 cs22882-006
10 he0200-5701
11 cs22968-029
12 sc10-1838
13 v15923_12
14 hd193901_12
15 cs22964-161_12
16 he2122-4707
17 cs22945-017
18 cs22945-029
19 sc10-137844
20 cs22882-030
21 sc4-630
22 sc10-1896
23 hr1621_12
24 v6840_13
25 v32_13
26 cs22898-047
27 cs30492-110
28 he2319-5228
29 cs22945-058
30 cs22964-061_11
Note that these are similar to the OBJECT entries in the database file. My script turns your OBJECT names into things that are a bit more useful for making filenames (e.g. stripping spaces, converting to lower case). One of the reasons for doing this to the object names is to make them a bit more repeatable so that multiple instances of the same target may be more easily matched up. If you see targets listed in there (e.g. funny ones like "bias", or "zero", etc) that you do not really want to reduce, then you should either (1) comment those lines out with a "#", or (2) delete the relevant lines, or (3) put the word "ignore" in the OBJECT entries of those frames (see note below). Don't forget to do these things for both the red and blue frames!
You will also notice that nothing else has happened other than the printing of a list of target names. No pipelines or new subdirectories are created when "mikesetup" is run in this manner. By running "mikesetup" in this way, you will see all of the assigned target names. This is important if you wish to (1) check for bogus targets; (2) check for typos that may have led the scripts to not recognize identical targets for multiple frames; or (3) to ultimately allow you to generate pipelines only for specific targets since you may not know my scripts will call your science targets. After seeing the list of processed target names, you may generate a pipeline for a specific target:
<csh prompt%> mikesetup -db sept06MIKE.db cs22898-047red -red -mk Makefile.red
This should generate several subdirectories and files. There will now be a makefile called "Makefile.red" and in it are the commands to (1) generate a new normalized flatfield for the red data, (2) generate a good reference wavelength solution for a lamp frame from the red data, and (3) reduce the red data for the science target. The flatfield will be called "flatred.fits" and the reference lamp frame will be called "lampred.fits". In your working directory these will be symbolic links to files within two of the new subdirectories. If you already have a good flatfield from, for example, the previous night then feel free to copy (or link) it into your working directory with the name "flatred.fits". In that case "miksetup" will skip setting up a flatfield pipeline and use the one you provided. If you need to regenerate the pipelines and makfiles, including the ones for the flatfield and lamp, your simplest option is to type:
<csh prompt%> make -f Makefile.red regenerate
Note that if you told "mikesetup" to name your makefile "Makefile" or "makefile" then you will not need to name the makefile when invoking "make". The "make" command always looks for "Makefile" or "makefile" as a default.
The subdirectory in which you will find the science pipeline is called "ngc3201-e3red" in which are located a few files. If you had typed "-blue" on the command line then you would have generated "ngc3201-e3blue" instead. You can have both "-blue" and "-red" on the command line to generate both in one go. You can also forgo specifying individual targets and tell "mikesetup" to generate subdirectories and pipelines for all targets:
<csh prompt%> mikesetup -db sept06MIKE.db -red -blue -all -mk Makefile
This is the most fruitful way to run things as long as you don't have bizarre "objects" listed as targets (such as "bias", or "zero", or other things that you do not wish to reduce as targets).
After "mikesetup" generates all the pipelines, go reduce your data by typing
<csh prompt%> make
IMPORTANT: Please try to cull bad data out of the "MIKE.db" file. Please take a look at the database file and either delete useless frames or place a keyword (such as "ignore") within the OBJECT entry. When you run "mikesetup" you can specify an "-ignore VVVVV" and thus any frame with "VVVVV" in its OBJECT entry will be ignored. The default is the word "ignore". In the example database file "sept06MIKE.db" shown above, frames 0163 are to be ignored.
IMPORTANT:
- The first time you run mikesetup it will generate subdirectories and pipelines for processing the milky flats (see below).
- The script by default assumes that if an entry in the database has "thar" in the modified OBJECT (i.e., spaces removed, concatenated, forced to lower-case, etc) then the file is a comparison lamp. Run "mikesetup" with "-lampkey XXXXX" where "XXXXX" is the string that quasi-uniquely identifies the lamp frames. The lamp exposures are associated with the science frames by searching for the lamp with the nearest MJD (at the midpoint of the exposure), with a small perturbation to search for the frame with an appropriate slit angle.
- The script by default assumes that if an entry in the database has "milky" in the modified OBJECT (i.e., spaces removed, concatenated, etc) then the file is a flatfield. Run "mikesetup" with "-flatkey YYYYY" where "YYYYY" is the string that quasi-uniquely identifies the flatfield frames.
- The script by default assumes that if an entry in the database has "quartz" in the modified OBJECT (i.e., spaces removed, concatenated, etc) then the file is a flatfield. Run "mikesetup" with "-orderkey ZZZZZ" where "ZZZZZ" is the string that quasi-uniquely identifies the quartz (non-milky) frames. Note that "miksetup" identifies the flatfields first so that if a frame's OBJECT entry already has YYYYY in it, then it will not be identified for use in the task of finding the order edges.
Top
5. The First Pipeline in the Makefile Creates your Normalized Flatfield
As stated above, the first time you run "mikesetup" such that it generates pipelines and subdirectories, you will find (assuming you ran "mikesetup" with "-blue"):
- A new subdirectory called YYYYYblue. In the above case where YYYYY is "milky" we now have a subdirectory called "milkyblue".
- A symbolic link called "flatblue.fits" in the current working directory. The link points to an identically named file within the "milkyblue" subdirectory. At the moment, this file does not yet exist.
You can manually go into the "milkyblue" (or "YYYYYblue") subdirectory and type
<csh prompt%> make
because "mikesetup" actually creates a symoblic link called "Makefile" to the pipeline.
After a while you will find a new file called "flatblue.fits". This
file need only be generated once because the pipelines generated for
the science targets will use symbolic links to the "flatblue.fits" file.
As the individual pipelines run, each individual step, or stage,
generates an empty file to denote completion of that stage. These tell
"make" what steps remain to be completed. Thus if you run your
flatfield pipeline to completion and for some reason, go back to your
master working directory and type "make regenerate", the flatfield
pipeline will not be rerun by typing "make" alone. When you type
"make", it checks to see if there stages that remain to be executed by
looking for these empty "stage-XXXX" files. Those that are missing
indicate what will be run by "make".
If you have generated pipelines for the red data, then you will likely be required to go into the equivalent "milkyred" subdirectory (or "YYYYYred") and run the pipeline script that is there.
If you already have good clean, normalized flatfields then you can avoid the flatfield pipelines by placing a copy of your clean, normalized flat in your working directory. Please call them "flatblue.fits" and/or "flatred.fits" so that they can be identified by "mikesetup" and linked to appropriately.
Here is an early example of a flatfield pipeline. Recent updates to the pipeline scripts will look a little different.
Top
6. Running the Science Pipelines
For every target processed by "mikesetup", there are new
subdirectories, with names that are a concatenation of the target's
name and the spectrograph side (e.g. "ngc3201-v18blue"). In each of
these subdirectories you will find symbolic links to the relevant
normalized flatfield and you will also find shell scripts that contain
the long list of commands for reducing the data for that target.
When you run "mikesetup" to generate the pipelines, such as in
<csh prompt%> mikesetup -db sept06MIKE.db -red -mk Makefile -all
you will get a file called "Makefile" and all you have to do is then type:
<csh prompt%> maketo reduce the data for all of your science targets. And yes, the flatfield pipeline (if needed and generated) would be the first entry in the Makefile.
If, however, you type:
<csh prompt%> mikesetup -db sept06MIKE.db -red -mk Makefile-RED -all
You will get a file called "Makefile-RED" and you would have to type:
<csh prompt%> make -f Makefile-RED
The individual science pipelines contain many steps. Here is an example science pipeline. Recent updates to the pipeline produce pipelines (makefiles) that may contain slightly different sets of commands than the example.
Top
7. Other Optional Parameters for "mikesetup"
7.1. Including the Generation and Usage of a Fringe Frame
If you run "mikesetup" with "-fringekey FFFFF", then there will be a subdirectory created called "FFFFFred" (no fringe frames are needed or generated for blue data). There is a pipeline inside which is included as the second pipeline to be run in any output shell script (if you used the "-sh" option). The fringe frame is produced using some complicated image filtering. The process is not straightforward because you will generally use the quartz frames, and unfortunately they can have bizarrre patterns of scattered light. The frames are fit using bivariate b-splines (Dierckx 1983) and the residuals are filtered using an anisotropic diffusion filter along with some median filtering.
The result is called "fringe.fits" and symbolic links to it are created and used in the remaining (science) pipelines. This fringe frame is treated as a second flatfield frame (see the example science pipeline above) because fringing is multiplicative in spectroscopy.
7.2. Skipping the Generation and Usage of a Slit Function
If you do not desire to use a quartz frame to derive a slit function, then run "mikesetup" with "-noslitfn".
7.3. Logarithmic Wavelength Binning
If you stick "-logb" into your "miksetup" command, you will (though I think I just broke this and have to make sure...) get extracted spectra that are binned logarithmically in wavelength. This is handy for those of you doing velocity work.
Top
8. If All Goes Well
Currently many images from intermediate steps are not removed. Also, tons of information are stored and updated in the headers of images. Some postscript plots are generated by particular tasks and these can be helpful when debugging problems.
At the end of running all of the commands in a given pipeline script, you should be left with a "multispec" FITS file, in which is contained, for every order:
- Sum of the sky over the extraction aperture.
- Sum of the object over the extraction aperture.
- Expected noise from sum of object and sky plus the read noise.
- Signal-to-noise spectrum (per pixel).
- The sum of the lamp spectrum over the extraction aperture. However, this is only from the first lamp exposure since small wavelength shifts between lamp and science frames would cause unwarranted blurring of the lamp lines in this extracted spectrum.
- Flattened-flat, or blaze.
- Spectra divided by the flattened-flat, or relative fluxing.
These small shifts are measured in the science frames using the positions of the night sky emission lines. The dispersion solutions for the science frames, as modified by the sky lines, are then technically invalid for the lamp exposures.
The last command in each pipeline is to copy this "multispec" FITS file (the one called "OOOOOOO_multi.fits") into the "Final-Products/" directory. Thus after running all of the pipelines, you should (hopefully) only have to look in "Final-Products/" to get your extracted spectra.
Top
9. To Do
- Add "clean" to the Makefiles to remove temporary files.
- Allow verbosity to control output to stdout as well (to /dev/null?)
- Add step to generate reports --- regardless of failure(s).