Panic Pipeline Steps
LOAD PACKAGE
"p_reduce"
Type the above at the IRAF "cl>" prompt to load all scripts in panic_scripts directory.
You will see all available scripts when you type '?'. If a PANIC script you wish to use
does not appear, you can add it to the "p_reduce.cl" file in your panic_scripts directory.
OPTIONAL
"p_load"
If you want to use steps later than p_linearize but do not plan to run p_linearize, you
must run "p_load" (a loading step that is run automatically w/in p_linearize).
EDIT HEADER INFORMATION
"p_chk_headers n1 n2 > header_info"
This will print 'header_info' text file which displays FILE, TITLE, OBJECT, OBSTYPE,
AIRMASS & NIGHT parameters for all data. The resulting file 'header_info' should be
checked, edited to fix any incorrect header informtation, then saved. Note, this will only
correct the header of the 1st image in a series of LOOPS (since all loops will be stacked,
this header will trump the others). The minmum # of characters for an "OBJECT" name that
PANIC will accept is 8, if your object name is shorter the remaining spots will be filled
with garbage automatically. It is critical that the "OBSTYPE" be correct as these will be
used in the automated portions of the pipeline to locate darks, standards, flats, etc.
This script works in conjunction with 'p_fix_headers'.
UNCORRECTED - RAW DATA HEADERS:
File TITLE OBJECT OBSTYPE AIRMASS NIGHT
-----------------------------------------------------------------------------------------
irp_00025_001.fits SN33310- SN33310- astro 1.108 01Jan2004
(science)
irp_00011_001.fits twilights twilights tflat 1.030 01Jan2004
(twilight flats)
irp_00155_001.fits SJ91360- SJ91360- standard 1.111 01Jan2004
(standards)
irp_00198_001.fits dflat10- dflat10- dlat 1.180 01Jan2004
(dome flats)
irp_00201_001.fits dark-10- dark-10- dark 1.169 01Jan2004
(darks)
CORRECTED DATA HEADERS:
File TITLE OBJECT OBSTYPE AIRMASS NIGHT
-----------------------------------------------------------------------------------------
irp_00025_001.fits SN3331 SN3331 astro 1.108 01Jan2004
(entry not used in pipeline, but you may want to omit extra characters in target name)
irp_00011_001.fits twilight twilight tflat 1.030 01Jan2004
(used in the automated portions of the pipeline - must be corrected. Recognizable names
are: twilight and tflat, anything else will not be acknowledged as a flat)
irp_00155_001.fits SJ9136 SJ9136 standard 1.111 01Jan2004
(used in the automated portions of the pipeline - must be corrected. Recognizable names
are: sj9136, sj_9136, SJ9136 & 9136 anything else will not be acknowledged as a standard)
irp_00198_001.fits dflat dflat dlat 1.180 01Jan2004
(used in the automated portions of the pipeline - must be corrected)
irp_00201_001.fits dark dark dark 1.169 01Jan2004
(used in the automated portions of the pipeline - must be corrected. Recognizable names
are: Dark and dark anything else will not be acknowledged as a dark)
"p_fix_headers header_info"
Fixes all 'irp*_001' frame header parameters using the file 'header_info' edited with
'p_chk_headers'.
CREATE THE FOLLOWING TEXT FILES: (each will consist of 2-6 colums)
dflat_pairs 2 column list of all dome flats; p1 p2
These are used only for badpixel masks, therefore only lamps on & lamps off
Hc filtered flats are suggested here.
e.g. 1 5
tflat_pairs 2 column list of all twilight flats; p1 p2
e.g. 6 11
12 26...
unc_pairs 2 column list of all uncrowded fields; p1 p2
e.g. 27 35
36 44...
crd_pairs 6 column list of all crowded fields; p1 p2 p3 p4 p5 p6
The first 2 columns (p1 p2) are for science objects, next two (p3 p4) for
the first off-source observations (sky frames), and the last two (p5 p6) are
for the second sky observations. It is assumed that sky frames were taken
before and after each science frame, if only one sky observation was made
you can repeat this range for the last two columns.
e.g. 45 49 55 59 65 69
50 54 60 64 60 64...
std_pairs 4 column list of all standard star fields; p1 p2 p3 p4
p1-p2 are for standard frames and p3-p4 for blank/uncrowded fields to be
used for sky.
e.g. 65 69 27 35
70 75 60 64...
defaults 2 column list that Specifies where to get default darks, flats, bp_masks,
etc. You MUST HAVE darks for every exposure time taken thoughout the night,
twilights for each filter used AND either dome flats with lamps on and off
or a mask_date.pl. If you did not get these on this night, you must indicate
where the most recent versions of these can be found. Darks and flats are
automatically copied to respective directories in the IRAF 'home$/panic_files'
directory for this purpose.
e.g. dk_004_22Jan2005.fits /data1/bishop/plwyatt/iraf/panic_files/darks/
nptflatJc_22Jan2005.fits /data1/bishop/plwyatt/iraf/panic_files/flats/
mask_22Jan2005.pl /data1/bishop/plwyatt/sne/highz/2005/22jan/
Files should NOT have an extention (i.e. save as "crd_pairs" not "crd_pairs.txt").
PICK A FIDUCIAL STAR - TO BE USED IN STACKING PROCEDURE
"p_fid p1 p2 "
allows fiducial star to be selected from individual passes
OR
"p_mfid flag/s" (See below for flag definitions)
allows fiducial star to be selected from entire range of data using '_pairs' lists
RUN THE PIPELINE
"p_1go n1 n2 flag/s"
Use the following 'flags' to indicate which '_pairs' list to use, you can use combinations
of these, e.g. "p_1go 1 200 nus", "p_1go 1 200 n", "p_1go 1 200 u", or "p_1go 1 200 s"
n routines that apply to all frames in the night
u uncrowded frames (e.g. LCIRS galaxies)
c crowded frames (e.g. rich starfields with separate skies)
s standard star frames
NOTE - running 'p_1go' with just the 'n' flag will result in linearized, loop-combined,
dark subtracted & flatten images of individual passes. All others will perform sky
subtraction, stacking of individual passes & result in fully reduced images. You CANNOT
run 'u', 'c' or 's' without having run 'n' first.
*************************
SPECIFICS OF THE PIPELINE
*************************
RUNNING "p_1go" WILL CALL THE FOLLOWING SCRIPTS:
You can run any of these scripts individually by typing "p_scriptname" and any associated
info at the IRAF "cl>" prompt.(e.g. cl> p_loops_combine izp 1 220)
STACKS THAT APPLY TO ALL FRAMES (the 'n' flag):
"p_linearize p n1 n2"
Linearizes 'irp' frames n1 through n2, skipping missing ones, and producing 'izp'
frames. Here 'p' denotes PANIC. Parameters & basic algorithm from Paul Martini.
*This script also runs "p_load" if you start at a step later than this you MUST
run p_load on your own to access the PANIC scripts
"p_defaults n1"
Access files in 'defaults' list, where 'n1' is the # of the first file in data.
"p_loops_combine izp n1 n2"
Calls 'p_loopavg_sep.cl' (modified version of P. Martini's 'loopsum.cl'). Input
frames are 'izp_NNNNN_001','izp_NNNNN_002' etc. output is 'icp_NNNNN' (no extention
since loops 1 and 2 have now been combined in this new single frame).
"p_dark n1 n2"
Searches through loop-combined 'icp' frames for darks & creates 'dk_exptime_date'
frames. Expects only one 'icp' frame per exptime & that the directory
"home$panic_files/darks" exists so darks can be stored for future use if needed.
"p_subdk n1 n2"
Searches through 'icp' frames for non-dark frames & subtracts the appropriate dark
created by 'p_dark'. Produces 'icd_NNNNN' files.
"p_dflat p1 p2"
Dome flats are used ONLY to compute a badpixel mask. Note this script will use only
the last set of images in the list of 'dflat_pairs', again, it is suggested to use
the Hc lamps on & lamps off images. Produces 'mask_22Jan2005.pl' file.
"p_tflat p1 p2"
Assumes that 'icd' (linearized, loop-combined & dark-subtracted) frames exist.
'p1' and 'p2' are the starting and ending frame numbers (of valid, non-saturated)
frames of a set taken as the sky is changing during twilight with a single filter.
It is assumed that 'p_chk_headers' and 'p_fix_headers' have been run and that
therefore it is not necessary to check for filter consistency. It is assumed
FOR NOW that they all have the same exposure time.
The existence of frames within a set is not assumed (in case it has been necessary
to delete some) but is worked around. However frames p1 and p2 must exist.
This script assumes that the data have been taken according to the procedure
specified for twiflats in the PANIC camera control gui. IN PARTICULAR, the telescope
must be moved between frames so that field stars can be medianed away. If there are
any bright stars in frames, this script will not work well unless N > 10 or even 15.
These difference frames are imcombined with scaling on. Produces files of the
following form: nptflatJc_22Jan2005.fits, nptflatKs_22Jan2005.fits, etc.
"p_flatten n1 n2"
Flattens 'icd' frames to produce 'idf' frames. Flats must exist for each filter.
Assumes that 'icd' (linearized, loop-combined & dark-subtracted) frames exist.
Rev 30aug2004 SEP: Now add the two mask files 'mask_date.pl' & mask_NNNNN.pl
and store in header.
END OF "PROCEDURES THAT APPLY TO ALL FRAMES (the 'n' flag)"
***********************************************************
SCRIPTS THAT APPLY TO THE UNCROWDED FRAMES (the 'u' flag):
Does uncrowded frame (e.g. faint galaxy) stacks by accessing "unc_pairs" file
"p_sky1 idf p1, p2"
First-pass sky creation and subtraction. Assumes that 'idf' frames exist.
Produces 'isx' frames with median sky value = 0.0. Assumes that the median value
of each frame has been stored in the header as 'FRMEDIAN".
"p_fx_offsets p1 p2" (OPTIONAL)
Finds offsets of each frame relative to a "reference frame" within a dither
sequence and writes them into the header as XOFF and YOFF, in pixels. This is
an interactive script requiring user to select same fiducial star in fields.
"p_dither isx p1, p2"
For each dither sequence, finds offset of each frame wrt the REFERENCE frame
by comparing frame's coordinates for a user selected star (from p_fid/p_mfid)
to the coordinates for those stars in the REFERENCE frame. Writes offset values
(in pixels) into the header of the 'isx' and 'icx' files as XOFF and YOFF, as
well as the REFERENC frame (keyword REFERENCE). Display tool must be running.
"p_stk1 p1 p2"
Combines the dither to create an average sky subtracted image for each pointing.
This average file will be used for creating an object mask. Input is a list of
the isx*fits files. Output is ave_isx_NNNNN.fits files--one per pointing.
"p_objmask p1 p2"
Finds objects in averaged frames ('ave_isx' files) and makes OBJECT MASKS
('mask_isx' files)--one per pointing. Task input is a list of all ave_isx*.fits
files. Display tool must be running.
"p_skysub p1 p2"
Imcombines files to create a SKY IMAGE (using a combined object and bad pixel
mask), which is scaled by the object_mode/sky_mode ratio & is subtraced from the
'idf' file to create 'is2' files. Output files are is2*fits, iss*fits & mask2*pl
files.
"p_crlast p1 p2"
Makes a 'cosmicrays' pass over the 'is2' frames for the case that nloops = 1. In
this case p_fix_cr was not run on the 'izp' frames in pairs. This sets a high
(5-sigma) threshold for CRs.
"p_distcor mcp mcd p1 p2"
"p_distcor is2 isd p1 p2"
These two scripts correct 'mcp' and 'is2' frames for distortion. These become 'mcd'
& 'isd' frames.
"p_stack p1 p2 test_p1_p2"
For each dither sequence, finds offset of each frame wrt the REFERENCE frame by
comparing frame's coordinates for a user selected group of stars (from p_fid/p_mfid)
to the coordinates for those stars in the REFERENCE frame. Writes offset values
(in pixels) into the header of the 'is2' and 'idf' files as XOFF2 and YOFF2, as well
as the REFERENCE frame (keyword REFERENCE). Input is a list of 'is2*.fits' files.
Display tool must be running. Produces 'stack_test_p1_p2.fits',
'stack_test_p1_p2a.fits' & 'stack_test_p1_p2b.fits files' for each pass.
"p_hchen comblist_test_p1_p2 chen_p1_p2" (OPTIONAL)
Runs Hsaio-Wen Chen's C-code stacking program. 'filein' is created in 'p_stack'.
The frames being stacked are 'is2' versions. Display tool must be running.
"disp chen_p1_p2 1" (RUN ONLY IF PREVIOUS STEP IS USED)
Displays the "chen_filename" stack. Display tool must be running.
END OF PROCEDURES THAT APPLY TO UNCROWDED FRAMES (the 'u' flag)
*****************************************************************
SCRIPTS THAT APPLY TO THE STANDARD FRAMES (the 's' flag):
Creates stacks of all standard stars by accessing "std_pairs" file
p_get_date "idf" p3
date = p_get_date.output
pclear "ngt_"//date
#If 'sky_p3_p4' is needed, create it or recreate it:
"p_mskedge p3 p4"
"p_sky1 idf p3 p4"
"p_dither isx p3 p4"
"p_stk1 p3 p4"
"p_objmask p3 p4"
"p_skysub p3 p4"
"p_mskedgep1 p2"
"p_sky2 p1 p2 p3 p4"
"p_dither isx p1 p2"
"p_stk1 p1 p2"
"p_std p1 p2"
END OF PROCEDURES THAT APPLY TO STANDARD FRAMES (the 's' flag)
*****************************************************************
SCRIPTS THAT APPLY TO THE CROWDED FIELD FRAMES (the 'c' flag):
Does stacks of all standard stars by accessing "crd_pairs" file
# Do crowded frame stacks; off-source sky frames were done before and/or after
if(crd) {
if ( access ( "crd_pairs" )) {
# First go through all of-source "sky" frames, creating stacks and cleaned skies,
# for each set of up to two off-source dither locations. The 'crd_pairs' file is
# first processed to make a single non-repeating list of pairs of off-source
# position frames; this is 'crdlist'.
mk_crdlist ("crd_pairs")
# Now call the usual stacking routines, saving the sky frames for
# all off-source pairs. In the following set of calls p1,p2 are
# the off-source frames, originally p3,p4 and p5,p6 in 'crd_pairs'.
list1 = "" ; list1 = "crdlist"
while ( fscan ( list1, p1, p2 ) != EOF ) {
p_sky1 ( "idf", p1, p2 )
p_dither ( "isx", p1, p2 )
p_stk1 ( p1, p2 )
p_objmask( p1, p2 )
p_skysub ( p1, p2 )
}
#
# Now call a special sky subtraction routine that expects the two
# off-source sky frames 'sky_000p3' and 'sky_000p5' to exist. They
# do, because 'p_sky1' was just run on all the off-source pairs.
# NEED WORK HERE 'p_skysub' has also created 'sky_000p3_000p4'
# as well as 'sky_000p5_000p6'. So could use either one in 'p_sky3'.
# In general the 'sky_000p3_000p4'-type should be better because
# the stars have been masked out.
list1 = "" ; list1 = "crd_pairs"
while ( fscan ( list1, p1, p2, p3, p4, p5, p6 ) != EOF ) {
p_sky3 ( p1, p2, p3, p4, p5, p6 )
p_dither ( "isx", p1, p2 )
p_stk1 ( p1, p2, "test_"//p1//"_"//p2 )
p_hchen ( "comblist_test_"//p1//"_"//p2, "chen_"//p1//"_"//p2 )
}
} else {}
} else {}
END OF PROCEDURES THAT APPLY TO CROWDED FRAMES (the 'c' flag)
*****************************************************************