The Carnegie Python Distribution (CarPy)

CarPy, begun in 1998, is a suite of Python scripts for astronomical data handling, reduction, and analysis. Routines, mostly undocumented, are in place for the processing of raw astronomical imaging to calibrated, astrometrically rectified image stacks, including routines for automated and semi-automated astrometric analysis. Included are a large number of Kelson's routines for processing single- and multi-slit spectroscopic data, as well as echellograms. Many tasks can be distributed over multiple cores by forking to child processes (using the included scripts and routines) and many can also be distributed over multiple, distinct nodes, as long as the main data directories can be mounted over NFS (or AFP, etc). The code for distributing the computing uses Twisted, with all message passing performed over ports forwarded through secure SSH connections, with (optional) authentication, such that multiple processes and users have privacy with their distributed tasks. Furthermore, the code automatically selects random, available ports, ensuring zero port collisions.

Users of CarPy for spectroscopic data reduction should cite the following papers:

  1. Kelson, D.D., Illingworth, G.D., van Dokkum, P.G., & Franx, M. 2000, ApJ, 531, 159
  2. Kelson, D.D. 2003, PASP, 115, 688

Here are some vague instructions on downloading and installing the Carnegie Python Distribution.

Overview


CarPy is a full, self-contained python distribution bundled with additional libraries and modules from 3rd party providers (e.g., numpy, scipy, VTK, etc).  It also contains pure python modules and scripts that are not OS- or architecture- dependent.

Supported Operating Systems

While we try to be as helpful as possible and maintain "legacy"  systems, our policy is to fully support only those operating systems that are used by the Observatories and that have not yet reached "End of Life" (EOL). If you need to run on an older version of MacOS or CentOS that is no longer supported, we encourage the use of our CentOS Virtual machine as the next easiest option.

Installing Binary Distributions


Due to the way certain python packages are built, the absolute path of the python interpreter gets hard-coded into some of the scripts.  Also, Mac OSX seems to hard-code paths into its frameworks.  CarPy must therefore be installed in /opt/CarPy (or similar system level folder) to keep everything consistent.  You will need permission to write to this location.

0. System pre-requisites

Make sure you run CarPy in as clean an environment as possible. Other python distributions like Anaconda, Ureka, and scisoft are known to interfere with both the binary distribution and building from source. Disable all such environments in your startup scripts (.tcshrc, .bash_profile, etc) before using CarPy,.

Before you can install CarPy, you need to make sure certain libraries and/or packages are installed on your system.

Linux Systems: Make sure the following packages are installed (use your package manager, e.g., yum): atlas, mesa-libGL, gcc-gfortran, and lib-gfortran. If you still end up with errors about libraries not being found, try installing the *-devel packages as well.

Mac OS X: As of 10.10 (El Capitan), you need to install XQuartz (either using the package installer or with homebrew). You will also need to install the Xcode command-line tools by running the command "xcode-select --install" in a terminal.

1. Using Native Package Installer

If an installer is available for your system, you can use it in the usual way (double click the .pkg file on OSX or use rpm to install RPM file on Linux). Note that you should still make sure you have the system requirements listed above. Also, be sure to read the section below:  Using CarPy. The native installers are located in the download area.



2. Using CarPy


The only thing left is to setup the environment with variables that will ensure that the correct dynamic libraries are used and that python finds the correct modules. To do this, simply source one of the setup scripts provided in the CarPy folder. Depending on which install option you used, CarPy will either be located in /usr/local/ or /opt. Examples:


# zsh or bash:
. /opt/CarPy3/Setup.bash
# csh or tcsh:
source /opt/CarPy3/Setup.csh


You can place these in your startup scripts if you wish. Once the environment is setup, you are ready to use CarPy. Another option, especially for those of us who have multiple python environments, is to run CarPy in a clean sub-shell. For example, you could create the following alias:


alias carpy='env -i PS1="(carpy) [\u@\h \W]\$ " bash --noprofile --init-file /opt/CarPy3/Setup.bash'


You may need to customize this to reflect your setup and make sure the bash shell has the basic environment (e.g., $HOME is set). Running 'carpy' will start a "clean" sub-shell and activate CarPy within it. When you are done, simply "exit" and you're back in your original shell. 


If you are trying to use a binary package that does not match your system (say using Fedora binary on a CentOS system),then you must set an environment variable, PYTHONBASE, to point to the binary you are intending to use before you source the setup scripts.  For example:


% setenv PYTHONBASE /usr/local/CarPy/builds/Fedora.13.x86_64


Getting Help


If you come across problems, please contact the CarPy team at mailto:carpy@obs.carnegiescience.edu