Personal tools
You are here: Home / Carnegie Python Distribution

The Carnegie Python Distribution (CarPy)

CarPy, begun in 1998, is a suite of Python scripts for astronomical data handling, reduction, and analysis. Routines, mostly undocumented, are in place for the processing of raw astronomical imaging to calibrated, astrometrically rectified image stacks, including routines for automated and semi-automated astrometric analysis. Included are a large number of Kelson's routines for processing single- and multi-slit spectroscopic data, as well as echellograms. Many tasks can be distributed over multiple cores by forking to child processes (using the included scripts and routines) and many can also be distributed over multiple, distinct nodes, as long as the main data directories can be mounted over NFS (or AFP, etc). The code for distributing the computing uses Twisted, with all message passing performed over ports forwarded through secure SSH connections, with (optional) authentication, such that multiple processes and users have privacy with their distributed tasks. Furthermore, the code automatically selects random, available ports, ensuring zero port collisions.
Users of CarPy for spectroscopic data reduction should cite the following papers:
  1. Kelson, D.D., Illingworth, G.D., van Dokkum, P.G., & Franx, M. 2000, ApJ, 531, 159
  2. Kelson, D.D. 2003, PASP, 115, 688

Here are some vague instructions on downloading and installing the Carnegie Python Distribution.

Overview


CarPy is a full, self-contained python distribution bundled with additional libraries and modules from 3rd party providers (e.g., numpy, scipy, VTK, etc).  It also contains pure python modules and scripts that are not OS- or architecture- dependent.  Given that the pure python will presumably change on a shorted timescale, we decided to split the distribution into architecture dependent and independent portions, similar to IRAF's distribution scheme.  Installing CarPy therefore consists of downloading and installing the universal package, followed by the binary package appropriate to your specific system.  We currently maintain 32-bit and 64-bit versions of CarPy for both Linux and Mac OSX.

Installing Binary Distributions


Due to the way certain python packages are built, the absolute path of the python interpreter gets hard-coded into some of the scripts.  Also, Mac OSX seems to hard-code paths into its frameworks.  CarPy must therefore be installed in /usr/local/CarPy to keep everything consistent.  You will need permission to write to this location.  If you can't get permission or must have the package live elsewhere, you will need to build CarPy from source.  Instructions for doing this are provided here.

0. For the impatient

Download our installer script and run it as root (or using sudo):

sudo sh install_CarPy

The script will attempt to make sure you've got what you need to run CarPy, detect the architecture, download the appropriate binary release, and install it in the correct location.

1. System pre-requisites

Make sure you run CarPy in as clean an environment as possible. Other python distributions like Anaconda, Ureka, and scisoft are known to interfere with both the binary distribution and building from source. Disable all such environments in your startup scripts (.tcshrc, .bash_profile, etc) before using CarPy,.

Before you can install CarPy, you need to make sure certain libraries and/or packages are installed on your system.

Linux Systems: Make sure the following packages are installed (use your package manager, e.g., yum): atlas, mesa-libGL, gcc-gfortran, and lib-gfortran. If you still end up with errors about libraries not being found, try installing the *-devel packages as well.

Mac OS X: You will need to install Xcode and the gfortran package appropriate for your system. Xcode comes with your software DVD or you can get it from the "App Store". Download gfortran from here. For OSX 10.7 and 10.8, you will need to start up Xcode (in Applications), then select Xcode->Preferences from the Menu, click on the Downloads button, select "Command Line Tools" from the list and click "install". For 10.8, you will have to install X11 separately. You can get it here.

2. Universal Package


First, begin by downloading and installing the universal package.  There are two ways to do this.  First, you can simply download the latest tarball of the package by following this link.  Then extract the contents into /usr/local.  For example:

% wget ftp://ftp.obs.carnegiescience.edu/pub/CarPy/CarPy_2014-08-28_universal.tgz
% cd /usr/local
% sudo tar -zxf ~/CarPy_2014-08-28_universal.tgz

Alternately, you can also download the universal package from our SVN repository.  The advantage of this is that you can update to the latest version very easily. 

% cd /usr/local
% svn co svn://svn.obs.carnegiescience.edu/CarPy/trunk CarPy

Once the universal package is installed, there will be a folder /usr/local/CarPy that contains the pure python content and static data.  In the next step, we download and install the binary content into this folder.

3. Binary Package

The binary packages can be found by following this link.  Try to choose the OS and architecture that best fit your system.  Once you have downloaded the appropriate binary file, un-tar it in the CarPy folder. For example:

% cd /usr/local/CarPy
% sudo tar -zxf ~/Downloads/CarPy.07.17.2011_Darwin.10.6.i386.tgz

 

4. Using CarPy

The only thing left is to setup the environment with variables that will ensure that the correct dynamic libraries are used and that python finds the correct modules. To do this, simply source one of the setup scripts provided in the CarPy folder.

You can place these in your startup scripts if you wish. Once the environment is setup, you are ready to use CarPy.

If you are trying to use a binary package that does not match your system (say using Fedora binary on a CentOS system), then you must set an environment variable, PYTHONBASE, to point to the binary you are intending to use before you source the setup scripts.  For example:

% setenv PYTHONBASE /usr/local/CarPy/builds/Fedora.13.x86_64

For Ureka users:  

Ureka is another popular distribution of bundled software which contains its own python modules and libraries. Unfortunately, it sets some of the same environment  variables used by CarPy and will lead to problems. For this reason, you will need to disable Ureka before using CarPy using 'ur_forget'.

Getting Help

If you come across problems, please contact the CarPy team at mailto:carpy@obs.carnegiescience.edu