The Carnegie Python Distribution (CarPy)

CarPy, begun in 1998, is a suite of Python scripts for astronomical data handling, reduction, and analysis. Routines, mostly undocumented, are in place for the processing of raw astronomical imaging to calibrated, astrometrically rectified image stacks, including routines for automated and semi-automated astrometric analysis. Included are a large number of Kelson's routines for processing single- and multi-slit spectroscopic data, as well as echellograms. Many tasks can be distributed over multiple cores by forking to child processes (using the included scripts and routines) and many can also be distributed over multiple, distinct nodes, as long as the main data directories can be mounted over NFS (or AFP, etc). The code for distributing the computing uses Twisted, with all message passing performed over ports forwarded through secure SSH connections, with (optional) authentication, such that multiple processes and users have privacy with their distributed tasks. Furthermore, the code automatically selects random, available ports, ensuring zero port collisions.

Users of CarPy for spectroscopic data reduction should cite the following papers:

Kelson, D.D., Illingworth, G.D., van Dokkum, P.G., & Franx, M. 2000, ApJ, 531, 159
Kelson, D.D. 2003, PASP, 115, 688

Here are some vague instructions on downloading and installing the Carnegie Python Distribution.

Overview

CarPy is a full, self-contained python distribution bundled with additional libraries and modules from 3rd party providers (e.g., numpy, scipy, VTK, etc). It also contains pure python modules and scripts that are not OS- or architecture- dependent. Given that the pure python will presumably change on a shorted timescale, we decided to split the distribution into architecture dependent and independent portions, similar to IRAF's distribution scheme. Installing CarPy therefore consists of downloading and installing the universal package, followed by the binary package appropriate to your specific system. We currently maintain 32-bit and 64-bit versions of CarPy for both Linux and Mac OSX.

Supported Opertating Systems

While we try to be as helpful as possible and maintain "legacy" systems, our policy is to fully support only those operating systems that are used by the Observatories and that have not yet reached "End of Life" (EOL). If you need to run on an older version of MacOS or CentOS that is no longer supported, you should use the the install script option, as this will ensure that a consistent set of binary packages and python modules/scripts are installed. If this fails, we encourage the use of our CentOS Virtual machine as the next easiest option.

Installing Binary Distributions

Due to the way certain python packages are built, the absolute path of the python interpreter gets hard-coded into some of the scripts. Also, Mac OSX seems to hard-code paths into its frameworks. CarPy must therefore be installed in /usr/local/CarPy to keep everything consistent. You will need permission to write to this location. If you can't get permission or must have the package live elsewhere, you will need to build CarPy from source. Instructions for doing this are provided here.

0. System pre-requisites

Make sure you run CarPy in as clean an environment as possible. Other python distributions like Anaconda, Ureka, and scisoft are known to interfere with both the binary distribution and building from source. Disable all such environments in your startup scripts (.tcshrc, .bash_profile, etc) before using CarPy,.

Before you can install CarPy, you need to make sure certain libraries and/or packages are installed on your system.

Linux Systems: Make sure the following packages are installed (use your package manager, e.g., yum): atlas, mesa-libGL, gcc-gfortran, and lib-gfortran. If you still end up with errors about libraries not being found, try installing the *-devel packages as well.

Mac OS X: As of 10.10 (El Capitan), you need to install XQuartz (either using the package installer or with homebrew). You will also need to install the Xcode command-line tools by running the command "xcode-select --install" in a terminal.

1. Use a Native Package Installer

If an installer is available for your system, you can use it in the usual way (double click the .pkg file on OSX or use rpm to install RPM file on Linux). Note that you should still make sure you have the system requirements listed above. If a package installer is not available, then try the next step. Also, be sure to read the section below: Using CarPy. The native installers are located in the same download area as the binary packages.

2. Using the Install Script

In an attempt to make things more automatic, you can try downloading our installer script and running it as root (or using sudo):

sudo sh install_CarPy

The script will attempt to make sure you've got what you need to run CarPy, detect the architecture, download the appropriate binary release, and install it in the correct location. If it works, congratulations, you're done! Make sure you read the section on Using CarPy. If it fails, read on for more detailed instructions, or build from source.

3. Universal Package

First, begin by downloading and installing the universal package. There are two ways to do this. First, you can simply download the latest tarball of the package by following this link. Then extract the contents into /usr/local. For example:

% wget ftp://ftp.obs.carnegiescience.edu/pub/CarPy/binaries/CarPy_2018-Apr-24_universal.tgz
% cd /usr/local
% sudo tar -zxf ~/CarPy_2018-Apr-24_universal.tgz

Once the universal package is installed, there will be a folder /usr/local/CarPy that contains the pure python content and static data. In the next step, we download and install the binary content into this folder.

4. Binary Package

The binary packages can be found by following this link. Try to choose the OS and architecture that best fit your system. Once you have downloaded the appropriate binary file, un-tar it in the CarPy folder. For example:

% cd /usr/local/CarPy
% sudo tar -zxf ~/Downloads/CarPy_2018-Apr-24_Darwin.10.13.x86_64.tgz

5. Using CarPy

The only thing left is to setup the environment with variables that will ensure that the correct dynamic libraries are used and that python finds the correct modules. To do this, simply source one of the setup scripts provided in the CarPy folder. Depending on which install option you used, CarPy will either be located in /usr/local/ or /opt.

You can place these in your startup scripts if you wish. Once the environment is setup, you are ready to use CarPy. Another option, especially for those of us who have multiple python environments, is to run CarPy in a sub-shell. For example, you could create the following alias:

alias carpy='env -i PS1="(carpy) $PS1" bash --noprofile --init-file /usr/local/CarPy/Setup.bash'

Then, running 'carpy' will start a "clean" sub-shell and activate CarPy within it. When you are done, simply "exit" and you're back in your original shell.

If you are trying to use a binary package that does not match your system (say using Fedora binary on a CentOS system),then you must set an environment variable, PYTHONBASE, to point to the binary you are intending to use before you source the setup scripts. For example:

% setenv PYTHONBASE /usr/local/CarPy/builds/Fedora.13.x86_64

Getting Help

If you come across problems, please contact the CarPy team at mailto:carpy@obs.carnegiescience.edu

Personal tools