Install Python 2.6.2 on CentOS 5.3

Normally I don't use CentOS but rather Ubuntu since I like the shiny new stuff. Unfortunately for me, though, my work gave me some new processing machines with the provision I use CentOS. CentOS is great in that it's rock-solid stable. It sucks because to get that stability the software versions are rarely updated except for back-ports of security fixes. This isn't much of an issue unless you need some features existing in newer versions. What sucked is my current projects take full advantage of the multiprocessing library in python 2.6. This was a bit of a pain to figure out.

For example, CentOS 5.3 has python 2.4 which is hella old. It works in general, but if your project requires the use of something like the multiprocessing library (available in python 2.6) then you're hosed. Furthermore, just upgrading CentOS to 2.6 will significantly break the system. Bunches of things are tightly bound to version 2.4, really important stuff like yum for instance.

The way around this is to install python 2.6 along side 2.4 and adjust your bash environment variables to look there. Once that's established you can take advantage of all sorts of tasty things like multiprocessing and virtualenv. Since it was a nuisance to figure out, I'm gonna go ahead and document the steps here. Much of this is swiped from Villa Road with some alterations determined from my own experience and some other sites I've already forgotten.

Step 1: login as root

ssh root@yourbox.com
cd

Step 2: Install the extra packages for enterprise linux repository. It adds some previously unavailable packages and also provides some updates to existing repo packages

rpm -ivh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm

Step 3: Install development tools, ssl and zlib. These development tools will allow us to build python properly as well as aide setuptools when using easy_install later in this tutorial.

yum groupinstall 'Development Tools'
yum install openssl-devel* zlib*.x86_64

Step 4: Download sqlite, python and setuptools

wget http://www.sqlite.org/sqlite-amalgamation-3.6.18.tar.gz
wget http://www.python.org/ftp/python/2.6.2/Python-2.6.2.tgz
wget http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c9-py2.6.egg

Step 5: Build and Install SQLite

cd
tar zxvf sqlite-amalgamation-3.6.18.tar.gz
cd sqlite-3.6.18/
./configure
make
make install

Step 6: Build and install Python 2.6.2. Now, there are some important things to discuss here. First and foremost we have given the option –prefix=/opt/python2.6. This option installs the python binaries and the python library in /opt/python2.6 (it will make the dir for us) rather than in /usr/local/ which would, as we stated above, replace the standard python interpreter and inherently be bad juju. The /opt directory in redhat based distributions is a directory provides a home for larger, mostly custom built, binaries and applications.

Also, we made sure that the interpreter is going to make use of multiple threads by adding the –with-threads option. The –enable-shared option just allows python to be embedded into other apps.

cd
tar xfz Python-2.6.2.tgz
cd Python-2.6.2
./configure --prefix=/opt/python2.6 --with-threads --enable-shared
make
make install

Step 7: Now we need to make sure all our users can use the new python. To do this, we will need to add a couple of aliases and an addition to the $PATH to each users .bash_profile. This file is kept in the home directory of each user (eg: /home/rwilson/.bash_profile)

su - root
cd
nano .bash_profile
# add the following lines to the bottom of the file
alias python='/opt/python2.6/bin/python'
alias python2.6='/opt/python2.6/bin/python'
PATH=$PATH:/opt/python2.6/bin
# 'ctrl + o' to save the file and 'ctrl+x' to close the file

# now do the same for every other user, like this:
nano /home/rwilson/.bash_profile
alias python='/opt/python2.6/bin/python'
alias python2.6='/opt/python2.6/bin/python'
PATH=$PATH:/opt/python2.6/bin

Step 8: Now we need to update BASH so that it knows about the new shared libraries that we have put on the system. Create a symlink to them and then reload the cache of the shared libraries

su -
cd
cat >> /etc/ld.so.conf.d/opt-python2.6.conf
/opt/python2.6/lib    #hit 'enter' and then 'ctrl+d'
ldconfig

Step 9: Now that bash is aware of our new libraries and such, let's go ahead and make /opt/python2.6 writable by everyone. Normally this may be a no no, but it was the only way I could make a clean break from the system library when it came time to install setuptools.

chmod -R a+w /opt/bin/python2.6

Step 10: At this point I chose to close out my terminal and bring it back up, then log back in as a non-root user to be certain my .bash_profile was loaded correctly. There is most certainly another way to do it, but I'm lazy and know this works fine. Then I checked my python version (which should be 2.6.2) and went from there

#do this after logging out, closing the terminal, bringing the terminal back up, and logging back in as non-root user
which python
# should say:
# alias python='/opt/python2.6/bin/python'
# /opt/python2.6/bin/python

python -V
# should say:
#Python 2.6.2

Step 11: OK now we're all good to go with install setuptools to our new python

cd
wget http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c9-py2.6.egg
sh setuptools-0.6c9-py2.6.egg

Step 12: And finally we make a symlink for our shared library

cd /opt/python2.6/lib/python2.6/config
ln -s ../../libpython2.6.so .

Now you can use easy_install for grabbing things like numpy for your shiny new Python 2.6.2 =)

The next article will be about getting virtualenv setup and running. That one should be fairly universal for most distributions. Then later we'll be looking at building BLAS, ATLAS, FFTW and friends from source so they take advantage of our specific hardware. =)

Python - Recursively Zip Directories (extended)

This looks pretty useful, especially for work since we rip through hundreds of files for zipping and unzipping.

The idea and snippet are from Corey Goldberg's post sharing the work he did to improve a bit of recursive zipping code. I extended what he made to include command line options and some usage help, complete with lazy loading in case somebody wants to use it as a library later. Just wanted to share and record for my own purposes.

#!/usr/bin/env python

import os, zipfile

def zipper(dir, zip_file):
    zip = zipfile.ZipFile(zip_file, 'w', compression=zipfile.ZIP_DEFLATED)
    root_len = len(os.path.abspath(dir))
    for root, dirs, files in os.walk(dir):
        archive_root = os.path.abspath(root)[root_len:]
        for f in files:
            fullpath = os.path.join(root, f)
            archive_name = os.path.join(archive_root, f)
            print f
            zip.write(fullpath, archive_name, zipfile.ZIP_DEFLATED)
    zip.close()
    return zip_file

if '__main__' == __name__:
    # Late import, in case this project becomes a library, never to be run as main again
    import optparse
    
    # Populate our options, -h/--help is already there for you
    usage = "usage: %prog [options]"
    version="%prog 1.0"
    parser = optparse.OptionParser(usage=usage, version=version)
    parser.add_option("-d", "--dir", dest="inputDir", default="~/test", action="store", help="sets the input directory to something other than the default (~/test)")
    parser.add_option("-f", "--file", dest="outputFile", default="~/temp/test.zip", action="store", help="sets the output zip file to something other than the default (~/temp/test.zip)")
    parser.set_defaults()
    
    # Parse the arguments (defaults to parsing sys.argv)
    (options, args) = parser.parse_args()

    # Here would be a good place to check what came in on the command line and
    # call parser.error("Useful message") to exit if all is not well
    if len(args) > 0 and (1 != options.inputDir or 1 != options.outputFile):
        parser.error("Additional arguments are not supported\nYou can only change the inputDir or outputFile using the -d and -f options.\nType zippy.py -h for help.\n")

    
    # Do the actual work
    zipper(options.inputDir, options.outputFile)

Determining Multiple CPUs with Python

Just a quick entry for something i learned about that's pretty nifty. So if you wanted to take advantage of multiple cpus in python, it used to be you'd have to do a bit of OS detection first. This post has the source i repeat here:

def detectCPUs():
"""
Detects the number of CPUs on a system. Cribbed from pp.
"""
# Linux, Unix and MacOS:
if hasattr(os, "sysconf"):
  if os.sysconf_names.has_key("SC_NPROCESSORS_ONLN"):
      # Linux & Unix:
      ncpus = os.sysconf("SC_NPROCESSORS_ONLN")
      if isinstance(ncpus, int) and ncpus > 0:
          return ncpus
  else: # OSX:
      return int(os.popen2("sysctl -n hw.ncpu")[1].read())
# Windows:
if os.environ.has_key("NUMBER_OF_PROCESSORS"):
      ncpus = int(os.environ["NUMBER_OF_PROCESSORS"]);
      if ncpus > 0:
          return ncpus
return 1 # Default

That is a bit of a pain to remember all the time. Thankfully since Python 2.6, you can use the multiprocessing library to handle this for you. All the details of how it determines the number of cpus are now abstracted away from me. I don't really care, just do it =)

import multiprocessing
numCPUs = multiprocessing.cpu_count()

Then you can go from there. For instance, if i have 1 cpu, the program could decide to use the threading library. If i have 2+ cpus then it may benefit from using the full on multiprocessor library. Depends on your application.

Python, Arduino, and CUDA

It seems uncommon anymore to get really pumped about something. And by pumped, i mean real ultimate power pumped. It just so happens that a few things have recently got me that pumped.

1. The birth of my son. No question there and pretty self-explanatory so we'll move onto the others.
2. Python programming language. Holy heck is this a rocking language.
3. Arduino prototyping platform. Once again, rocking.
4. nVidia CUDA library. Using your video card as a math coprocessor? Awesome.

I've been programming for a little while now, mostly Matlab and Perl. Some C/C++ from classes but nothing production. It's only been recent that I've discovered Python, and quite frankly I wish I'd discovered it earlier. I'm enjoying how straightforward the syntax is and how much you can do with little coding. It's very programmer friendly. I intend to write much more about Python, especially since there is now a Python interface to nVidia's CUDA library.

The Arduino is essentially the face of physical computing. Generally speaking, it isn't a trivial thing to get your computer to interface with the real world and act upon it. There's all sorts of kits and what have you for the crummy basic stamp and related trash, but they were always extremely limiting and rather proprietary. In fact I always found those to be rather discouraging.

The Arduino, however, is open source, powerful, and very flexible. There are tons of projects on instructables involving the Arduino. There's tons of info out there where all sorts of people have fiddled with it and made really cool things. Heck here's one that has your plants twitter you when they need watering. Freaking cool. The biggest thing about what I find on instructables is how inspiring the projects are. Everyone swears by how easy the Arduino is to program. It's time to start fiddling.

And then there's nVidia's CUDA library. It essentially allows you to use your video card for matrix math operations. The one I have here at work was able to run a n-body simulation with 27,000 objects at 360 GFlops. Trust me, that's freaking insane. It also did an eigenvector decomposition of a 2048 x 2048 randomly generated matrix in 4.8ms. So yeah, if it's matrix math you need done, especially on a large matrix or system of equations, the CUDA library lets you have a supercomputer on your desktop.

Anyways I'll be writing more about these later. This is just the starting point.