Exploring Node.js

Over the past few days I've been messing around with node.js trying to figure it out. The fundamental thing I've been struggling with is the event-driven programming, which I've never done before. This whole process has been a substantial change in how I think about problem solving with software.

So why am I looking at node.js? Essentially I'm interested in having a very light weight server for a data logging application. The idea is the data logger should be able to take the sensor measurements and push them to a server on the local network. The server will store the values in some manner (maybe a small database, maybe a file). A webpage could be hosted to present the recorded data.

I have some experience with setting up a LAMP stack, but my intuition is that having an Apache server (though it will work) may be a bit overkill for what I'm envisioning. So I'm exploring alternative technology that may be out there.

Hamiltonian Mechanics: Now I Get It

So today i was reading this and followed some of the links to the wikipedia entries. The one for Hamiltonian Mechanics really made a bunch of stuff clear and make sense.

Wanted to share.

Install Python 2.6.2 on CentOS 5.3

Normally I don't use CentOS but rather Ubuntu since I like the shiny new stuff. Unfortunately for me, though, my work gave me some new processing machines with the provision I use CentOS. CentOS is great in that it's rock-solid stable. It sucks because to get that stability the software versions are rarely updated except for back-ports of security fixes. This isn't much of an issue unless you need some features existing in newer versions. What sucked is my current projects take full advantage of the multiprocessing library in python 2.6. This was a bit of a pain to figure out.

For example, CentOS 5.3 has python 2.4 which is hella old. It works in general, but if your project requires the use of something like the multiprocessing library (available in python 2.6) then you're hosed. Furthermore, just upgrading CentOS to 2.6 will significantly break the system. Bunches of things are tightly bound to version 2.4, really important stuff like yum for instance.

The way around this is to install python 2.6 along side 2.4 and adjust your bash environment variables to look there. Once that's established you can take advantage of all sorts of tasty things like multiprocessing and virtualenv. Since it was a nuisance to figure out, I'm gonna go ahead and document the steps here. Much of this is swiped from Villa Road with some alterations determined from my own experience and some other sites I've already forgotten.

Step 1: login as root

ssh root@yourbox.com
cd

Step 2: Install the extra packages for enterprise linux repository. It adds some previously unavailable packages and also provides some updates to existing repo packages

rpm -ivh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm

Step 3: Install development tools, ssl and zlib. These development tools will allow us to build python properly as well as aide setuptools when using easy_install later in this tutorial.

yum groupinstall 'Development Tools'
yum install openssl-devel* zlib*.x86_64

Step 4: Download sqlite, python and setuptools

wget http://www.sqlite.org/sqlite-amalgamation-3.6.18.tar.gz
wget http://www.python.org/ftp/python/2.6.2/Python-2.6.2.tgz
wget http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c9-py2.6.egg

Step 5: Build and Install SQLite

cd
tar zxvf sqlite-amalgamation-3.6.18.tar.gz
cd sqlite-3.6.18/
./configure
make
make install

Step 6: Build and install Python 2.6.2. Now, there are some important things to discuss here. First and foremost we have given the option –prefix=/opt/python2.6. This option installs the python binaries and the python library in /opt/python2.6 (it will make the dir for us) rather than in /usr/local/ which would, as we stated above, replace the standard python interpreter and inherently be bad juju. The /opt directory in redhat based distributions is a directory provides a home for larger, mostly custom built, binaries and applications.

Also, we made sure that the interpreter is going to make use of multiple threads by adding the –with-threads option. The –enable-shared option just allows python to be embedded into other apps.

cd
tar xfz Python-2.6.2.tgz
cd Python-2.6.2
./configure --prefix=/opt/python2.6 --with-threads --enable-shared
make
make install

Step 7: Now we need to make sure all our users can use the new python. To do this, we will need to add a couple of aliases and an addition to the $PATH to each users .bash_profile. This file is kept in the home directory of each user (eg: /home/rwilson/.bash_profile)

su - root
cd
nano .bash_profile
# add the following lines to the bottom of the file
alias python='/opt/python2.6/bin/python'
alias python2.6='/opt/python2.6/bin/python'
PATH=$PATH:/opt/python2.6/bin
# 'ctrl + o' to save the file and 'ctrl+x' to close the file

# now do the same for every other user, like this:
nano /home/rwilson/.bash_profile
alias python='/opt/python2.6/bin/python'
alias python2.6='/opt/python2.6/bin/python'
PATH=$PATH:/opt/python2.6/bin

Step 8: Now we need to update BASH so that it knows about the new shared libraries that we have put on the system. Create a symlink to them and then reload the cache of the shared libraries

su -
cd
cat >> /etc/ld.so.conf.d/opt-python2.6.conf
/opt/python2.6/lib    #hit 'enter' and then 'ctrl+d'
ldconfig

Step 9: Now that bash is aware of our new libraries and such, let's go ahead and make /opt/python2.6 writable by everyone. Normally this may be a no no, but it was the only way I could make a clean break from the system library when it came time to install setuptools.

chmod -R a+w /opt/bin/python2.6

Step 10: At this point I chose to close out my terminal and bring it back up, then log back in as a non-root user to be certain my .bash_profile was loaded correctly. There is most certainly another way to do it, but I'm lazy and know this works fine. Then I checked my python version (which should be 2.6.2) and went from there

#do this after logging out, closing the terminal, bringing the terminal back up, and logging back in as non-root user
which python
# should say:
# alias python='/opt/python2.6/bin/python'
# /opt/python2.6/bin/python

python -V
# should say:
#Python 2.6.2

Step 11: OK now we're all good to go with install setuptools to our new python

cd
wget http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c9-py2.6.egg
sh setuptools-0.6c9-py2.6.egg

Step 12: And finally we make a symlink for our shared library

cd /opt/python2.6/lib/python2.6/config
ln -s ../../libpython2.6.so .

Now you can use easy_install for grabbing things like numpy for your shiny new Python 2.6.2 =)

The next article will be about getting virtualenv setup and running. That one should be fairly universal for most distributions. Then later we'll be looking at building BLAS, ATLAS, FFTW and friends from source so they take advantage of our specific hardware. =)

It's Been a While

Indeed it has. So much has happened within the past month or so. Mom and Lowell came to visit. Benedikt our German intern moved in downstairs. My son Ben has grown a lot, holding his head up, smiling, etc. We even took him to Sears for a photo session. He was such a ham. I'll get some of the photos up sometime. I got offered a cool opportunity at work to present a scientific paper in November. Becky, Ben and I went to Anchorage for her cousins wedding. We shopped a lot, I had Arby's =), and we met up with a friend of hers. Yeah a lot has been going on.

I've been spending most of my time working on the science project. I was told I'm the only person at the facility that could possibly pull it off so naturally I accepted. The appeal to my ego worked it seems and now i'm spending many late nights in the office cranking out code and building a number crunching machine. That, actually, has been a real pain in the but. It's an 8-core xeon rig, with 12GB ram, and about 10TB storage. I've been having networking issues that after about 5 full days and extremely late nights of troubleshooting ended up being that it was plugged into a network switch that was failing. Good going linksys. Your $20 switch cost us about $1000+ in man-hours. Stupid.

Over the past month I've been coding a lot in python. It rocks my face off so hard I can't begin to describe. Part of my project involved doing a baseline set of image processing. I had the majority of the code completed in about two days because python handled a lot of the tedious stupid stuff like file management, memory management, and options parsing. I spent the next few days learning about the multiprocessing library so i could fork the number crunching software to multiple cpus. I only had 70 images to process for this round, but it took a whole 13 hours because i only had 2 cpus on my desktop. With my new server I can fork them to 8 cpus running about the same speed. Essentially it will be a switch from having a cpu limited process, to a throughput limited process. Each image is about 1GB, and I can only push data around from the hdd's so fast.

Anyways I should get going for now.

Laters

Ok, so I've toyed with the absolute minimal Arduino I could figure, see: http://itp.nyu.edu/physcomp/Tutorials/ArduinoBreadboard



I've drawn the electrical schematic (also with the corresponding Arduino pins from Arduino.cc):

Now for the best part - I priced the components (including a DIP socket for the ATMega8) from Jameco.com it comes out to $5.73! (8 pennies less if you don't care to have the power LED)

From what I can gather, it has no serial I/O, so complex sensors are out, but for simple read a sensor, take an action it doesn't get any better than this!

I'm hoping that it'll be as simple as swapping out the ATMega chips on an existing Arduino to burn the bootloader, then download your program. Then simply put in the ATMega chip and power up. Not 100% sure, I'll have to check the bootloader stuff on Arduino.cc

The best thing here is you won't have to waste your $30 Arduino for simple stuff!

Python - Recursively Zip Directories (extended)

This looks pretty useful, especially for work since we rip through hundreds of files for zipping and unzipping.

The idea and snippet are from Corey Goldberg's post sharing the work he did to improve a bit of recursive zipping code. I extended what he made to include command line options and some usage help, complete with lazy loading in case somebody wants to use it as a library later. Just wanted to share and record for my own purposes.

#!/usr/bin/env python

import os, zipfile

def zipper(dir, zip_file):
    zip = zipfile.ZipFile(zip_file, 'w', compression=zipfile.ZIP_DEFLATED)
    root_len = len(os.path.abspath(dir))
    for root, dirs, files in os.walk(dir):
        archive_root = os.path.abspath(root)[root_len:]
        for f in files:
            fullpath = os.path.join(root, f)
            archive_name = os.path.join(archive_root, f)
            print f
            zip.write(fullpath, archive_name, zipfile.ZIP_DEFLATED)
    zip.close()
    return zip_file

if '__main__' == __name__:
    # Late import, in case this project becomes a library, never to be run as main again
    import optparse
    
    # Populate our options, -h/--help is already there for you
    usage = "usage: %prog [options]"
    version="%prog 1.0"
    parser = optparse.OptionParser(usage=usage, version=version)
    parser.add_option("-d", "--dir", dest="inputDir", default="~/test", action="store", help="sets the input directory to something other than the default (~/test)")
    parser.add_option("-f", "--file", dest="outputFile", default="~/temp/test.zip", action="store", help="sets the output zip file to something other than the default (~/temp/test.zip)")
    parser.set_defaults()
    
    # Parse the arguments (defaults to parsing sys.argv)
    (options, args) = parser.parse_args()

    # Here would be a good place to check what came in on the command line and
    # call parser.error("Useful message") to exit if all is not well
    if len(args) > 0 and (1 != options.inputDir or 1 != options.outputFile):
        parser.error("Additional arguments are not supported\nYou can only change the inputDir or outputFile using the -d and -f options.\nType zippy.py -h for help.\n")

    
    # Do the actual work
    zipper(options.inputDir, options.outputFile)

Weekend

My Arduino has arrived and I'm stoked =P Now I need to make some time to mess around with it. There are some oscilloscope projects I'm interested in fiddling with. Link Link Link

This morning I found a nifty little snippet for adding an ascii spinner to your output. So if you have a long process, which we face often here at work, you can have this displaying to let you know it's doing stuff.

And of course this Saturday is when Mom, Lowell, and my Grandparents are arriving for a week long visit =D To say we're excited is an understatement. So much to do in just a week.

Ben is looking into some local positioning systems. I don't know much about these yet so some investigation is needed. Surely there's a way to use bluetooth or rfid in a manner similar to how gps works.