Python Development on OSX

Setting up OS X for python development can be confusing because there are multiple options. Below I go into more detail but the TL;DR version is that I use:

1. the command line extensively
2. pyenv to manage multiple python versions
3. pyenv-virtualenv to manage virtual environments
4. Sublime Text 3 with a few choice packages to write code
5. ack to search code
6. IPython for a better python shell (REPL)

A note about the command line

I consider a familiarity with the OS X command line an essential developer skill. Learning about shells, shell scripting, and the history of terminals is fascinating stuff. This post relies on the use of the command line and knowledge of the PATH environment variable.

OS X python

OS X includes python out of the box. If you open terminal and type python --version you will see something like Python 2.7.10. If you type python -c "import sys; print sys.path" you will see the python path in use. The version of python included on OS X El Capitan is 2.7 and has been modified by Apple. I never use that version for development. I leave it alone. It lives in /System/Library/Frameworks/Python.framework with binaries copied to /usr/bin, e.g., /usr/bin/python. It is possible to have multiple versions of python in /System/Library/Frameworks/Python.framework if you have upgraded over the years. Binaries for other versions are symlinked in /usr/bin, e.g., /usr/bin/python2.6 -> /System/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6. One other thing to note is that the site-packages directory isn't where you would expect. Instead of /System/Library/Frameworks/Python.framework/Versions/Current/lib/python2.7/site-packages it is at /Library/Python/2.7/site-packages.

Python.org python

A second option is to use one of the installers from python.org. For example, the python 3.5 python.org installer installs python in /Library/Frameworks/Python.framework/Versions/3.5 and the binaries are symlinked to /usr/local/bin, e.g., /usr/local/bin/python3 -> /Library/Frameworks/Python.framework/Versions/3.5/bin/python3. The site-packages directory lives at /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages. The latest python 2 and python 3 installers include pip. During installation the installer adds /Library/Frameworks/Python.framework/Versions/3.5/bin to the beginning of your PATH environment variable so your shell can find commands created by python modules like virtualenv. If you are interested you can see how this is done in the file /Applications/Python\ 3.5/Update Shell Profile.command. If you install python 2 and python 3 from python.org your PATH variable will include both /Library/Frameworks/Python.framework/Versions/3.5/bin and /Library/Frameworks/Python.framework/Versions/2.7/bin. The one that was installed last will be searched first. Python co-existence is handled by appending the version number to the python binary simlinked in /usr/local/bin. For python 2 python2 --version and for python 3 python3 --version. You can install multple minor release versions at the same time. For example, you can install python 3.4 and python 3.5 and the binaries symlinked in /usr/local/bin will include the version number, e.g., /usr/local/bin/python3.4 and /usr/local/bin/python3.5. You can't install micro-level versions side by side. For example, you can't install 2.7.9 and 2.7.10. The last 2.7 version you install overwrites the existing 2.7 version.

Uninstalling python.org python

Unfortunately the python.org installers don't come with an uninstaller. The following commands will remove all python.org versions from your system

% sudo rm -rf /Applications/Python\ *  # delete the OS X extras installed with the distribution
% sudo rm -rf /Library/Frameworks/Python.framework  # delete the python distributions
% sudo rm /private/var/db/reciepts/org.python* # delete the pkg receipts created by the installer
% ls -l /usr/local/bin | grep '../Library/Frameworks/Python.framework' | awk '{print $9}' | tr -d @ | sudo xargs rm   # delete the symlinks created by the installer

Manually find `org.python` entries in the /Library/Receipts/InstallHistory.plist and delete them
Manually clean up the PATH manipulation in .bash_profile

Python.org links

• python.org downloads

Homebrew python

A third option is to use homebrew. Homebrew is a package manager for OS X. It's a fantastic tool but it can only really handle installing one version of python 2 and one version of python 3. Homebrew installs packages into /usr/local/Cellar and symlinks them into various places under /usr/local, e.g., /usr/local/opt and /usr/local/bin. Homebrew builds python from source so make sure you have installed xcode and the xcode command line tools. To install python 3 type brew install python3. Brew installs python3 in /usr/local/Cellar/python3, site-packages are in /usr/local/lib/python3.5/site-packages, and it is symlinked into /usr/local at various places. It even symlinks a Framework version at /usr/local/Frameworks/Python.framework. It also installs pip.

Uninstalling hombrew python

Unfortunately brew uninstall python only removes python. It doesn't uninstall its dependencies. Dependency management is a tricky business and brew hasn't solved it yet. You can view the dependencies by typing brew info python and brew info python3. For example, the following packages are installed along with python3: xz, pkg-config, readline, sqlite, gdbm. Please note that if a package is being used by another package uninstalling it will cause problems. It is probably best to leave them alone if you use a lot of tools installed from brew.

brew uninstall python3
rm -rf /usr/local/lib/python3.5/

Homebrew links

• Homebrew
• Homebrew docs

pyenv

Homebrew is a great way to get started with python but if you need to run specific versions of python it isn't the right tool. My preferred way to handle multiple python versions is to use pyenv. You can install pyenv with homebrew brew install pyenv and add the following to .bash_profile if which pyenv > /dev/null; then eval "$(pyenv init -)"; fi and then reload it with source ~/.bash_profile. Pyenv builds python from source and requires xcode and the xcode command line tools. To install a specific version of python type pyenv install 3.5.1. Pyenv installs python in ~/.pyenv/versions/3.5.1. Pyenv uses shell shimming and PATH manipulation techniques to invoke the correct python binary. It is explained in more detail in the how it works section of the documentation. Pyenv lets you set a global python version with the pyenv global command. For example, typing pyenv global 3.5.1 creates a file at ~/.pyenv/version containing the string 3.5.1 that the pyenv shims use to decide which binary to execute. In this case typing python will result in the python shim executing the python 3.5.1 binary in ~/.pyenv/versions/3.5.1/bin.

Pyenv makes development even easier with the pyenv local command. It lets you specify a version of python to use in a directory tree. For example, say I had a project in ~/projects/my-command and I wanted to use python 2.7.1. I would install python 2.7.1 by typing pyenv install 2.7.1 then change into the project directory cd ~/projects/my-command, and specify the project's python version with pyenv local 2.7.1. The pyenv local 2.7.1 command creates a .python-version file in the current directory that the pyenv shims use to determine which python binary to execute when in the current directory or any subdirectories. For example, typing python --version in ~/projects/my-command or any of its subdirectories will result in running the python 2.7.1 binary even if I had set my global version to 3.5.1 with pyenv global 3.5.1.

pyenv-virtualenv

Pyenv is great for handling multiple versions of python but you still need some kind of project dependency management tool like virutalenv or pyvenv. Pyenv has a plugin tool called pyenv-virtualenv that combines the power of pyenv and virtualenv/pyvenv. You can install it with homebrew brew install pyenv-virtualenv and add the following to .bash_profile if which pyenv-virtualenv-init > /dev/null; then eval "$(pyenv virtualenv-init -)"; fi and then reload it with source ~/.bash_profile. It will use virtualenv if it is installed or use pyvenv if virtualenv isn't installed and pyvenv is part of the standard library in the version of python you are using (3.3 and above.) Using the same example as above:

% cd ~/projects/my-command
% pyenv install 2.7.1                # install python version
% pyenv virtualenv 2.7.1 my-command  # create a virtual environment called my-command based on python 2.7.1
% pyenv local my-command             # set the project's pyenv version to the my-command virtual environment
% pip install -r requirments.txt     # install the project's requirements into the new virtual environment

A really handy feature of pyenv-virtualenv is that it will automatically activate/deactivate a virtual environment on entering/leaving directories which contain a .python-version file that includes a virtual environment. This feature essentially obviates the need for virtualenvwrapper. That said, if you still want to use virtualenvwrapper there is a pyenv plugin for it called pyenv-virtualenvwrapper but I haven't used it.

Pyenv will let you activate multiple versions of python at the same time to help with tools like tox. The documentation is well written and worth reading in full.

pyenv links

• pyenv
• pyenv-virtualenv

Sublime Text 3

Sublime Text 3 is great. Here are a few tips:

• Symlink the subl command into your PATH, e.g., ln -s /Applications/Sublime\ Text.app/Contents/SharedSupport/bin/subl /usr/local/bin/ and use subl . to open the current directory, subl my-command.sublime-project to open a sublime text project, or subl myfile.py to open a single file.
• Use sublime text projects, e.g., Project - Save Project As...
• Use the Package Control plugin to manage packages
• My favorite packages for python development:
  • Terminal
  • Anaconda
  • Markdown Preview
  • Djaneiro
  • Theme Solarized Dark
  • Solarized Color Scheme
• The unofficial Sublime Text 3 docs are your friend

ack

Forget grep use ack for searching source. You can install it with homebrew brew install ack

IPython and ipdb

IPython is an improved interactive shell for python. ipdb is an improved python debugger built on ipython. IPython started out as just the interactive shell but has evolved to be part of a full interactive computing environment. I personally only use the shell and debugger. You can install both with pip install ipdb.

Cover photo of a Female Corynura Bee