Pyrex for dummies

Pyrex is a very powerful tool if you need to speed up some Python code or connect it with some C code. My main motivation to use it was the ability to distribute only the compiled files and not also a C compiler etc.

Unfortunately, the learning curve is a bit steep if you want to do anything non-trivial (if you are not a skilled programmer, that is). I find it very difficult to read and understand just about anything about Python C API and related topics. So any attempt to learn new tricks means spending hours with Google and trying to find an example of usage which is simple enough for me to understand.

What follows is a few steps which I generally follow when rewriting Python into Pyrex code.

• Profile your code. Start with rewriting the slowest parts of your code (unless you want to know how fast you can make everything).


• Try to compile unchanged Python code with Pyrex. This way you will start with a working code and if something goes wrong you probably broke it with later changes.


• Declare loop control variables as C types:

  cdef int i

  and change Python for loops into Pyrex for loops

  for i from 0 <= i < 10:



• Numpy array data should be accessed directly. It's fairly easy to do now, that I finally know how to do this (see further).
  


• Additional speedup may be achieved by eliminating or minimising Python function calls and/or replacing them with C function calls.
  


• To help understanding your functions, add at least one line to the docstring such as:

  """myfunc( N, array x) -> array y"""

  Pyrex (or C) function parameters are not accessible as in Python functions by using help(myfunc) therefore you must explicitly write it to the docstring.
  

These steps have so far done the trick for my purposes. I essentially only need fast maths code so I have no idea about other areas. What I may have to learn later is some string stuff but so far I had neither guts nor reason to try it. This means I'm using pure Python strings in my Pyrex modules.

Now some selected details as promised:
To use some standard C function you have to declare it before use. So, if I e.g. need asin() from the math.h library I put this at the beginning of the Pyrex module:

cdef extern from "math.h":
   double asin(double)

Using Numpy C API to access array data directly was tough to learn, this is my current way:

• put to the beginning of the script:

  cimport c_numpy

  for this to work you have to copy 'c_numpy.pxd' file from 'Python\Lib\site-packages\numpy\doc\pyrex' into the directory with your script (there is a warning about future removal, I hope the same-named file in '\doc\cython' will work as well).


• initialize numpy by :

  c_numpy.import_array()



• declare new numpy array:

  cdef c_numpy.ndarray x



• create numpy array:

  x = numpy.zeros(10)

  There is another (possibly faster) way of creating new arrays but this is what i use now (I also do not know the other way, should have made a note...).


• declare pointer to store the adress of the numpy array data:

  cdef double *xdata



• copy data adress to your pointer:

  xdata = <double *>x.data

  The x.data is a char poiter to the first number in your array. I have no idea what this means (why char?).


• you may now index xdata to get desired element value:

  xdata[6] = 12.54

  or

  tempvar = xdata[1]



• you may declare the numpy array in the same way during function declaration:

  def myfunc(double step, c_numpy.ndarray x):
  ...

Using the Numpy C API is more cumbersome than just indexing numpy arrays but the code speedup is often significant.
Looking through my Pyrex modules this should in essence be all that is needed to get started with Pyrex.

Pyrex - mixing C and Python code

I have had a little optimization mania recently. After realizing that scipy.weave is ill suited for deployment on the computers of my non-programming colleagues I was looking for other options and tried the Pyrex extension. I recommend reading the Pyrex home page for details of what exactly Pyrex is. A title sentence of its home page explains it nicely:

Pyrex lets you write code that mixes Python and C data types any way you want, and compiles it into a C extension for Python.

This has the (for me) significant advantage that there is no need to install scipy and gcc on any computer that you want your code to run on (as compared to the scipy.weave model). Distributing the compiled extension file is enough.
To make Pyrex work on Windows follow these instructions (here seems to be a copy). As usual you have to change something a bit to make it work with gcc. The key step of these instructions is creation (or edit) of distutils.cfg and adding these lines into it:

[build]
compiler = mingw32

Aside from this I also use a modified python setup command:

python setup.py build-ext --inplace

It suits me better to have the compiled extension in the working directory and not in Python dir as happens with the 'install' command.
I am aware of the Cython language which has developed from Pyrex but I have not tried it. I guess I don't need to fix what ain't broken ;-) Pyrex works great for me right now.

Speeding up Python code with Psyco

Possibly the easiest way of speeding up Python code is to use Psyco. Psyco is extremely easy to use and it is at least worth a try before delving into C/C++ mess either directly or with the help of scipy.weave.
The usage is simple, just download Psyco, install it and add following to your code:

import psyco
psyco.full()

The full() directive tells Psyco to optimize everything. I usually put these two lines to the beginning of the code, just after other imports. Psyco website states that possible speedups can be between 2x-100x. I have seen every number between 10% faster to 50x faster code execution.
The following is an example of very high acceleration of very simple code:

def test(N, M):
    res = []
    for i in range(N):
        for j in range(M):
            if j == 0:
                x = i*i*i
                res.append(x)
    return res

Running this function with N = 10000 and M = 100000 takes ~60 seconds on Intel E8200 processor. After importing and running Psyco the very same code takes ~1.2 seconds. The speedup is then ~50x.
Psyco works best on functions which run at least couple of seconds per each call and are similar to the presented test function. Psyco only optimizes code within functions and scripts classes but this isn't really a problem.
To only optimize one function use

psyco.bind(yourfunctionname)

Improving blogger template style - Sand Dollar

There is something what I don't like on my currently selected blogger template style - the Sand Dollar. It spans all available screen real estate from left to right. I would like to make it fixed-width and centered. To achieve this, I've changed the '#outer-wrapper' part of the template by adding 'width: 980px;'. This makes the rendered web page stay only 980 pixels wide which should be enough even on 1024 pixel wide common displays.
To center web page contents, just add 'margin: 0 auto;' to the same section. This means in english: top and bottom margins are zero, left and right margins are set by browser (which means content will be centered). You can find the specifications of the margin command here.
The entire outer-wrapper section now looks like this:

#outer-wrapper {
font:$bodyfont;
width: 980px;
margin: 0 auto;
}

scipy.weave doesn't work - compiler troubles

The scientific python package SciPy contains a very powerful library called weave. Weave enables one to relatively easily embed C/C++ code into Python code. Thus, if you happen to have a computationally intensive code you can gain a lot of speed by rewriting it in C/C++. There are a lot of ways of how to accomplish this but weave makes this feat relatively easy even for an amateur like me.
There is however one caveate for Windows users. I commonly install all software into the 'C:\Program files\' directory. If you do this with Python and a weave requirement, MinGW compiler, weave will not work. The trouble is the space within directory name. Weave scripts and possibly also gcc are not written to deal with spaces within paths and will not find required files like gcc.exe. You will end up with error messages like:

error: command 'gcc' failed: No such file or directory
ValueError: The 'mingw32' compiler was not found.

At first I have made a hack in the first affected script, the 'platform_info.py', but this only resulted in another error:

Could not locate executable g++
Executable g++ does not exist
Traceback (most recent call last):
 File "G:\Temp\tmpfgm8bn\_ev-uq.py", line 8, in 
   s,o = exec_command(cmd, _with_python=0, **{})
 File "C:\Program Files\Python25\lib\site-packages\numpy\distutils\exec_command.py", line 253, in exec_command
   use_tee=use_tee,**env)
 File "C:\Program Files\Python25\lib\site-packages\numpy\distutils\exec_command.py", line 402, in _exec_command
   so_dup = os.dup(so_fileno)
OSError: [Errno 9] Bad file descriptor

So, to make things short, just install Python and MinGW into C:\. This will save you from a lot of trouble.
UPDATE 08/10/22: It turned out today that your user name also may not have a space within it. I've just spent an hour trying to get weave working on a colleague's computer where the username is actually a "user name". No, renaming the user does not do the trick. I 'solved' it by creating a new user with 'valid' username.

Strange scipy.weave import error

Today, I've booted my PC and tried to run a script which uses scipy.weave package. What I got was a rather unusual error:

EOFError: EOF read where object expected

I've tried to type

import scipy.weave

which just reproduced the error. After some thinking I got an idea that some files were broken. A scipy reinstall however warned that some files are broken and cannot be (re)written. So I run the scandisk/chkdsk/whatever its name currently is. It did find some errors within files mentioned in the error message. Nonetheless the import still didn't work, this time replying with another error:

ValueError: bad marshal data

After this I tried repairing both scipy and Python installations to no success. No warnings this time though.
Finally, I uninstalled Python alltogether and deleted all files and directories within C:\Python25\. After installing everything back everything seems to fine allright. I think it is the first such error since I've started using NTFS as my filesystem some five or six years ago.

Text box with scrollbar

A very nice example can be found here.
I you want to move focus so that the last entry in the text box will be visible, you may modify the code as follows:

import Tkinter

root = Tkinter.Tk()
s = Tkinter.Scrollbar(root)
T = Tkinter.Text(root)

T.focus_set()
s.pack(side=Tkinter.RIGHT, fill=Tkinter.Y)
T.pack(side=Tkinter.LEFT, fill=Tkinter.Y)
s.config(command=T.yview)
T.config(yscrollcommand=s.set)

for i in range(40):
   T.insert(Tkinter.END, "This is line %d\n" % i)
   T.yview(Tkinter.MOVETO, 1.0)

Tkinter.mainloop()

The 1.0 parameter of .yview means the end of the Text widget contents.
I'm usually using the Grid geometry manager and aligning the scrollbar with it requires some more work than with pack. Nothing out of ordinary though.