unicode problem during build


21-04-2010 22:53:50

I'm getting a funny unicode problem during build. Is it possible that part of the python-ogre build process changes python from using 2 byte unicode to 4 byte unicode?

The rest of this post explains why I think a change in the unicode character size is the problem.

I'm using CentOS, and following these instructions: http://wiki.python-ogre.org/index.php/LinuxBuild
At step 5, when I run

python python-ogre/BuildModule.py -g -c ogre ois cegu

I get messages saying that some tasks failed.
Checking the log I see this:

04-21 14:54 PythonOgre.BuildModule DEBUG Traceback (most recent call last):
File "generate_code.py", line 36, in <module>
import common_utils
File "../common_utils/__init__.py", line 3, in <module>
import shared_ptr
File "../common_utils/shared_ptr.py", line 9, in <module>
from pyplusplus import messages
File "/afs/cs.stanford.edu/u/ia3n/development/root/usr/lib/python2.5/site-packages/pyplusplus/__init__.py", line 23, in <module>
import code_creators
File "/afs/cs.stanford.edu/u/ia3n/development/root/usr/lib/python2.5/site-packages/pyplusplus/code_creators/__init__.py", line 18, in <module>
from code_creator import code_creator_t
File "/afs/cs.stanford.edu/u/ia3n/development/root/usr/lib/python2.5/site-packages/pyplusplus/code_creators/code_creator.py", line 8, in <module>
from pyplusplus import decl_wrappers
File "/afs/cs.stanford.edu/u/ia3n/development/root/usr/lib/python2.5/site-packages/pyplusplus/decl_wrappers/__init__.py", line 13, in <module>
import algorithm
File "/afs/cs.stanford.edu/u/ia3n/development/root/usr/lib/python2.5/site-packages/pyplusplus/decl_wrappers/algorithm.py", line 9, in <module>
from pygccxml import declarations
File "/afs/cs.stanford.edu/u/ia3n/development/root/usr/lib/python2.5/site-packages/pygccxml/__init__.py", line 30, in <module>
import pygccxml.parser as parser
File "/afs/cs.stanford.edu/u/ia3n/development/root/usr/lib/python2.5/site-packages/pygccxml/parser/__init__.py", line 14, in <module>
from project_reader import COMPILATION_MODE
File "/afs/cs.stanford.edu/u/ia3n/development/root/usr/lib/python2.5/site-packages/pygccxml/parser/project_reader.py", line 9, in <module>
import source_reader
File "/afs/cs.stanford.edu/u/ia3n/development/root/usr/lib/python2.5/site-packages/pygccxml/parser/source_reader.py", line 19, in <module>
import declarations_cache
File "/afs/cs.stanford.edu/u/ia3n/development/root/usr/lib/python2.5/site-packages/pygccxml/parser/declarations_cache.py", line 10, in <module>
import cPickle
ImportError: /usr/local/lib/python2.5/lib-dynload/cPickle.so: undefined symbol: PyUnicodeUCS2_DecodeUTF8

After reading this page: http://effbot.org/pyfaq/when-importing- ... deucs2.htm
I concluded that pygccxml was using 2 byte unicode characters and my copy of python was using 4 byte unicode characters.

So I fired up python and printed out sys.maxunicode. It is 65535. Not what I expected. I tried importing pygccxml. It worked. So my copy of python is compatible with pygccxml and uses 2 byte unicode characters, not 4.

At this point I thought I must have more than one copy of python installed, and the wrong one is running for the python-ogre build. So I edited shared_ptr.py to output sys.executable and sys.maxunicode. It printed out /usr/local/bin/python, which is the same copy that printed out 65535 for sys.maxunicode earlier. This time however, it printed out 1114111.

So somehow the same python executable is using different unicode binaries when run for ogre than when run from the command line, and this means that it doesn't work with the pygccxml I built earlier. Any ideas for how to fix this?


24-04-2010 10:15:35

The problem looks like cPickle, what happens if you simply open a python prompt and try importing cPickle?

The only thought I have is parts of the python-ogre build process append '.' and '..' to the sys.path so it could be picking up modules from a 'local' path??