Bug fix: TypeError: 'NoneType' object is unsubscriptable

Phobius

07-11-2009 22:56:03

While generating code for ogre in Linux I got:
Traceback (most recent call last):
File "generate_code.py", line 1659, in <module>
generate_code()
File "generate_code.py", line 1471, in generate_code
common_utils.AutoExclude ( mb, MAIN_NAMESPACE )
File "../common_utils/__init__.py", line 554, in AutoExclude
Remove_Static_Consts ( main_ns )
File "../common_utils/__init__.py", line 616, in Remove_Static_Consts
if checker( var ):
File "../common_utils/var_checker.py", line 31, in __call__
varline = self.source[declaration.location.line-1]
TypeError: 'NoneType' object is unsubscriptable

I couldn't google anything about this so i did the usual thing that I do, which is to start digging. In code_generators/common_utils/__init__.py I changed:
def Remove_Static_Consts ( mb ):
""" linux users have compile problems with vars that are static consts AND have values set in the .h files
we can simply leave these out
This function is not currently used as source was fixed.."""
checker = varchecker.var_checker()
for var in mb.vars():
if type(var.type) == declarations.cpptypes.const_t:
if checker( var ):
print "Excluding static const ", var
var.exclude()

To:
def Remove_Static_Consts ( mb ):
""" linux users have compile problems with vars that are static consts AND have values set in the .h files
we can simply leave these out
This function is not currently used as source was fixed.."""
for var in mb.vars():
if type(var.type) == declarations.cpptypes.const_t:
checker = varchecker.var_checker()
if checker( var ):
print "Excluding static const ", var
var.exclude()

I moved the checker instantiation to the inside of the loop because the checker is not reusable, and the second time around produces bogus results as the construction variables are not initialised afresh for the next operator().

andy

07-11-2009 23:52:01

I'm not sure that this fix is suitable as Var_Checker 'is' intended to be reusable -- with the change you made it will reload the source file on every var lookup...

The only way I can see the problem being caused is if declaration.location.file_name == None (which is bad)..

Could you perhaps go back to the original code but uncomment some of the print statements in var_checker.py to see what is actually happening..

Andy if possible could you post the results to the Google Mailing list along with version of Linux details etc..

Many Thanks
Andy

Phobius

08-11-2009 00:27:24

Here's what's happening, for me it bails on Ogre::Bone, specifically when it tries to remove static const from BOX_NULL and BOX_INFINITE. Because there are two of these static const BOX_ instances, we iterate twice in __init__.py's Remove_Static_Consts. For the first variable, which, by printing var I discovered is BOX_NULL, the following holds for the var_checker:
self.file_name == None
self.source == None

so when we execute:
try:
if self.file_name != declaration.location.file_name:
self.file_name = declaration.location.file_name
self.source = open(declaration.location.file_name).readlines()
except:
import traceback
traceback.print_stack()
print "Var Checker called without location:", declaration
return ""

We get the message "Var Checker called without location: BOX_NULL[variable]". That returns "" from this function and we're back in Remove_Static_Consts. Now we iterate over BOX_INFINITE and the following is true of var_checker:
self.file_name != None
self.source == None

So the block:
if self.file_name != declaration.location.file_name:
self.file_name = declaration.location.file_name
self.source = open(declaration.location.file_name).readlines()

never gets executed and we do:
try:
varline = self.source[declaration.location.line-1]
except IndexError:
return False # Bug showed up with Caelum where decl location was bigger than source ??

Because open() failed on BOX_NULL, self.source is None and we get TypeError: 'NoneType' object is unsubscriptable.

Phobius

08-11-2009 00:29:16

Yeah, I totally agree. What I did was a hack. That function needs it's structure changed for it to be considered a fix.

andy

08-11-2009 01:16:45

Can you print declaration.location.file_name and declaration.location.line each time through and let me know what they are..

Thanks for the help here

Andy

Phobius

08-11-2009 09:57:23

Ogre::AxisAlignedBox::BOX_NULL [variable] [b]# First iteration[/b]
/home/user/python-ogre/ogre/OgreMain/include/OgreAxisAlignedBox.h
797 [b] # Going into var_checker...[/b]
Ogre::AxisAlignedBox::BOX_NULL [variable] [b]# In var checker[/b]
Var Checker called without location: Ogre::AxisAlignedBox::BOX_NULL [variable]
Ogre::AxisAlignedBox::BOX_INFINITE [variable] [b]# Second iteration[/b]
/home/user/python-ogre/ogre/OgreMain/include/OgreAxisAlignedBox.h
798 [b]# Going into var_checker...[/b]
Ogre::AxisAlignedBox::BOX_INFINITE [variable]

11-08 20:59 PythonOgre.BuildModule DEBUG ../common_utils/__init__.py:7: DeprecationWarning: the md5 module is deprecated; use has$
import md5
INFO Parsing xml file "/home/user/python-ogre/code_generators/cache/ogre_1.6.1_cache.xml" ...
INFO GCCXML version - 0.9( 1.128 )
File "generate_code.py", line 1659, in <module>
generate_code()
File "generate_code.py", line 1471, in generate_code
common_utils.AutoExclude ( mb, MAIN_NAMESPACE )
File "../common_utils/__init__.py", line 554, in AutoExclude
Remove_Static_Consts ( main_ns )
File "../common_utils/__init__.py", line 619, in Remove_Static_Consts
if checker( var ):
File "../common_utils/var_checker.py", line 23, in __call__
traceback.print_stack() <--- [b]MANUALLY INVOKED[/b]
Traceback (most recent call last):
File "generate_code.py", line 1659, in <module>
generate_code()
File "generate_code.py", line 1471, in generate_code
common_utils.AutoExclude ( mb, MAIN_NAMESPACE )
File "../common_utils/__init__.py", line 554, in AutoExclude
Remove_Static_Consts ( main_ns )
File "../common_utils/__init__.py", line 619, in Remove_Static_Consts
if checker( var ):
File "../common_utils/var_checker.py", line 31, in __call__
varline = self.source[declaration.location.line-1]
TypeError: 'NoneType' object is unsubscriptable


And the two functions in question so that this makes more sense.
def Remove_Static_Consts ( mb ):
""" linux users have compile problems with vars that are static consts AND have values set in the .h files
we can simply leave these out
This function is not currently used as source was fixed.."""
checker = varchecker.var_checker()
for var in mb.vars():
print var
print var.location.file_name
print var.location.line
if type(var.type) == declarations.cpptypes.const_t:
print var
if checker( var ):
print "Excluding static const ", var
var.exclude()

def __call__(self, declaration):
# print "var_checker called with:", declaration
try:
if self.file_name != declaration.location.file_name:
self.file_name = declaration.location.file_name
self.source = open(declaration.location.file_name).readlines()
except:
import traceback
traceback.print_stack() # <-- [b]I, Phobius, put this crap in here.[/b]
print "Var Checker called without location:", declaration
return ""

## note the gccxml uses a 1 based line scheme whereas we are using python arrays - 0 based
## so we need to subtract 1 from the line number.
# print declaration.location.line, len(self.source)
try:
varline = self.source[declaration.location.line-1]
except IndexError:
return False # Bug showed up with Caelum where decl location was bigger than source ??

# print "Checker info:",declaration.location.line, declaration.location.file_name
# print "Checker Line:", varline
if not '=' in varline:
return False
# do a simple check to ensure the '=' hasn't been commented out
for char in varline:
if char =='/' or char =='\\':
return False
elif char == '=':
return True
return False

andy

08-11-2009 10:37:42

I don't understand what's happening -- almost looks like the open/readlines is failing.. How about changing var_checker to look something like: def __call__(self, declaration):
print "var_checker called with:", declaration
if not declaration.location.file_name :
print __file__, ": location file name is empty", declaration
return False
if self.file_name != declaration.location.file_name:
self.file_name = declaration.location.file_name
try:
self.source = open(self.file_name).readlines()
except:
print __file__, ": Unable to open ", self.file_name, "source is unknown", declaration
self.file_name= None # ensure we try again next time
return False

## note the gccxml uses a 1 based line scheme whereas we are using python arrays - 0 based
## so we need to subtract 1 from the line number.
# print declaration.location.line, len(self.source)
try:
varline = self.source[declaration.location.line-1]
except IndexError:
print __file__, ":IndexError ", self.source, declaration.location.line
return False # Bug showed up with Caelum where decl location was bigger than source ??
except TypeError:
print __file__, ":TypeError ", self.source, declaration.location.line
return False
# print "Checker info:",declaration.location.line, declaration.location.file_name
# print "Checker Line:", varline
if not '=' in varline:
return False
# do a simple check to ensure the '=' hasn't been commented out
for char in varline:
if char =='/' or char =='\\':
return False
elif char == '=':
return True
return False


Thanks
Andy

Phobius

14-11-2009 07:51:34

I got it. This is a threading issue. I figured that out when one time I deleted all generated sources and then regenerated and it ran fine. There's some race condition somewhere. I don't have time to debug it, but if I regenerate again it will sometimes fail. I haven't tried doing a single core generation, but I have a feeling it will work consistently. Anyway, the flaky nature of it *sometimes* being able to generate without a problem immediately indicated threading because it's non deterministic. Just for reference I'm...
/home/user/stor/pyog/python-ogre/environment.pyc LOG:: Loaded Config (based on systemtype) from PythonOgreConfig_posix
Using the 2 cores