Demos autotest


06-11-2008 20:25:29

Hi all,

There are often broken demos... This is natural, there are so many features to test and, as far as I know no autotest feature is available... yet. Today I tried to implement such a feature on the demos. This would help to check if everything is ok, right upon package compilation, or in order to test a new hardware platform, or different openGL/DirectX settings...

My idea is to add switches to the demos, so that they automatically take screenshots that can later be compared to reference screenshots.

I implemented my first tests on

In order to take reference screenshots, i introduced the --reference switch:

C:\PythonOgre_1.6RC1\demos\ogre> --reference
NEW_REFERENCE: ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.000000.PNG
NEW_REFERENCE: ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.200000.PNG
NEW_REFERENCE: ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.400000.PNG
NEW_REFERENCE: ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.600000.PNG
NEW_REFERENCE: ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.800000.PNG
NEW_REFERENCE: ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=1.000000.PNG
Screenshots dumped

And then, at any time, you may run the same demo with the --runtests switch:
SCREENSHOT: ./Screenshots/SCREENSHOT_Demo_Bezier_FACTOR=0.000000.PNG
SCREENSHOT: ./Screenshots/SCREENSHOT_Demo_Bezier_FACTOR=0.200000.PNG
SCREENSHOT: ./Screenshots/SCREENSHOT_Demo_Bezier_FACTOR=0.400000.PNG
SCREENSHOT: ./Screenshots/SCREENSHOT_Demo_Bezier_FACTOR=0.600000.PNG
SCREENSHOT: ./Screenshots/SCREENSHOT_Demo_Bezier_FACTOR=0.800000.PNG
SCREENSHOT: ./Screenshots/SCREENSHOT_Demo_Bezier_FACTOR=1.000000.PNG
Screenshots dumped

Finally, makes a comprehensive report of the difference between files.
Images match: ./ScreenShots/SCREENSHOT_Demo_Bezier_FACTOR=0.000000.PNG ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.000000.PNG
Images match: ./ScreenShots/SCREENSHOT_Demo_Bezier_FACTOR=0.200000.PNG ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.200000.PNG
Images match: ./ScreenShots/SCREENSHOT_Demo_Bezier_FACTOR=0.400000.PNG ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.400000.PNG
Images match: ./ScreenShots/SCREENSHOT_Demo_Bezier_FACTOR=0.600000.PNG ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.600000.PNG
Images match: ./ScreenShots/SCREENSHOT_Demo_Bezier_FACTOR=0.800000.PNG ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.800000.PNG
!! Images differ: ./ScreenShots/SCREENSHOT_Demo_Bezier_FACTOR=1.000000.PNG ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=1.000000.PNG
!! Images differ: at position 78x399 pixel value: (39, 36, 34) also produces a HTML summary (in this case I moved the mouse just before the last screenshot in order to produce a non matching picture):

Regarding modifications to Demo_Bezier, they just consist in adding one OnFrameEnded method, and modifying __init__ for class BezierListener:
def __init__(self, renderWindow, camera ):
self.timeLapse = 0.0
self.factor = 0.0
self.wireframe = 0
self.renderWindow = renderWindow
sf.FrameListener.__init__(self, renderWindow, camera)

# Are we in test mode?
from optparse import OptionParser
parser = OptionParser()
parser.add_option("-t", "--runtests" , dest="test" ,action="store_true",default=False)
parser.add_option("-r", "--reference", dest="reference",action="store_true",default=False)
(self.options, self.args) = parser.parse_args()
self.firstPass = True

# Make sure target directories do exist
import os
self.screenshotDir = "./Screenshots"
self.referenceDir = "./ReferenceScreenShots"
if (not os.path.exists(self.screenshotDir)):
if (not os.path.exists(self.referenceDir)):

# Empty existing screenshots
pythonProgramName = os.path.splitext(os.path.basename(sys.argv[0]))[0]
if (self.firstPass == True):
targetDir = self.screenshotDir
startString = "SCREENSHOT_"
if self.options.reference:
targetDir = self.referenceDir
startString = "REFERENCE_"
for entry in (os.listdir(targetDir)):
if (entry.find(startString+pythonProgramName)==0):
print "Deleting: ",targetDir+'/'+entry

def frameEnded(self,frameEvent):
# Testing factory #

def partialScreenShot(box,targetFile):
# Get temporary file name for full size screenshot
import tempfile
(handle,tmpFile) = tempfile.mkstemp(prefix="Screenshot_",suffix=".PNG")
# Make screenshot
# Extract needed zone
import Image
im =
cropppedImage = im.crop(box)
# Save to final file
# Remove temp file
if (os.path.exists(tmpFile)):

if (self.options.test == True) or (self.options.reference == True):
# Make sure self.previousFactor is defined
self.previousFactor = None

# Check if factor has changed
if (self.factor != self.previousFactor):

# Define screenshot names
import os
pythonProgramName = os.path.splitext(os.path.basename(sys.argv[0]))[0]
baseName = '_'+pythonProgramName
baseName += '_'+'FACTOR=%05f' % self.factor
baseName += '.PNG'
screenshotFullName = self.screenshotDir+'/'+"SCREENSHOT"+ baseName
referenceFullName = self.referenceDir +'/'+"REFERENCE" + baseName

# Make screenshots
box = (0,0,800,400)
if (self.options.reference):
# Create a new reference
print "NEW_REFERENCE: ",referenceFullName
# Make screenshot
print "SCREENSHOT:",screenshotFullName

# Exit when all tests are completed
if (self.factor == 1.0):
print "Screenshots dumped"

# Get ready for next factor
self.previousFactor = self.factor
return True

Complete source files can be downloaded here:

This a just a first and basic try. It shows that autotesting can be implemented quite easily, without the need to rewrite the demos from scratch.

Right now, I am trying to get rid of the "Rendering subsystem" window. I would like to be able to control those settings from the command line. Does anyone know how to do this?

Any input greatly appreciated,


08-11-2008 14:51:41

I implemented something simular in the SVN a few weeks back :)

I took a slightly different approach in that I use an environment variable to tell the sampleframework that it's in unittest mode and to run for a number of frames, take a snapshot and exit.

I don't compare the snapshots as it's reasonably easy to view the outputs..

Grab the SVN and checkout and unittestall.bat in the ./scripts directory - perhaps you can add your snapshot comparison to this code?



11-11-2008 08:17:50

Thanks Andy. My code comes too late :?, but I will hopefully be able to add some new feature to what you already proposed ;).

I don't compare the snapshots as it's reasonably easy to view the outputs...Actually I prefer to automatically check the snapshots. It makes life easier and allows to check stuff such as antialiasing, which is not so easy to see on a screenshot.

IGrab the SVN and checkout and unittestall.bat in the ./scripts directory - perhaps you can add your snapshot comparison to this code?I did so, but I am stuck with silly path problems as I am unable to figure out which SampleFramework I should work with. I am not able to see the changes I make either to .\demos\ogre\ or .\packages_2.5\ogre\renderer\OGRE\

For your information, I grabbed the SVN to C:\ogredev\python-ogre, but I also have RC1 installed on my machine under C:\PythonOgre1.6RC1.

Is there a way to ask python which module file are used?