Demos autotest

Hi all,

There are often broken demos... This is natural, there are so many features to test and, as far as I know no autotest feature is available... yet. Today I tried to implement such a feature on the demos. This would help to check if everything is ok, right upon package compilation, or in order to test a new hardware platform, or different openGL/DirectX settings...

My idea is to add switches to the demos, so that they automatically take screenshots that can later be compared to reference screenshots.

I implemented my first tests on Demo_Bezier.py.

In order to take reference screenshots, i introduced the --reference switch:

C:\PythonOgre_1.6RC1\demos\ogre>demo_bezier.py --reference
NEW_REFERENCE: ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.000000.PNG
NEW_REFERENCE: ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.200000.PNG
NEW_REFERENCE: ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.400000.PNG
NEW_REFERENCE: ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.600000.PNG
NEW_REFERENCE: ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.800000.PNG
NEW_REFERENCE: ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=1.000000.PNG
Screenshots dumped


And then, at any time, you may run the same demo with the --runtests switch:
SCREENSHOT: ./Screenshots/SCREENSHOT_Demo_Bezier_FACTOR=0.000000.PNG
SCREENSHOT: ./Screenshots/SCREENSHOT_Demo_Bezier_FACTOR=0.200000.PNG
SCREENSHOT: ./Screenshots/SCREENSHOT_Demo_Bezier_FACTOR=0.400000.PNG
SCREENSHOT: ./Screenshots/SCREENSHOT_Demo_Bezier_FACTOR=0.600000.PNG
SCREENSHOT: ./Screenshots/SCREENSHOT_Demo_Bezier_FACTOR=0.800000.PNG
SCREENSHOT: ./Screenshots/SCREENSHOT_Demo_Bezier_FACTOR=1.000000.PNG
Screenshots dumped

Finally, autotest.py makes a comprehensive report of the difference between files.
C:\PythonOgre_1.6RC1\demos\ogre>autotest.py
Images match: ./ScreenShots/SCREENSHOT_Demo_Bezier_FACTOR=0.000000.PNG ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.000000.PNG
Images match: ./ScreenShots/SCREENSHOT_Demo_Bezier_FACTOR=0.200000.PNG ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.200000.PNG
Images match: ./ScreenShots/SCREENSHOT_Demo_Bezier_FACTOR=0.400000.PNG ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.400000.PNG
Images match: ./ScreenShots/SCREENSHOT_Demo_Bezier_FACTOR=0.600000.PNG ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.600000.PNG
Images match: ./ScreenShots/SCREENSHOT_Demo_Bezier_FACTOR=0.800000.PNG ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=0.800000.PNG
!! Images differ: ./ScreenShots/SCREENSHOT_Demo_Bezier_FACTOR=1.000000.PNG ./ReferenceScreenShots/REFERENCE_Demo_Bezier_FACTOR=1.000000.PNG
!! Images differ: at position 78x399 pixel value: (39, 36, 34)

autotest.py also produces a HTML summary (in this case I moved the mouse just before the last screenshot in order to produce a non matching picture):


Regarding modifications to Demo_Bezier, they just consist in adding one OnFrameEnded method, and modifying __init__ for class BezierListener:
def __init__(self, renderWindow, camera ):
self.timeLapse = 0.0
self.factor = 0.0
self.wireframe = 0
self.renderWindow = renderWindow
sf.FrameListener.__init__(self, renderWindow, camera)

# Are we in test mode?
from optparse import OptionParser
parser = OptionParser()
parser.add_option("-t", "--runtests" , dest="test" ,action="store_true",default=False)
parser.add_option("-r", "--reference", dest="reference",action="store_true",default=False)
(self.options, self.args) = parser.parse_args()
self.firstPass = True

# Make sure target directories do exist
import os
self.screenshotDir = "./Screenshots"
self.referenceDir = "./ReferenceScreenShots"
if (not os.path.exists(self.screenshotDir)):
os.makedirs(self.screenshotDir)
if (not os.path.exists(self.referenceDir)):
os.makedirs(self.referenceDir)

# Empty existing screenshots
pythonProgramName = os.path.splitext(os.path.basename(sys.argv[0]))[0]
if (self.firstPass == True):
targetDir = self.screenshotDir
startString = "SCREENSHOT_"
if self.options.reference:
targetDir = self.referenceDir
startString = "REFERENCE_"
for entry in (os.listdir(targetDir)):
if (entry.find(startString+pythonProgramName)==0):
os.unlink(targetDir+'/'+entry)
print "Deleting: ",targetDir+'/'+entry


def frameEnded(self,frameEvent):
###################
# Testing factory #
###################

def partialScreenShot(box,targetFile):
# Get temporary file name for full size screenshot
import tempfile
(handle,tmpFile) = tempfile.mkstemp(prefix="Screenshot_",suffix=".PNG")
os.close(handle)
# Make screenshot
self.renderWindow.writeContentsToFile(tmpFile)
# Extract needed zone
import Image
im = Image.open(tmpFile)
cropppedImage = im.crop(box)
# Save to final file
cropppedImage.save(targetFile)
# Remove temp file
if (os.path.exists(tmpFile)):
os.unlink(tmpFile)
return(True)

if (self.options.test == True) or (self.options.reference == True):
# Make sure self.previousFactor is defined
try:
self.previousFactor
except:
self.previousFactor = None

# Check if factor has changed
if (self.factor != self.previousFactor):

# Define screenshot names
import os
pythonProgramName = os.path.splitext(os.path.basename(sys.argv[0]))[0]
baseName = '_'+pythonProgramName
baseName += '_'+'FACTOR=%05f' % self.factor
baseName.replace(".","dot")
baseName += '.PNG'
screenshotFullName = self.screenshotDir+'/'+"SCREENSHOT"+ baseName
referenceFullName = self.referenceDir +'/'+"REFERENCE" + baseName

# Make screenshots
box = (0,0,800,400)
if (self.options.reference):
# Create a new reference
partialScreenShot(box,referenceFullName)
print "NEW_REFERENCE: ",referenceFullName
else:
# Make screenshot
partialScreenShot(box,screenshotFullName)
print "SCREENSHOT:",screenshotFullName

# Exit when all tests are completed
if (self.factor == 1.0):
print "Screenshots dumped"
quit()

# Get ready for next factor
self.previousFactor = self.factor
return True

Complete source files can be downloaded here:
http://chpod.free.fr/screenshots/20081106_Demo_Bezier.py
http://chpod.free.fr/screenshots/20081106_autotest.py

This a just a first and basic try. It shows that autotesting can be implemented quite easily, without the need to rewrite the demos from scratch.

Right now, I am trying to get rid of the "Rendering subsystem" window. I would like to be able to control those settings from the command line. Does anyone know how to do this?

Any input greatly appreciated,
Thanks,
chpod

I implemented something simular in the SVN a few weeks back

I took a slightly different approach in that I use an environment variable to tell the sampleframework that it's in unittest mode and to run for a number of frames, take a snapshot and exit.

I don't compare the snapshots as it's reasonably easy to view the outputs..

Grab the SVN and checkout unittest.py and unittestall.bat in the ./scripts directory - perhaps you can add your snapshot comparison to this code?

Regards
Andy

Thanks Andy. My code comes too late , but I will hopefully be able to add some new feature to what you already proposed .

I don't compare the snapshots as it's reasonably easy to view the outputs...Actually I prefer to automatically check the snapshots. It makes life easier and allows to check stuff such as antialiasing, which is not so easy to see on a screenshot.

IGrab the SVN and checkout unittest.py and unittestall.bat in the ./scripts directory - perhaps you can add your snapshot comparison to this code?I did so, but I am stuck with silly path problems as I am unable to figure out which SampleFramework I should work with. I am not able to see the changes I make either to .\demos\ogre\sf_OIS.py or .\packages_2.5\ogre\renderer\OGRE\sf_OIS.py.

For your information, I grabbed the SVN to C:\ogredev\python-ogre, but I also have RC1 installed on my machine under C:\PythonOgre1.6RC1.

Is there a way to ask python which module file are used?

Cheers,
chpod