[2.1][SOLVED] Render to texture with stencil operations

Design / architecture / roadmap discussions related to future of Ogre3D (version 2.0 and above)

[2.1][SOLVED] Render to texture with stencil operations

Postby zxz » Mon Dec 19, 2016 4:32 pm

Hello,

I have implemented a very basic kind of object highlighting by the following method:

1. Render scene
2. Render highlighted objects, and write a specific value to stencil buffer
3. Render slightly enlarged versions of the highlighted objects, except where the stencil buffer has been written previously

The stencil operation configuration is made in a few stencil passes in our compositor workspace (in 1.8 we used a RenderQueueListener instead).

All that works very well for our purposes. However, when the target of the compositor workspace is a texture target instead of a render window, there is no sign of any stencil operation being carried out.

Is there anything that I have to do in order to get a stencil buffer for my texture target? Is there any other reason for this setup not to work?

Using Ogre 2.1 trunk on Linux with an NVIDIA card.
Last edited by zxz on Wed Dec 21, 2016 4:52 pm, edited 1 time in total.
zxz
Halfling
 
Posts: 85
Kudos: 1
Joined: 16 Apr 2016

Re: [2.1] Render to texture with stencil operations

Postby dark_sylinc » Mon Dec 19, 2016 6:44 pm

This is likely a bug. I'll have to dig a little into it.

Although please check that the RenderTexture has a depth buffer attached with a stencil format. To check that, after having rendered to it (i.e. do this on the second frame), call renderTexture->getDepthBuffer()->getFormat();
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
 
Posts: 3289
Kudos: 502
Joined: 21 Jul 2007
Location: Buenos Aires, Argentina

Re: [2.1] Render to texture with stencil operations

Postby zxz » Mon Dec 19, 2016 9:30 pm

The format is:

Code: Select all
        /// Depth texture format. 32 bits for depth. 8 bits for stencil
        PF_D32_FLOAT_X24_S8_UINT = 101,


which looks fine.

Hmm.
zxz
Halfling
 
Posts: 85
Kudos: 1
Joined: 16 Apr 2016

Re: [2.1] Render to texture with stencil operations

Postby dark_sylinc » Mon Dec 19, 2016 9:38 pm

Thanks. It's most likely an Ogre bug. I was just ruling out that your app could be requesting a non-stencil RTT.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
 
Posts: 3289
Kudos: 502
Joined: 21 Jul 2007
Location: Buenos Aires, Argentina

Re: [2.1] Render to texture with stencil operations

Postby zxz » Mon Dec 19, 2016 9:44 pm

Do you have any hunch regarding what sort of bug could cause this kind of problem? Any ideas where to start looking?

Thanks
zxz
Halfling
 
Posts: 85
Kudos: 1
Joined: 16 Apr 2016

Re: [2.1] Render to texture with stencil operations

Postby dark_sylinc » Tue Dec 20, 2016 12:57 am

Fixed. (Note: 2.1-pso branch; non-pso branch has broken stencil format for a long time)

Thanks for the report.

For this message the author dark_sylinc has received kudos
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
 
Posts: 3289
Kudos: 502
Joined: 21 Jul 2007
Location: Buenos Aires, Argentina

Re: [2.1] Render to texture with stencil operations

Postby zxz » Tue Dec 20, 2016 2:25 pm

Thanks for looking into the issue!

I ported to the 2.1-pso branch in order to try out this fix. That went fairly smoothly, but I still have issues with stencil operations when drawing to texture targets.

The stencil operation is performed nicely when rendering to any render window.

The difference on the 2.1-pso branch with your fix is that the object disappears when highlighted, whereas it just wasn't masking properly on the 2.1 branch.

Here's what my workspace does (created by exactly the same code for render windows and render textures):

1. Draw scene.
2. Setup stencil to write 1, always pass
3. Draw highlighted objects
4. Setup stencil compare function to not equal, comparing to value 1
5. Draw enlarged objects with unlit material
6. Disable stencil
7. Draw overlays and some other stuff

On the 2.1-pso branch, when drawing to a texture, the objects drawn in step 3 and 5 are not visible (but still cast shadows).
zxz
Halfling
 
Posts: 85
Kudos: 1
Joined: 16 Apr 2016

Re: [2.1] Render to texture with stencil operations

Postby dark_sylinc » Tue Dec 20, 2016 11:01 pm

Please note that:

1. The PSO slightly changed the compositor script syntax. The reason for this change was to enable support for explicitly setting dual-sided stencil. There is a sample called "StencilTest" showing how to use stencil. i.e. the most notable change is that some settings have moved to the "both" block:
Code: Select all
pass stencil
{
   check true
   
   mask      0xff
   read_mask   0xff
   
   ref_value   1
   
   both
   {
      pass_op         replace
      depth_fail_op   keep
      fail_op         keep

      comp_func      always_pass
   }
}

Other options instead of "both" are "front" and "back" for setting them independently.

2. You may be thinking the depth buffers are getting shared but for some reason the RTTs are getting independent depth buffers. I'd suggest RenderDoc to check what happened but you're on Linux. So I'd recommend adding Windows support just for the debugging tools. Otherwise, pay close attention or use assert( rtt->getDepthBuffer() == rtt2->getDepthBuffer() ) after having rendered one frame to ensure your RTTs share the depth buffer.
Note it's also possible that if you're doing something in the middle; that this RTT in the middle ends up sharing the same depth buffer and you end up clearing/overwriting it by accident.

3. The RenderWindow cannot share the depth buffer with anyone. So if you depend on the stencil results done to an RTT, the RenderWindow won't be able to use them.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
 
Posts: 3289
Kudos: 502
Joined: 21 Jul 2007
Location: Buenos Aires, Argentina

Re: [2.1] Render to texture with stencil operations

Postby zxz » Tue Dec 20, 2016 11:27 pm

I think there's some confusion regarding my setup. It's a bit difficult to describe these things clearly, when the space of possible configurations is so huge.

While I do have support for multiple RTT's (many views), I do use them independently, and I do expect them to have independent depth buffers. The same issue happens in the case of drawing to a single RTT.

The numbered list in my previous message pretty much represents the pass setup I use in my workspace definition. pass scene (all except highlighted), pass stencil (ref 1, always pass), pass scene (highlighted object rq), pass stencil (ref 1, not equal) pass scene (highlights), pass stencil (disable), pass scene (overlays). All these passes are drawing to the same RTT. So I don't quite understand what might get in between my passes when I draw to RTT.

The difference is that the stencil operation works as expected in a workspace created like this (using an equivalent workspaceDef in both cases):

Code: Select all
workspace = compositorManager->addWorkspace(sceneManager, renderWindow, camera, workspaceDef->getName(), true);


but not when created like this (target is a single RTT):
Code: Select all
workspace = compositorManager->addWorkspace(sceneManager, texture->getBuffer()->getRenderTarget(), camera, workspaceDef->getName(), true);


I am just failing to understand why these two would give a different result in the rendered image. One performing the stencil op as expected, and the other not.
zxz
Halfling
 
Posts: 85
Kudos: 1
Joined: 16 Apr 2016

Re: [2.1] Render to texture with stencil operations

Postby dark_sylinc » Tue Dec 20, 2016 11:37 pm

Would you be able to reproduce this by modifying one of our samples? (so that I can work on the bug)

Handling OpenGL stencil buffers is a PITA as RenderWindow and RTTs follow very different paths. It's not unreasonable to think we have yet another bug.
The bug I fixed was causing RTTs to not actually use a stencil buffer at all even when requested.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
 
Posts: 3289
Kudos: 502
Joined: 21 Jul 2007
Location: Buenos Aires, Argentina

Re: [2.1] Render to texture with stencil operations

Postby zxz » Tue Dec 20, 2016 11:51 pm

I might try to reproduce it in a sample, it just requires a bit of code for setting things up.

I suppose that the bugfix you made now gives me a stencil buffer, where there was none before.

It did change the behavior from no stencil op happening at all with a RTT, to disappearing objects rendered while stencil is enabled.

We'll see if I will get enough time to write a simple reproducer.
zxz
Halfling
 
Posts: 85
Kudos: 1
Joined: 16 Apr 2016

Re: [2.1] Render to texture with stencil operations

Postby zxz » Wed Dec 21, 2016 4:52 pm

Meh. I found a solution to the problem.

It has to do with the separate stencil operations for front and backfaces (on the 2.1-pso branch), which you mentioned.

I expected the backface stencil operation to be irrelevant since I expect backfaces to be culled anyway. If I configure both front and back stencil operations, everything works, even for RTT. It's not clear to me why that is needed only for RTT though. My compositor configuration is done in code, so I didn't have the convenience of the 'both' block. I did make an attempt to duplicate the settings for front and back before, but had missed setting the actual stencil operation for backfaces. The default is to never pass.

Thanks for the help, dark_sylinc.
zxz
Halfling
 
Posts: 85
Kudos: 1
Joined: 16 Apr 2016

Re: [2.1][SOLVED] Render to texture with stencil operations

Postby dark_sylinc » Wed Dec 21, 2016 9:27 pm

I wonder if you stumbled on a different bug, like back face & front face settings being swapped. I've had similar issues with OpenGL due to FBOs being upside down.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
 
Posts: 3289
Kudos: 502
Joined: 21 Jul 2007
Location: Buenos Aires, Argentina

Re: [2.1][SOLVED] Render to texture with stencil operations

Postby zxz » Wed Dec 21, 2016 10:31 pm

That is indeed the case. If I configure everything only for backfaces, I get the opposite result as initially described. It works in RTTs, but not in renderwindows.

Interesting.
zxz
Halfling
 
Posts: 85
Kudos: 1
Joined: 16 Apr 2016

Re: [2.1][SOLVED] Render to texture with stencil operations

Postby dark_sylinc » Wed Dec 21, 2016 10:36 pm

Thanks. I'll add this bug to my TODO list.

At least you're now able to continue development.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
 
Posts: 3289
Kudos: 502
Joined: 21 Jul 2007
Location: Buenos Aires, Argentina

Re: [2.1][SOLVED] Render to texture with stencil operations

Postby zxz » Wed Dec 21, 2016 10:38 pm

I'm happy for now. I don't actually need the separate stencil op's yet anyway. Duplicating as a workaround is not an issue.

Thanks for the quick and precise help.
zxz
Halfling
 
Posts: 85
Kudos: 1
Joined: 16 Apr 2016


Return to Ogre 2.0+

Who is online

Users browsing this forum: claytongarrett, Google [Bot] and 3 guests