[2.1][SOLVED] Render to texture with stencil operations
-
- Gremlin
- Posts: 184
- Joined: Sat Apr 16, 2016 9:25 pm
- x 19
[2.1][SOLVED] Render to texture with stencil operations
Hello,
I have implemented a very basic kind of object highlighting by the following method:
1. Render scene
2. Render highlighted objects, and write a specific value to stencil buffer
3. Render slightly enlarged versions of the highlighted objects, except where the stencil buffer has been written previously
The stencil operation configuration is made in a few stencil passes in our compositor workspace (in 1.8 we used a RenderQueueListener instead).
All that works very well for our purposes. However, when the target of the compositor workspace is a texture target instead of a render window, there is no sign of any stencil operation being carried out.
Is there anything that I have to do in order to get a stencil buffer for my texture target? Is there any other reason for this setup not to work?
Using Ogre 2.1 trunk on Linux with an NVIDIA card.
I have implemented a very basic kind of object highlighting by the following method:
1. Render scene
2. Render highlighted objects, and write a specific value to stencil buffer
3. Render slightly enlarged versions of the highlighted objects, except where the stencil buffer has been written previously
The stencil operation configuration is made in a few stencil passes in our compositor workspace (in 1.8 we used a RenderQueueListener instead).
All that works very well for our purposes. However, when the target of the compositor workspace is a texture target instead of a render window, there is no sign of any stencil operation being carried out.
Is there anything that I have to do in order to get a stencil buffer for my texture target? Is there any other reason for this setup not to work?
Using Ogre 2.1 trunk on Linux with an NVIDIA card.
Last edited by zxz on Wed Dec 21, 2016 4:52 pm, edited 1 time in total.
- dark_sylinc
- OGRE Team Member
- Posts: 5296
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1278
- Contact:
Re: [2.1] Render to texture with stencil operations
This is likely a bug. I'll have to dig a little into it.
Although please check that the RenderTexture has a depth buffer attached with a stencil format. To check that, after having rendered to it (i.e. do this on the second frame), call renderTexture->getDepthBuffer()->getFormat();
Although please check that the RenderTexture has a depth buffer attached with a stencil format. To check that, after having rendered to it (i.e. do this on the second frame), call renderTexture->getDepthBuffer()->getFormat();
-
- Gremlin
- Posts: 184
- Joined: Sat Apr 16, 2016 9:25 pm
- x 19
Re: [2.1] Render to texture with stencil operations
The format is:
which looks fine.
Hmm.
Code: Select all
/// Depth texture format. 32 bits for depth. 8 bits for stencil
PF_D32_FLOAT_X24_S8_UINT = 101,
Hmm.
- dark_sylinc
- OGRE Team Member
- Posts: 5296
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1278
- Contact:
Re: [2.1] Render to texture with stencil operations
Thanks. It's most likely an Ogre bug. I was just ruling out that your app could be requesting a non-stencil RTT.
-
- Gremlin
- Posts: 184
- Joined: Sat Apr 16, 2016 9:25 pm
- x 19
Re: [2.1] Render to texture with stencil operations
Do you have any hunch regarding what sort of bug could cause this kind of problem? Any ideas where to start looking?
Thanks
Thanks
- dark_sylinc
- OGRE Team Member
- Posts: 5296
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1278
- Contact:
Re: [2.1] Render to texture with stencil operations
Fixed. (Note: 2.1-pso branch; non-pso branch has broken stencil format for a long time)
Thanks for the report.
Thanks for the report.
-
- Gremlin
- Posts: 184
- Joined: Sat Apr 16, 2016 9:25 pm
- x 19
Re: [2.1] Render to texture with stencil operations
Thanks for looking into the issue!
I ported to the 2.1-pso branch in order to try out this fix. That went fairly smoothly, but I still have issues with stencil operations when drawing to texture targets.
The stencil operation is performed nicely when rendering to any render window.
The difference on the 2.1-pso branch with your fix is that the object disappears when highlighted, whereas it just wasn't masking properly on the 2.1 branch.
Here's what my workspace does (created by exactly the same code for render windows and render textures):
1. Draw scene.
2. Setup stencil to write 1, always pass
3. Draw highlighted objects
4. Setup stencil compare function to not equal, comparing to value 1
5. Draw enlarged objects with unlit material
6. Disable stencil
7. Draw overlays and some other stuff
On the 2.1-pso branch, when drawing to a texture, the objects drawn in step 3 and 5 are not visible (but still cast shadows).
I ported to the 2.1-pso branch in order to try out this fix. That went fairly smoothly, but I still have issues with stencil operations when drawing to texture targets.
The stencil operation is performed nicely when rendering to any render window.
The difference on the 2.1-pso branch with your fix is that the object disappears when highlighted, whereas it just wasn't masking properly on the 2.1 branch.
Here's what my workspace does (created by exactly the same code for render windows and render textures):
1. Draw scene.
2. Setup stencil to write 1, always pass
3. Draw highlighted objects
4. Setup stencil compare function to not equal, comparing to value 1
5. Draw enlarged objects with unlit material
6. Disable stencil
7. Draw overlays and some other stuff
On the 2.1-pso branch, when drawing to a texture, the objects drawn in step 3 and 5 are not visible (but still cast shadows).
- dark_sylinc
- OGRE Team Member
- Posts: 5296
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1278
- Contact:
Re: [2.1] Render to texture with stencil operations
Please note that:
1. The PSO slightly changed the compositor script syntax. The reason for this change was to enable support for explicitly setting dual-sided stencil. There is a sample called "StencilTest" showing how to use stencil. i.e. the most notable change is that some settings have moved to the "both" block:
Other options instead of "both" are "front" and "back" for setting them independently.
2. You may be thinking the depth buffers are getting shared but for some reason the RTTs are getting independent depth buffers. I'd suggest RenderDoc to check what happened but you're on Linux. So I'd recommend adding Windows support just for the debugging tools. Otherwise, pay close attention or use assert( rtt->getDepthBuffer() == rtt2->getDepthBuffer() ) after having rendered one frame to ensure your RTTs share the depth buffer.
Note it's also possible that if you're doing something in the middle; that this RTT in the middle ends up sharing the same depth buffer and you end up clearing/overwriting it by accident.
3. The RenderWindow cannot share the depth buffer with anyone. So if you depend on the stencil results done to an RTT, the RenderWindow won't be able to use them.
1. The PSO slightly changed the compositor script syntax. The reason for this change was to enable support for explicitly setting dual-sided stencil. There is a sample called "StencilTest" showing how to use stencil. i.e. the most notable change is that some settings have moved to the "both" block:
Code: Select all
pass stencil
{
check true
mask 0xff
read_mask 0xff
ref_value 1
both
{
pass_op replace
depth_fail_op keep
fail_op keep
comp_func always_pass
}
}
2. You may be thinking the depth buffers are getting shared but for some reason the RTTs are getting independent depth buffers. I'd suggest RenderDoc to check what happened but you're on Linux. So I'd recommend adding Windows support just for the debugging tools. Otherwise, pay close attention or use assert( rtt->getDepthBuffer() == rtt2->getDepthBuffer() ) after having rendered one frame to ensure your RTTs share the depth buffer.
Note it's also possible that if you're doing something in the middle; that this RTT in the middle ends up sharing the same depth buffer and you end up clearing/overwriting it by accident.
3. The RenderWindow cannot share the depth buffer with anyone. So if you depend on the stencil results done to an RTT, the RenderWindow won't be able to use them.
-
- Gremlin
- Posts: 184
- Joined: Sat Apr 16, 2016 9:25 pm
- x 19
Re: [2.1] Render to texture with stencil operations
I think there's some confusion regarding my setup. It's a bit difficult to describe these things clearly, when the space of possible configurations is so huge.
While I do have support for multiple RTT's (many views), I do use them independently, and I do expect them to have independent depth buffers. The same issue happens in the case of drawing to a single RTT.
The numbered list in my previous message pretty much represents the pass setup I use in my workspace definition. pass scene (all except highlighted), pass stencil (ref 1, always pass), pass scene (highlighted object rq), pass stencil (ref 1, not equal) pass scene (highlights), pass stencil (disable), pass scene (overlays). All these passes are drawing to the same RTT. So I don't quite understand what might get in between my passes when I draw to RTT.
The difference is that the stencil operation works as expected in a workspace created like this (using an equivalent workspaceDef in both cases):
but not when created like this (target is a single RTT):
I am just failing to understand why these two would give a different result in the rendered image. One performing the stencil op as expected, and the other not.
While I do have support for multiple RTT's (many views), I do use them independently, and I do expect them to have independent depth buffers. The same issue happens in the case of drawing to a single RTT.
The numbered list in my previous message pretty much represents the pass setup I use in my workspace definition. pass scene (all except highlighted), pass stencil (ref 1, always pass), pass scene (highlighted object rq), pass stencil (ref 1, not equal) pass scene (highlights), pass stencil (disable), pass scene (overlays). All these passes are drawing to the same RTT. So I don't quite understand what might get in between my passes when I draw to RTT.
The difference is that the stencil operation works as expected in a workspace created like this (using an equivalent workspaceDef in both cases):
Code: Select all
workspace = compositorManager->addWorkspace(sceneManager, renderWindow, camera, workspaceDef->getName(), true);
Code: Select all
workspace = compositorManager->addWorkspace(sceneManager, texture->getBuffer()->getRenderTarget(), camera, workspaceDef->getName(), true);
- dark_sylinc
- OGRE Team Member
- Posts: 5296
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1278
- Contact:
Re: [2.1] Render to texture with stencil operations
Would you be able to reproduce this by modifying one of our samples? (so that I can work on the bug)
Handling OpenGL stencil buffers is a PITA as RenderWindow and RTTs follow very different paths. It's not unreasonable to think we have yet another bug.
The bug I fixed was causing RTTs to not actually use a stencil buffer at all even when requested.
Handling OpenGL stencil buffers is a PITA as RenderWindow and RTTs follow very different paths. It's not unreasonable to think we have yet another bug.
The bug I fixed was causing RTTs to not actually use a stencil buffer at all even when requested.
-
- Gremlin
- Posts: 184
- Joined: Sat Apr 16, 2016 9:25 pm
- x 19
Re: [2.1] Render to texture with stencil operations
I might try to reproduce it in a sample, it just requires a bit of code for setting things up.
I suppose that the bugfix you made now gives me a stencil buffer, where there was none before.
It did change the behavior from no stencil op happening at all with a RTT, to disappearing objects rendered while stencil is enabled.
We'll see if I will get enough time to write a simple reproducer.
I suppose that the bugfix you made now gives me a stencil buffer, where there was none before.
It did change the behavior from no stencil op happening at all with a RTT, to disappearing objects rendered while stencil is enabled.
We'll see if I will get enough time to write a simple reproducer.
-
- Gremlin
- Posts: 184
- Joined: Sat Apr 16, 2016 9:25 pm
- x 19
Re: [2.1] Render to texture with stencil operations
Meh. I found a solution to the problem.
It has to do with the separate stencil operations for front and backfaces (on the 2.1-pso branch), which you mentioned.
I expected the backface stencil operation to be irrelevant since I expect backfaces to be culled anyway. If I configure both front and back stencil operations, everything works, even for RTT. It's not clear to me why that is needed only for RTT though. My compositor configuration is done in code, so I didn't have the convenience of the 'both' block. I did make an attempt to duplicate the settings for front and back before, but had missed setting the actual stencil operation for backfaces. The default is to never pass.
Thanks for the help, dark_sylinc.
It has to do with the separate stencil operations for front and backfaces (on the 2.1-pso branch), which you mentioned.
I expected the backface stencil operation to be irrelevant since I expect backfaces to be culled anyway. If I configure both front and back stencil operations, everything works, even for RTT. It's not clear to me why that is needed only for RTT though. My compositor configuration is done in code, so I didn't have the convenience of the 'both' block. I did make an attempt to duplicate the settings for front and back before, but had missed setting the actual stencil operation for backfaces. The default is to never pass.
Thanks for the help, dark_sylinc.
- dark_sylinc
- OGRE Team Member
- Posts: 5296
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1278
- Contact:
Re: [2.1][SOLVED] Render to texture with stencil operations
I wonder if you stumbled on a different bug, like back face & front face settings being swapped. I've had similar issues with OpenGL due to FBOs being upside down.
-
- Gremlin
- Posts: 184
- Joined: Sat Apr 16, 2016 9:25 pm
- x 19
Re: [2.1][SOLVED] Render to texture with stencil operations
That is indeed the case. If I configure everything only for backfaces, I get the opposite result as initially described. It works in RTTs, but not in renderwindows.
Interesting.
Interesting.
- dark_sylinc
- OGRE Team Member
- Posts: 5296
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1278
- Contact:
Re: [2.1][SOLVED] Render to texture with stencil operations
Thanks. I'll add this bug to my TODO list.
At least you're now able to continue development.
At least you're now able to continue development.
-
- Gremlin
- Posts: 184
- Joined: Sat Apr 16, 2016 9:25 pm
- x 19
Re: [2.1][SOLVED] Render to texture with stencil operations
I'm happy for now. I don't actually need the separate stencil op's yet anyway. Duplicating as a workaround is not an issue.
Thanks for the quick and precise help.
Thanks for the quick and precise help.