[2.1][SOLVED] Render to texture with stencil operations

Discussion area about developing with Ogre-Next (2.1, 2.2 and beyond)


Post Reply
zxz
Gremlin
Posts: 184
Joined: Sat Apr 16, 2016 9:25 pm
x 19

[2.1][SOLVED] Render to texture with stencil operations

Post by zxz »

Hello,

I have implemented a very basic kind of object highlighting by the following method:

1. Render scene
2. Render highlighted objects, and write a specific value to stencil buffer
3. Render slightly enlarged versions of the highlighted objects, except where the stencil buffer has been written previously

The stencil operation configuration is made in a few stencil passes in our compositor workspace (in 1.8 we used a RenderQueueListener instead).

All that works very well for our purposes. However, when the target of the compositor workspace is a texture target instead of a render window, there is no sign of any stencil operation being carried out.

Is there anything that I have to do in order to get a stencil buffer for my texture target? Is there any other reason for this setup not to work?

Using Ogre 2.1 trunk on Linux with an NVIDIA card.
Last edited by zxz on Wed Dec 21, 2016 4:52 pm, edited 1 time in total.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [2.1] Render to texture with stencil operations

Post by dark_sylinc »

This is likely a bug. I'll have to dig a little into it.

Although please check that the RenderTexture has a depth buffer attached with a stencil format. To check that, after having rendered to it (i.e. do this on the second frame), call renderTexture->getDepthBuffer()->getFormat();
zxz
Gremlin
Posts: 184
Joined: Sat Apr 16, 2016 9:25 pm
x 19

Re: [2.1] Render to texture with stencil operations

Post by zxz »

The format is:

Code: Select all

        /// Depth texture format. 32 bits for depth. 8 bits for stencil
        PF_D32_FLOAT_X24_S8_UINT = 101,
which looks fine.

Hmm.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [2.1] Render to texture with stencil operations

Post by dark_sylinc »

Thanks. It's most likely an Ogre bug. I was just ruling out that your app could be requesting a non-stencil RTT.
zxz
Gremlin
Posts: 184
Joined: Sat Apr 16, 2016 9:25 pm
x 19

Re: [2.1] Render to texture with stencil operations

Post by zxz »

Do you have any hunch regarding what sort of bug could cause this kind of problem? Any ideas where to start looking?

Thanks
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [2.1] Render to texture with stencil operations

Post by dark_sylinc »

Fixed. (Note: 2.1-pso branch; non-pso branch has broken stencil format for a long time)

Thanks for the report.
zxz
Gremlin
Posts: 184
Joined: Sat Apr 16, 2016 9:25 pm
x 19

Re: [2.1] Render to texture with stencil operations

Post by zxz »

Thanks for looking into the issue!

I ported to the 2.1-pso branch in order to try out this fix. That went fairly smoothly, but I still have issues with stencil operations when drawing to texture targets.

The stencil operation is performed nicely when rendering to any render window.

The difference on the 2.1-pso branch with your fix is that the object disappears when highlighted, whereas it just wasn't masking properly on the 2.1 branch.

Here's what my workspace does (created by exactly the same code for render windows and render textures):

1. Draw scene.
2. Setup stencil to write 1, always pass
3. Draw highlighted objects
4. Setup stencil compare function to not equal, comparing to value 1
5. Draw enlarged objects with unlit material
6. Disable stencil
7. Draw overlays and some other stuff

On the 2.1-pso branch, when drawing to a texture, the objects drawn in step 3 and 5 are not visible (but still cast shadows).
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [2.1] Render to texture with stencil operations

Post by dark_sylinc »

Please note that:

1. The PSO slightly changed the compositor script syntax. The reason for this change was to enable support for explicitly setting dual-sided stencil. There is a sample called "StencilTest" showing how to use stencil. i.e. the most notable change is that some settings have moved to the "both" block:

Code: Select all

pass stencil
{
	check true
	
	mask		0xff
	read_mask	0xff
	
	ref_value	1
	
	both
	{
		pass_op			replace
		depth_fail_op	keep
		fail_op			keep

		comp_func		always_pass
	}
}
Other options instead of "both" are "front" and "back" for setting them independently.

2. You may be thinking the depth buffers are getting shared but for some reason the RTTs are getting independent depth buffers. I'd suggest RenderDoc to check what happened but you're on Linux. So I'd recommend adding Windows support just for the debugging tools. Otherwise, pay close attention or use assert( rtt->getDepthBuffer() == rtt2->getDepthBuffer() ) after having rendered one frame to ensure your RTTs share the depth buffer.
Note it's also possible that if you're doing something in the middle; that this RTT in the middle ends up sharing the same depth buffer and you end up clearing/overwriting it by accident.

3. The RenderWindow cannot share the depth buffer with anyone. So if you depend on the stencil results done to an RTT, the RenderWindow won't be able to use them.
zxz
Gremlin
Posts: 184
Joined: Sat Apr 16, 2016 9:25 pm
x 19

Re: [2.1] Render to texture with stencil operations

Post by zxz »

I think there's some confusion regarding my setup. It's a bit difficult to describe these things clearly, when the space of possible configurations is so huge.

While I do have support for multiple RTT's (many views), I do use them independently, and I do expect them to have independent depth buffers. The same issue happens in the case of drawing to a single RTT.

The numbered list in my previous message pretty much represents the pass setup I use in my workspace definition. pass scene (all except highlighted), pass stencil (ref 1, always pass), pass scene (highlighted object rq), pass stencil (ref 1, not equal) pass scene (highlights), pass stencil (disable), pass scene (overlays). All these passes are drawing to the same RTT. So I don't quite understand what might get in between my passes when I draw to RTT.

The difference is that the stencil operation works as expected in a workspace created like this (using an equivalent workspaceDef in both cases):

Code: Select all

workspace = compositorManager->addWorkspace(sceneManager, renderWindow, camera, workspaceDef->getName(), true);
but not when created like this (target is a single RTT):

Code: Select all

workspace = compositorManager->addWorkspace(sceneManager, texture->getBuffer()->getRenderTarget(), camera, workspaceDef->getName(), true);
I am just failing to understand why these two would give a different result in the rendered image. One performing the stencil op as expected, and the other not.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [2.1] Render to texture with stencil operations

Post by dark_sylinc »

Would you be able to reproduce this by modifying one of our samples? (so that I can work on the bug)

Handling OpenGL stencil buffers is a PITA as RenderWindow and RTTs follow very different paths. It's not unreasonable to think we have yet another bug.
The bug I fixed was causing RTTs to not actually use a stencil buffer at all even when requested.
zxz
Gremlin
Posts: 184
Joined: Sat Apr 16, 2016 9:25 pm
x 19

Re: [2.1] Render to texture with stencil operations

Post by zxz »

I might try to reproduce it in a sample, it just requires a bit of code for setting things up.

I suppose that the bugfix you made now gives me a stencil buffer, where there was none before.

It did change the behavior from no stencil op happening at all with a RTT, to disappearing objects rendered while stencil is enabled.

We'll see if I will get enough time to write a simple reproducer.
zxz
Gremlin
Posts: 184
Joined: Sat Apr 16, 2016 9:25 pm
x 19

Re: [2.1] Render to texture with stencil operations

Post by zxz »

Meh. I found a solution to the problem.

It has to do with the separate stencil operations for front and backfaces (on the 2.1-pso branch), which you mentioned.

I expected the backface stencil operation to be irrelevant since I expect backfaces to be culled anyway. If I configure both front and back stencil operations, everything works, even for RTT. It's not clear to me why that is needed only for RTT though. My compositor configuration is done in code, so I didn't have the convenience of the 'both' block. I did make an attempt to duplicate the settings for front and back before, but had missed setting the actual stencil operation for backfaces. The default is to never pass.

Thanks for the help, dark_sylinc.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [2.1][SOLVED] Render to texture with stencil operations

Post by dark_sylinc »

I wonder if you stumbled on a different bug, like back face & front face settings being swapped. I've had similar issues with OpenGL due to FBOs being upside down.
zxz
Gremlin
Posts: 184
Joined: Sat Apr 16, 2016 9:25 pm
x 19

Re: [2.1][SOLVED] Render to texture with stencil operations

Post by zxz »

That is indeed the case. If I configure everything only for backfaces, I get the opposite result as initially described. It works in RTTs, but not in renderwindows.

Interesting.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [2.1][SOLVED] Render to texture with stencil operations

Post by dark_sylinc »

Thanks. I'll add this bug to my TODO list.

At least you're now able to continue development.
zxz
Gremlin
Posts: 184
Joined: Sat Apr 16, 2016 9:25 pm
x 19

Re: [2.1][SOLVED] Render to texture with stencil operations

Post by zxz »

I'm happy for now. I don't actually need the separate stencil op's yet anyway. Duplicating as a workaround is not an issue.

Thanks for the quick and precise help.
Post Reply