'Tindalos' (Ogre v2.0) SceneManager redesign

Discussion area about developing or extending OGRE, adding plugins for it or building applications on it. No newbie questions please, use the Help forum for that.
Post Reply
Chaster
OGRE Expert User
OGRE Expert User
Posts: 557
Joined: Wed May 05, 2004 3:19 pm
Location: Portland, OR, USA
Contact:

Post by Chaster »

sinbad wrote:I've renamed this thread since I've decided that these changes will go into Ogre v2.0 aka 'Tindalos' and not v1.6 aka 'Shoggoth'. I'll be developing it in parallel but the interface changes are so significant as to warrant insulating people from them if they just want the other 1.6 enhancements - the Google Summer of Code projects will be part of 1.6 for example, among other things. I also can't think of any other change that deserves a v2.0 moniker more than this one.
*nods* Hmmm, at least my Portal manager won't be rendered obsolete immediately... :lol:
As for abstraction vs complexity - yes in many ways abstraction can be used to reduce complexity if done right. The existing scene manager design originates from the very first version of Ogre and has suffered growing pains since then, it's time to not so much wipe the slate clean, but to rethink everything from first principles, given what I've learned since that time (which is a lot, thanks in many ways to the diversity of this community).
A difficult step to take - major architectural changes. But obviously, Ogre has developed such a following now that I think you can do so with the comfort of knowing that everyone will support and encourage it.
Here's something else to lay on you: in Tindalos I'm almost certain to be removing the need to use SceneNode at all. There has to be something which provides transform information to the renderable objects, but it won't have to be a SceneNode, although clearly that's a rather useful structure to use for many people. By decoupling this such that objects only need a generalised 'transform source', you get quite a lot of benefits. One of which is being able to tie Ogre objects directly into your own external world management system. Having an externally managed world with just incoming sync events to drive things like SM spatial structure updates was something I wish I'd done at the start, so now I think it's time to do it. It's made possible by the fact that custom SceneNodes will no longer be the mechanism for updating custom SM structures.
Whoah. That is, like HERESY... No, j/k. So you will basically have something outside of Ogre control spatial info, eh? That's a pretty good idea. Will eliminate all the unnecessary duplication that goes on when integrating with Physics (for example)... Of course, that seems to me like it is such a low level change that everything coded for pre 2.x Ogre will require complete and thorough re-implementation (but perhaps I am being hasty when I say that...)..
The other thing will be a redesigned event system. I plan on funneling all callbacks via a generic event system which can be listened in on and extended by custom plugins so new events can be added without having to bind to specific custom interfaces. External world management systems will be required to spit events into this system to notify of object transform changes (and yes, this means separate threads can be used for each purpose).
Sounds good to me.
The scope of this review is going way beyond just SceneManager techniques, I'm really looking at the whole world management & integration systems very closely looking for ways it could be done better. Clearly it's very daunting a task and will take a while, so be patient with me please. :) You should know by now that I prefer to take a bit of time over these things and try to cover all the angles.


Definitely a major X.0 release (versus a minor 0.x release).

Chaster
Shadow007
Gremlin
Posts: 185
Joined: Sat May 07, 2005 3:27 pm

Definately ...

Post by Shadow007 »

Quite a grand task, ... and quite aptly named !!!
After a quick wikipedia look, the Tindalos Hounds seem to be quite fond of 'non euclidian space' :)
User avatar
tau
Silver Sponsor
Silver Sponsor
Posts: 413
Joined: Wed Feb 11, 2004 11:44 am
Location: Austin (that's kept wierd :))

Post by tau »

sinbad wrote:I've renamed this thread since I've decided that these changes will go into Ogre v2.0 aka 'Tindalos' and not v1.6 aka 'Shoggoth'.

I know, I may be annoying :) but what is the very very very rough time frame for v2.0? I will be satisfied with an answer like "1 year" or "2 years"... What is your projection, Sinbad? Sinbad, you are not making a promise here, so dont be afraid to give a rough estimate, atleast we know the roadmap. the changes you are talking about (here and down) are the changes I mostly waiting for, and I'm starting feeling that I may join my forces with ogre team as well to give as much help as I can (I'm not very experienced in graphics coding, but tools and algorithms are my strong sides).
sinbad wrote: Here's something else to lay on you: in Tindalos I'm almost certain to be removing the need to use SceneNode at all. There has to be something which provides transform information to the renderable objects, but it won't have to be a SceneNode, although clearly that's a rather useful structure to use for many people.
The first time I read this - I felt fear :) but then I just realized that SceneNodes are really getting into my way when I combine Ogre with any other physics or sound engines - I have 2-3 separate worlds inside each engine, and it's really not elegant!
sinbad wrote: The other thing will be a redesigned event system. I plan on funneling all callbacks via a generic event system which can be listened in on and extended by custom plugins
A good example of such system will be the OSGi specification used as the base in Eclipse IDE. ;) If Ogre would have similar open framework, or an API to be plugged in similar framework - that would be the next step in the evolution :D
If Ogre will be pure event driven, it will be very easy to implement different plugins, or even rout events over network and create truly distributed systems for MMORPG lovers :)

Also, IMHO, Ogre is a good example of excellent OOP code, I wish it was using more decoupling technics, for instance it would be a good practice to use Interfaces(C++ doesnt have pure interfaces, but it can easily implemented), so no implementation would be inforced.
Here is a small example. I was going to implement true RTTI for SceneNodes, but those are using a concrete inheritance, which really gets in my way, and to complete such task I need to change Ogre guts which is bad in terms of compatibility with next versions. If SceneNode was based off Interfaces, I would provide my own implementation, and Ogre would pick it up, and be happy.

Another my thought is that there is a big abuse of the Singleton pattern in Ogre. Singletons are good in small or straight-forward applications, but I'm kind of person who pushes it to the limits :) In my case, I would like to have several Root classes to completely separate worlds, and resources (I may want to have different textures with the same name inside different worlds), etc... I wish, there was a way also to have some classes in two modes: Singleton and not.

Just my 2 cents.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

Praetor wrote: Are you saying that one method for managing a scene in 2.0 will be event-based? As in, the scene just sits there dumbly, and you control all its aspects by externally raising events (like, entity moved or rotated, material switched, etc.)?
I'd only gotten to the stage of thinking about transforms, since this is what is needed here, but in theory anything can be an event source or sink. I'm not sure I'd try to add events for every single thing you can do to a scene, but the point is that new events can be added whenever, so if you wanted to make material changes event based, you'd just have to add the event type and provide an event listener that dealt with it, thus allowing you to make changes across threads.

The general idea is that we have 2 event queues, and as soon as a frame update is requested (note frame, not rendertarget) we would switch the event queues over, like a double-buffered system. All new events will get pushed into the new queue whilst we process the old queue, so your other threads can go on making changes - and also any new events raised whilst processing this event queue will go to this other queue, preventing any infinite event loops. I may have to categorise events depending on when they are typically consumed - e.g. world management events are consumed at frame start, etc.

And yes, I'm thinking that SceneNode keeps a track on attached objects for example, but only for the purposes of raising transform change events. MovableObject won't know about its parent at all anymore (note - they will be managed by shared pointers). These events will be used to both update the MovableObject's transform but also to tell the SceneManager that's controlling the object's parent Locality (which it will know about) about the change. This is still musing to some degree though, I'll need to do more detailed work on this concept yet. For example, how to handle telling the transform source that the object has crossed a Locality boundary - one of the advantages of this is that you don't rely entirely on a Locality link from SceneNode which can be tricky if you have deep SceneNode structures with large transforms at intermediate layers, since a child could cross a boundary whilst the rest of the tree does not. This way, Locality changes 'bubble up' the SceneNode tree from the object level instead which is a bit more intuitive given the bounds information used to trigger it.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

@tau: Re timescales: hell, I dunno. I'm not sure how a completely inaccurate finger-in-the-air estimate would do anything but damage. But definitely not 2 years, and hopefully not 1 year this time. But who knows at this stage.

SceneNode is concrete because that's the way the SM customisation was designed. As you can see that's one of the aspects that will change.

Root has to be a singleton, no option. If you want separate scenes, use multiple SceneManagers (or unconnected Localities eventually ;)). If you want separate resources, use separate Resource groups. As for naming being unique, well, sorry we have to have some predictable structure here. How hard is it for you to use a prefix for different worlds?
User avatar
tau
Silver Sponsor
Silver Sponsor
Posts: 413
Joined: Wed Feb 11, 2004 11:44 am
Location: Austin (that's kept wierd :))

Post by tau »

sinbad wrote:@tau: Re timescales: hell, I dunno. I'm not sure how a completely inaccurate finger-in-the-air estimate would do anything but damage. But definitely not 2 years, and hopefully not 1 year this time. But who knows at this stage.
I know, it's difficult, and emotional :D However, you just gave me a rough estimate and I'm satisfied: now I know, that I'm going to follow a very flexible design in my project, and be ready to switch to Tindalos when time will come.
sinbad wrote: SceneNode is concrete because that's the way the SM customisation was designed. As you can see that's one of the aspects that will change.

Right, and I'm glad it's got the attention now ;)
sinbad wrote: Root has to be a singleton, no option. If you want separate scenes, use multiple SceneManagers (or unconnected Localities eventually ;)). If you want separate resources, use separate Resource groups. As for naming being unique, well, sorry we have to have some predictable structure here. How hard is it for you to use a prefix for different worlds?
Yes, I can use workarounds, I'm just whining for a pure approach, but life is rough, and I'm ready for the sacrifices :)
I cannot complain here, Ogre is a good match for most of the projects (if not for all of them), just wanted you to know my opinion, in the hope that it might give you a spark of new ideas.

Hey, Xavier ;) I think, it would be a good idea to start the second book for the Tindalos in couple of months ;)
User avatar
jwatte
Gnome
Posts: 347
Joined: Sat Feb 05, 2005 12:56 am

Post by jwatte »

hen you try to figure out all of these new design decisions, that probably nobody has done before
You probably haven't been in computer graphics for very long :-)

Back in the '80s, on Silicon Graphics machines, they solved a bunch of these problems already. And again in the '90s. The big difference now is that everybody can afford to run the solutions, not just big organizations with dozens of millions of dollars.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

Absolutely. The same goes for so many technologies, like early database systems. I used to talk to grizzled DB techs who would shake their heads when MS released a new version of SQL Server with a 'new' feature, and they'd tell me how they solved that in 1982 on a machine with a hundred-thousandth of the processing power :) Very humbling.

I started Ogre when I had a bunch of ideas but really no clue whether I knew what I was doing. 6-7 years later I still feel the same way :)
Last edited by sinbad on Mon Apr 02, 2007 6:23 pm, edited 1 time in total.
User avatar
raicuandi
Gargoyle
Posts: 1092
Joined: Wed Nov 09, 2005 12:56 pm
Location: Adelaide, Australia
Contact:

Post by raicuandi »

(born '89, been around computer graphics for ~3 years)

Then how come people are re-doing all of these? Why don't they just "recycle" the technologies they developed eons back?? They could adapt it a bit for the new shiny stuff, but what else is there to it?
User avatar
jwatte
Gnome
Posts: 347
Joined: Sat Feb 05, 2005 12:56 am

Post by jwatte »

Why don't they just "recycle" the technologies they developed eons back?? They could adapt it a bit for the new shiny stuff, but what else is there to it?
Because it's more fun to create stuff from scratch than just re-implement already tried and tested ideas?

In all honesty, though, some other problems (having to do with real time shading) weren't done in the '80s or '90s, but may have been done either offline, or as research, in the last 10 years. An engine like Unreal 3 will be an amalgamation of known techniques, adapted research, and topic-specific innovation (but less of the latter than you might think).
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

sinbad wrote:How hard is it for you to use a prefix for different worlds?
Pretty damn hard actually. That's the solution I'm working on now, but unfortunately some things are just too automatic to intercept and rename. Material scripts for instance get compiled and automatically load textures and gpu programs. I'm not worrying about it too much until I finish the new material compiler, which should let me sneak in there and change names easily. But, what would the disadvantage be of storing resource names per-group? So when you go to load a resource you check a source group first, and only if it isn't found you move on to other groups? Resources would be uniquely named per-group, but not necessarily system-wide.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

Thing is, sharing between groups is pretty common - having a common subset of resources and level-specific data for example. I don't really see what benefit making resource names group-specific would have except confusion and ambiguity. Bear in mind that you can use same-named files if you want by putting them in subfolders, like Quake3 did.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

I understand your points, and accept the burden is on me, not Ogre, to solve the problem. I actually have files in separate zip archives, but I must be able to load multiple archives at once, and not have resources clash with each other, whether their names overlap or not.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

After poking around a bit and thinking about the current scene managers I think I can articulate something that I think is important. The systems that handle visibility, paging, and terrain need to be completely separated. Think of the PLSM. It is a very complex project, since it supports all three functions. Imagine if a more generic paging system could be plugged in, allow PLSM to handle terrain, and then any visibility system could be added (octree, all HOQ based, etc.).
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

I think I've already covered that. And unfortunately they cannot be entirely separate, although to some degree they can be.

Locality is (potentially) a unit of paging. Deciding when to page is basically about when you expect to need / not need a Locality anymore, which can be a product of one of two things:

A. Automatic paging
This is where you have no specific 'paging points' created by a level designer, it's all automatic. For this you need:
1. Distance to the next point at which you could see the contents of the Locality. Note that this distance might not be as-the-crow-flies.
2. The speed you're likely to be travelling

The SceneManager controlling the Locality is the only one that can realistically answer question 1. It's a fairly simple stand-alone question when dealing with terrain, but for, say, a twisting cavern system it might be highly dependent on internal structure, not raw distance. You can't get away from needing the visual partitioning system to evaluate the paging in that situation and thus they cannot be entirely decoupled.

B. Manual paging
This is where a level designer assigns 'pinch points' where paging occurs. The advantage of this is that you can plan it for quieter play points so paging won't kick in when you're in the middle of a pitched battle.


My intention is that for A the external system will provide information about speed, which will be passed to the Locality the camera is in (and cascade to other Localities) to determine the set of Localities that should be in memory right now. The Locality is then going to ask its manager what connections are in range, or going to be in range. That list can be passed back to the paging manager (via the universe manager) which can then decide what to start paging in and what to start paging out.

B will probably be dealt with through events of some kind raised by the SceneManager as the camera traverses a defined pinch point.

Terrain is just a type of world geometry and yes that will be separated from the SceneManager although it can optionally have tight links with the SceneManager if required. Some visibility systems have a tight coupling between these 2 things so it must be allowed, if not assumed any more.
User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2653
Joined: Wed Sep 24, 2003 8:07 am
Location: Haute Garonne, France
x 4
Contact:

Post by tuan kuranes »

Already fond of Tindalos.

I'll just post some little notes of mine on scene management:

- Just to make you aware of, in plsm2 case, the most painful thing was the "Multi-camera Paging, with camera possibly in different pages". Specially when trying to handle some coherency ( Visibility "cache"). In CHC case, it even ended having per camera visibility info in each node.

- Per camera visibility/paging/localities determination is costly, but doing it several times per camera (shadows/light/reflection/clipped) is making it even more complicated (and CPU costly).

=> Idea : Perhaps a "camera region" concept, using all light/shadow/reflection/clipped information "regrouped" could be computed at first, then determining "locality" of a camera only once per frame. (in PLSM it's doing it several times per camera, not that costly, but I don't use portals/connector, only sort of a quadtree)
The other thing will be a redesigned event system.
Eagerly waiting for that one.
I had to overuse "getOption/setOption" in order to let user call my SM methods, register event listener (onpagepreload, on pageunload, ...), get Information on current SM state. All that using "Option" semantic is very disturbing for users.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

All this reading & thinking is making my head hurt ;)

I have one part of the design written up and ready for comments, this is just the part on world updates - ie removal of SceneNode being mandatory and how the event system could be used to multithread world updates in various ways. It's relatively high-level since I only went down as far as I had to to illustrate the concepts I was trying to get across. That and to finally get something down on an organised bit of paper so I can stop arguing with myself about it ;)

World Updates Design

More design docs will of course follow on the many other related areas as I feel I've bottomed them out. It's taking a long time since I keep stopping to read articles, second-guess myself and such. :? I'm pretty happy with this bit now (even if I didn't really have time to go into huger detail) and was keen to get it solidified first since everything else is dependent on it. I feel this will be a longish process!
User avatar
raicuandi
Gargoyle
Posts: 1092
Joined: Wed Nov 09, 2005 12:56 pm
Location: Adelaide, Australia
Contact:

Post by raicuandi »

After reading half of your paper, I've had an "ahaaa!". I would have otherwise kept 2 bytes (odd_frame) for each "thing" in the hierarchy (like a scene node), to check against the current frame's odd_frame. If they were not equal, I'd catch the result, and make the "thing" 's odd_frame to the value of the frame's. Then if I go again trough that thing, being in the same frame, I see that its "odd_frame" value (an unsigned short if I remember) equals to the "master" frame's odd_frame, so I'd just use the cached result.
Next frame, I'd just increment the frame's odd_frame value, forcing the "things" to update on the next request.

Heh, events & event queues are so much cooler :roll: :)
I should PAY you for all you teach me (without knowing) :lol:
User avatar
tau
Silver Sponsor
Silver Sponsor
Posts: 413
Joined: Wed Feb 11, 2004 11:44 am
Location: Austin (that's kept wierd :))

Post by tau »

Excellent job, Sinbad!

Now, I really want to freeze my projects and wait for Ogre3D v2.0 :), especially when it will be true View component in MVC and be very thread friendly! 8)

My questions (or ideas to think about):
1) Will the TransformSource be an interface-alike abstract class, so I can provide my own SceneNode implementations by implementing the abstract methods, for instance creating a true RTTI implementation?
2) Event system: will it provide an abstract interface, so it's possible to implement different connectors/queues: unix sockets, name pipes, STL queues, etc?
3)Event system: do we need event sorting and priorities to be sure that some events will occur before others?
User avatar
raicuandi
Gargoyle
Posts: 1092
Joined: Wed Nov 09, 2005 12:56 pm
Location: Adelaide, Australia
Contact:

Post by raicuandi »

Heh, just keep adding to the pile of ToDo's, Sinbad will do 'em all so who cares :)
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

tau wrote: 1) Will the TransformSource be an interface-alike abstract class, so I can provide my own SceneNode implementations by implementing the abstract methods, for instance creating a true RTTI implementation?
Yes, that's the point of it. You won't need to implement a SceneNode, just a TransformSource (which may not be hierarchical).
2) Event system: will it provide an abstract interface, so it's possible to implement different connectors/queues: unix sockets, name pipes, STL queues, etc?
No. It's to be a lock-free queue functioning on a callback basis only, most likely using boost::bind. It's necessary to be clear about usage to get lock-free thread safety. Bosst::bind function binding should be flexible enough for anyone.
3)Event system: do we need event sorting and priorities to be sure that some events will occur before others?
I intend to include priorities on listeners yes, with published priority values for the Ogre event processors so you can choose when your listener gets called in relation to others. Event types will have a category, which determines at which stage they are triggered to listeners (e.g. transform events are processed just before rendering). I intend to make it fairly pluggable so that you can use it to build your own event queues separately if you want independent of the event sequences Ogre needs.

No to sorting of events - events themselves will be processed on a FIFO basis. There will be double-buffering of each queue built in (to allow events to be raised in other threads whilst the current queue is processed).
User avatar
Falagard
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2060
Joined: Thu Feb 26, 2004 12:11 am
Location: Toronto, Canada
x 3
Contact:

Post by Falagard »

Will a SceneManager still have a standard way of creating a *something* (TransformSource?) that Renderables are associated with?

I ask because my engine sits on top of any type of SceneManager with the current Ogre design and is able to call createSceneNode knowing that the SceneManager may have a custom implementation of SceneNode but not caring. Also I know that I can get the SceneNode's position, orientation and scale in a standard way.

I don't want to limit your design in any way, just curious as to how this is going to work and possibly make some design decisions of my own.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

There will be a pluggable factory for TransformSource yes, but I didn't intend for it to be owned by the SceneManager - it would be global (on the Universe Manager which I'm now referring to as WorldManager in my notes). Since TransformSource's only purpose is to provide a view of the scene 'Model' and it's no longer specific to a SceneManager implementation like SceneNode is.

[edit]Actually I might put it on the ObjectRegistry where other scene instances are created. Either way, it will be on something of which there is only one instance of.[/edit]

To be visible, MovableObjects will have to be associated with a TransformSource. Renderables will have the same getWorldTransform kind of method which they'll delegate to a parent MovableObject, which will delegate to a TransformSource, which will provide the cached matrix. Renderables which are not associated with MovableObject will get their transform from elsewhere, but these are likely to be world geometry fragments and thus probably identity.
Last edited by sinbad on Thu Apr 05, 2007 7:01 pm, edited 1 time in total.
User avatar
Falagard
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2060
Joined: Thu Feb 26, 2004 12:11 am
Location: Toronto, Canada
x 3
Contact:

Post by Falagard »

There will be a pluggable factory for TransformSource yes, but I didn't intend for it to be owned by the SceneManager - it would be global. Since TransformSource's only purpose is to provide a view of the scene 'Model' and it's no longer specific to a SceneManager implementation like SceneNode is.
Sounds perfect.

[edit]
Actually, I'm not sure now, lol. Since the TransformSource is pretty much just the View of MVC, I'd need to update the Model to update positions and orientations. Is there a standard 'Model' interface?

I assume there'll be a sample SceneNode model implementation that you'll provide that pretty much matches the current SceneNode implementation, and that others can do something completely different if they wish. Still, there has to be *some* sort of standard way of setting positions and orientations right? Am I missing some part of the design?
[/edit]
Post Reply