Memory Usage

I originally PM'd Tuan this but I'm posting it here because his answers are interesting.

Tuan,

If I want to do a fairly large terrain, let's say 30km x 30km, with up to 5km viewable at a time, at a resolution of a vertex every meter, I'm worried about memory usage. Each page keeps height data in memory, and each tile loads the full resolution of the tile even if it's very far in the distance, right?

Same goes with the materials assigned to the pages. In the far distance I'll only want to display a lesser material LOD, however I *think* the full coverage + splatting textures are loaded into memory for all pages.

Another question - I think you once said that pages are never actually unloaded (all the height data and vertex buffers in the tiles), just no longer rendered. Is that the case?

I'm in the middle of looking into some form of mesh based terrain instead of plm2 heightmapped terrain because I'm concerned about two things - memory usage, and holes in terrain. I won't get into the specifics of what I'm thinking about for mesh based terrain, but the truth is I don't want to do it if I can solve the memory problems using the PLM2.

What I really want is for distant pages not to have their vertex buffers populated with the full resolution data, nor to keep the full resolution height data loaded in memory, until it's absolutely necessary to do so.

Thanks,

Clay

This is Tuan's answer - I assume he doesn't mind me posting the reply here.


If I want to do a fairly large terrain, let's say 30km x 30km, with up to 5km viewable at a time, at a resolution of a vertex every meter, I'm worried about memory usage. Each page keeps height data in memory, and each tile loads the full resolution of the tile even if it's very far in the distance, right?


Right.


Same goes with the materials assigned to the pages. In the far distance I'll only want to display a lesser material LOD, however I *think* the full coverage + splatting textures are loaded into memory for all pages.


Have to check with Ogre internals or ask sinbad about that.

If it is so, that would'nt be too hard to change actual material LOD plsm2 code to do the load and unload of different material, and could even use a listener to permit user to do whatever he choose.


Another question - I think you once said that pages are never actually unloaded (all the height data and vertex buffers in the tiles), just no longer rendered. Is that the case?


If I said that, I wongly express what I intended to say. It is indeed unloaded, but not immediatly, using a sort of timer, which "time" is configurable . Material are also manually unloaded. Timer is to avoid problem when on a edge of load/unload which was giving very bad pfs results.


I'm in the middle of looking into some form of mesh based terrain instead of plm2 heightmapped terrain because I'm concerned about two things - memory usage, and holes in terrain. I won't get into the specifics of what I'm thinking about for mesh based terrain, but the truth is I don't want to do it if I can solve the memory problems using the PLM2.


About mesh and plsm2, you can't beat the plsm2 vertex compression thing using mesh. it really gives a boost on mem saving.

- instead of 3 float for position, only one.
- instead of a texcoord buffer if 2 floats per page, a shared single texcoord buffer for all pages.

If you have a lot of page, you'll end up with only adding nearly one float per vertex, against 5 floats in the mesh case.
That's a 1/5 ratio...

About memory inside PLSM2, we can change the in RAM data to be 16 bits instead of Real (32 - 64 bits) if you think it is worth, or provide this as an option, scaling it when needed.


What I really want is for distant pages not to have their vertex buffers populated with the full resolution data, nor to keep the full resolution height data loaded in memory, until it's absolutely necessary to do so.


I did some code on that (try BigImage=yes with a mapsplitted map that produced a "small map" and you'll see the beginning) and am thinking about that regularly, still looking to find the best way to do that in a "recursive way".
Meaning a Page could be a tile of a bigger Page that could be a tile of a bigger map, etc.
(bigger map meaning actual bigger in 3D, but smaller in term of memory.)

Glad you posted here.

still looking to find the best way to do that in a "recursive way".
Meaning a Page could be a tile of a bigger Page that could be a tile of a bigger map, etc.
(bigger map meaning actual bigger in 3D, but smaller in term of memory.)

If someone comes with some nice design or ideas, I'd be glad to hear about.

Keep in mind that we have to find a way to make lower "Page" Lod stictch compatible with higher "Tile" Lod Stitch. (or find another way to prevents cracks.)

About mesh and plsm2, you can't beat the plsm2 vertex compression thing using mesh. it really gives a boost on mem saving.

- instead of 3 float for position, only one.
- instead of a texcoord buffer if 2 floats per page, a shared single texcoord buffer for all pages.

If you have a lot of page, you'll end up with only adding nearly one float per vertex, against 5 floats in the mesh case.
That's a 1/5 ratio...

About memory inside PLSM2, we can change the in RAM data to be 16 bits instead of Real (32 - 64 bits) if you think it is worth, or provide this as an option, scaling it when needed.


I can beat your memory usage if I'm not loading the memory until needed. It's true that you'll have less data because you have 1 float instead of 3, and shared texture coordinates, however if you take a look at this image:



This is how I'm planning on breaking up the meshes, where each progressively larger mesh far less vertices. Is this what you were looking at for BigImage functionality?

If I used PLM2 for this scene, the whole terrain would be made of the smallest sized pages seen in that image, and each page would contain the full resolution data (a vertex every meter, for example) which is a huge amount of memory. Each of those pages would store the complete height data, vertex buffer (a second copy of the height data basically) and index buffers. Let's say in my case that the smallest square has a vertex per meter, the second has a vertex every 2 meters, the next every 4, and the next every 16, I'm using far less memory than you are, even with compression, just because I'm not loading unnecessary stuff.

You're probably wondering about lod switching with meshes, and rightly so. A tile mesh will always have vertices along its boundary that allow it to switch to the next higher lod. I can't support vertex morphing with mesh based approach, (or perhaps I could with something similar to Sinbad's face morphing functionality, but I doubt I'd support it). However, the bonus of using meshes is that I can add more detail to the terrain wherever I need it, and punch holes in the terrain and include overhangs. Of all of those things, only cutting holes is the real problem with PLM2, since overhangs and extra detail can usually be added fairly well with externally placed meshes.

The other issue I have with the PLM2 is the max pixel error functionality can often make distant pages too highly tesselated even though you don't want it to be.

s this what you were looking at for BigImage functionality?
yes, that's the idea result. Using this + compression would result in a very small memory usage, being able to draw very huge maps, very fast, from very distant places.

The other issue I have with the PLM2 is the max pixel error functionality can often make distant pages too highly tesselated even though you don't want it to be.
Someone intended to add multiple LOD scheme and finally didn't do. But I still think it's a good idea, and will soon do it, as it's a recurrent problem on forum. There would be at least "roughness+camera LOD" (current) and "Camera LOD" (only distance taken in account) Lod.

Of all of those things, only cutting holes is the real problem with PLM2, since overhangs and extra detail can usually be added fairly well with externally placed meshes.
Multiple scene manager would a better solution than meshes if cave is the Idea.... or you'll end up drawing very complex mesh even when not seeing inside the cabe.

Multiple scene manager would a better solution than meshes if cave is the Idea.... or you'll end up drawing very complex mesh even when not seeing inside the cabe.


GOOF is a pseudo scene manager on its own. Partitions will allow me to do this easily without needing multiple scene managers. So in my case, the only problem is finding a way to cut a hole in the terrain. To be honest, this isn't a huge problem for games that I'm creating at the moment, but I'd like to have a solution to the problem in case someone else asks for it (and they have).

A much higher priority in my case it the memory usage of large terrains, and if the BigImage functionality is actually implemented then it'll mean that PLM will be really viable for games that require large terrains like flight sims, open world RPGs, etc.

I'll have to take a look at the big image stuff you've done. Ideally, we'd break up the heightmap data, as well as things like coverage maps, etc. into progressively larger pieces, but also lower their resolution. So for example, a really distant page would also have a lower resolution base map. I can imagine implementing this as a post process that you run over your existing pages, not something that necessarily has to be edited in realtime by allowing the user to paint and deform the terrain. What I mean is, while editing the terrain, the artist can afford to have a beefed up machine with a gig or two of RAM. Once the artist is finished with the terrain, run a process over it to convert it into "big image" mode which splits the terrain up into progressively larger and lower resolution pages and textures.

without needing multiple scene managers
Specialized optimized scene manager is a viable path user might ask.

A much higher priority in my case it the memory usage of large terrains, and if the BigImage functionality is actually implemented then it'll mean that PLM will be really viable for games that require large terrains like flight sims, open world RPGs, etc.
Cannot Agree more.

I'll have to take a look at the big image stuff you've done
mmh, not really interesting tough, basically a page has its onw renderable, like tile does. nothing like recursity and so on in there.

Ideally, we'd break up the heightmap data, as well as things like coverage maps, etc. into progressively larger pieces, but also lower their resolution.
Exaclty. It's sort of rendering using mutliple PLSM2 at a same time, each one using a pagesize which is the tilesize of the bigger one.

not something that necessarily has to be edited in realtime by allowing the user to paint and deform the terrai
Not at first, but I like the idea of recursively edit the whole to end up in.