Bug #600

Free up memory of unused assets

Added by ionosys (Forum User) about 4 years ago. Updated over 1 year ago.

Status:AcceptedStart date:10/23/2014
Priority:NormalDue date:
Assignee:Joel Palmius% Done:

0%

Category:Code correctness
Target version:MakeHuman 1.1.2

Description

Hi,

I don't know if it's a makehuman or a bug in Python 2.7.8, after a while Python consumes a lot of memory.
So i post it here, sorry ^^.

What i do till python and makehuman is "crashing".

I prepared some figure which i want later import into Blender to create a small game in Unity3D.
These i'm currently exporting from makehuman to DAE Files, to could create shapekeys in Blender.

If I load the mhm file, makehuman "reserves" memory for this figure, I export the figure and load the next figure,
but it will not free'd the memory before, which it has reserved for the previous figure.
If Python reaches 2,1 GB it crashes, in worst case i got a Bluescreen with a kernel page memory fault error.

kind regards

memoryconsumption.png (1.96 KB) ionosys (Forum User), 10/23/2014 07:01 PM

example.png (54.1 KB) ionosys (Forum User), 10/24/2014 09:02 AM

wFemale-African.7z (1.35 MB) ionosys (Forum User), 10/24/2014 09:17 AM

young_d_african_female.7z (420 KB) ionosys (Forum User), 10/24/2014 09:17 AM

484
486

History

#1 Updated by Jonas Hauquier about 4 years ago

  • Status changed from New to Accepted

Thank you for reporting this. I have taken some measures a while ago to try and release memory that is no longer in use, but I never formally verified this. This will need a closer inspection.

#2 Updated by ionosys (Forum User) about 4 years ago

Screenshot added

#3 Updated by Jonas Hauquier about 4 years ago

I know textures are kept in memory, but they are reused and not loaded again if models reuse the same texture.
That means that if you have many models with all completely new and unique textures and you load them consecutively, you might run out of memory after a while.

But if the memory consumption is caused by geometry remaining in memory (even though I tried to avoid that by using weakrefs) then this is something to fix.

Perhaps we might try to figure something out to unload textures from cache if they were not used for some time, if that would prove to be useful.

#4 Updated by ionosys (Forum User) about 4 years ago

Hi,

i'm using max 19 Textures (for the skin), the 2 which are already in MH and 16 derivates of them, 1 is currently unused (will be a derivate too, but with a navel).
Per race I'm using 32 config files (mhmat, the normal one and the sweat one like in the example.png, and attached in the "young_d_african_female", without textures).

The other textures are all the default one (hairs i'm currently didn't using), but the other geometries like teeth, tongue, and so on.

I have added one of my batches, if i reach the half of the batch, the memory consumption already increased up to 1,5 GB.

The Batches are splitted in one race each batch, with the same texture for every figure, so i assume that in this case the geometry is the problem, they are all "the same" but with different slidersettings, in that way, that i could create shapekeys in Blender.

I have attached an example of one of my batches (wFemale-African).
If you want i could upload the textures too, but they consumes 87 MB.

kind regards

#5 Updated by Jonas Hauquier about 4 years ago

ionosys (Forum User) wrote:

If you want i could upload the textures too, but they consumes 87 MB.

No need, thanks. This should already help more than enough to reproduce it ;)

#6 Updated by Jonas Hauquier almost 4 years ago

  • Subject changed from Bug? Makehuman or Python reserves memory for every loaded mhm file and if its reaches 2,1 GB it crashes. to Free up memory of unused assets
  • Assignee set to Jonas Hauquier

#7 Updated by Aranuvir # over 2 years ago

  • Target version set to MakeHuman 1.1.1

#8 Updated by Rob Baer about 2 years ago

  • Target version changed from MakeHuman 1.1.1 to MakeHuman 1.2.0

I did some looking at memory as I loaded every piece of clothing I have. Memory did rise from 0.35 to about 2.1 GB for python which registered 24 threads on Win 10. Neither removing clothes or hitting reset reduced the memory usage significantly.

It seems their might be issues as suggested by Jonas above but this seems more like a refactor than a bug fix. Changing to 1.2.0.

#9 Updated by Joel Palmius over 1 year ago

  • Category changed from Unsorted to Code correctness
  • Assignee changed from Jonas Hauquier to Joel Palmius
  • Target version changed from MakeHuman 1.2.0 to MakeHuman 1.1.2

From a mail I got:

Here a bug (windows 10, MH 1.1.1), I have some glitchs ... so I tried to use the updateSubdivisionMesh() in guicommon.py to correct it but this method does not free memory as expected ( self.__subdivisionMesh = self.__proxySubdivisionMesh = None). It takes more and more memory (1.5Go for 15 images rendered). If I use

self.setSubdivided(False)
#self.__subdivisionMesh = self.__proxySubdivisionMesh = None
self.setSubdivided(True)

no more memory problem and the subdivided Mesh is correctly updated, and my images too (probably a misfunction of the python garbage collector).

I'll take a look at this and see if it also has an impact on the overall memory consuption over time.

#10 Updated by Aranuvir # over 1 year ago

Isn't this equivalent to hitting the subdiv button twice? If this is true, I cannot see a great effect on memory usage. The easiest way to reproduce the problem was walking up and down the topologies. Perhaps we could use the garbage collector interface (https://docs.python.org/2/library/gc.html) to debug the issue?

#11 Updated by Aranuvir # over 1 year ago

Investigating on this issue, I think I was able to identify one problem: in 3_libraries_proxy_chooser.py selected topologies are stored in the list self.selectedProxies. On change of topologies, the list gets longer and longer, deselected topologies aren't removed. IMHO, storing topologies in self.selectedProxies is superfluous, since MH allows only one topology at the same time. Commenting this line out saved me a bunch of memory in the long run.

Also available in: Atom