Bug #934

Collada does not import textures

Added by Aranuvir # over 3 years ago. Updated almost 2 years ago.

Status:Fix exists, needs testingStart date:07/19/2015
Priority:NormalDue date:
Assignee:Joel Palmius% Done:

90%

Category:File export
Target version:MakeHuman 1.1.2

Description

When exporting a file via collada there is created a subfolder for the textures. On the other side, when importing to Blender the textures are expected to be in the same hierarchy level as the .dae-file.

Tested with MakeHuman r1958. Blender 2.75a. OS is Kubuntu 15.04.

See also here: http://www.makehumancommunity.org/forum/viewtopic.php?f=6&t=12040

screenshot_1_1437354108.png (247 KB) Rob Baer, 07/20/2015 03:03 AM

screenshot_1_1472687762.png (125 KB) Rob Baer, 09/01/2016 01:56 AM

issue934.patch Magnifier (1.15 KB) Aranuvir #, 09/01/2016 09:10 PM

fix_934 (1.47 KB) Aranuvir #, 03/13/2017 06:21 PM

fix_934.patch Magnifier (1.47 KB) Aranuvir #, 04/04/2017 05:09 PM

1075
1269

Related issues

Related to MakeHuman - Bug #1148: Test/fix collada pipeline vs various external platforms New 03/12/2017

Associated revisions

History

#1 Updated by Rob Baer over 3 years ago

Windows 8.1 r1958 Blender 2.74 (3-31-2015)

I am not reproducing the problem. Exporting does indeed put textures in a textures subfolder, but Blender importer finds them there just fine and attaches them appropriately to the corresponding materials (at least if the renderer is set to BI). The only thing remaining to do is fixing shadows and transparency for eyes, eyebrows, eyelashes, and hair.

I am in the process writing a Wiki entry with details on how to accomplish this that I hope to soon post to the Wiki. Of course, this nuisance work can be avoided in both BI AND Cycles by using MHX2 if Blender is your only downstream target.

The following image was exported as .dae and imported into Blender as collada with no special texture treatment. Note the presence of textures in the properties pane of Blender and their presence upon render:

#2 Updated by Aranuvir # over 3 years ago

Hello Rob,

the problem might be os specific. Opening the dae-file and editing each "<init_from>file:// " line to "<init_from>file:/ " or "<init_from>file:///" solved the problem to me. Unfortunately I wasn't able to figure out the correct uri syntax for relative file paths.

BTW, of course I know about MHX2, my preferred way for the blender pipeline. This was just a report from my test of MakeHuman's recommended way to export to blender, that didn't work, at least, on my linux system.

#3 Updated by Rob Baer over 3 years ago

It still seems to me that MakeHuman is following the collada spec, so we need to understand more about what is going wrong for you.

Here is a short texture snippet from one of my .dae test export files:

=========================================
  <library_images>
    <image id="middleage_lightskinned_male_diffuse2_png" name="middleage_lightskinned_male_diffuse2_png">
      <init_from>file://textures\middleage_lightskinned_male_diffuse2.png</init_from>
    </image>
    <image id="short04_diffuse_png" name="short04_diffuse_png">
      <init_from>file://textures\short04_diffuse.png</init_from>
    </image>
    <image id="brown02_eye_png" name="brown02_eye_png">
      <init_from>file://textures\brown02_eye.png</init_from>
    </image>
    <image id="eyebrow012_png" name="eyebrow012_png">
      <init_from>file://textures\eyebrow012.png</init_from>
    </image>
    <image id="eyelashes01_png" name="eyelashes01_png">
      <init_from>file://textures\eyelashes01.png</init_from>
    </image>
    <image id="teeth_png" name="teeth_png">
      <init_from>file://textures\teeth.png</init_from>
    </image>
    <image id="tongue01_diffuse_png" name="tongue01_diffuse_png">
      <init_from>file://textures\tongue01_diffuse.png</init_from>
    </image>
    <image id="male_worksuit01_diffuse_png" name="male_worksuit01_diffuse_png">
      <init_from>file://textures\male_worksuit01_diffuse.png</init_from>
    </image>
    <image id="male_worksuit01_normal_png" name="male_worksuit01_normal_png">
      <init_from>file://textures\male_worksuit01_normal.png</init_from>
    </image>
------------------------------

Except for the Widows-esque use of backslash, I'd expect yours to look similar.

The collada spec says (see https://www.khronos.org/collada/wiki/Using_URIs_in_COLLADA ):

Base URIs in a COLLADA document
To resolve relative references, a base URI is needed. In COLLADA, the base URI can be specified in the root <COLLADA> element:
<COLLADA xmlns="http://www.collada.org/2005/11/COLLADASchema" version="1.4.1"
xml:base="file:///home/sthomas/models/duck.dae">
If the xml:base attribute isn't specified, then the base URI is the document's URI according to the rules defined in RFC 3986 section 5.1. For example, if the document's URI is file:///car.dae, then the URI parsing logic in your software uses that as the base URI to resolve any relative references in the document.
It's important to note that a relative reference cannot indicate a change to the scheme part of the base URI.

It looks to me like there is no base URI specified in MH collada files, so that
<init_from>file://textures\tongue01_diffuse.png</init_from>
would refer to a .png image in the textures subfolder of whatever folder the parent .dae is located within. This seems to work on Windows 7, Windows 8, and Winfows 8.1.

Can you identify something that might be different on Kubuntu?

#4 Updated by Aranuvir # over 3 years ago

Hello Rob,

the part of the dae-file you posted looks mostly the same like my exported file, despite the "\", as you mentioned.
Looking at the collada wiki from your link above, thanks for that, I didn't find it before, and especially at this section https://www.khronos.org/collada/wiki/Using_URIs_in_COLLADA#Native_Paths_versus_URIs, it leads me to some questions:

1) A Windows-URI shouldn't contain a "\", should it? None of the given examples in the table show a "\" in a Windows-URI.

2) Every URI example containing "file" starts like "file:///" and not "file://".

3) Referencing a relative path, the "file:" is omitted. I thought, I read this somewhere.

I'm not really sure about these statements, please tell me if I'm wrong.

#5 Updated by Rob Baer over 3 years ago

Aranuvir writes:

1) A Windows-URI shouldn't contain a "\", should it? None of the given examples in the table show a "\" in a Windows-URI.

That is probably true, but Windows does NOT have an issue with it. It handles MakeHuman Collada files just fine. This seems in line with Postel's robustness principle ("an implementation must be conservative in its sending behavior, and liberal in its receiving behavior" ). It is likely that the appearance in MakeHuman files comes from the OS specific pyQT library used. It does not arise from MakeHuman code and does not have ill consequences because it only appears in a context where it is handled. I suppose it COULD create issues if files were created on Windows and used on Linux or OSX. Jonas would be able to provide more detail on where it arises in the code and how much work it would be to avoid.

Aranuvir writes:

2) Every URI example containing "file" starts like "file:///" and not "file://".

As I understand it, file:// is the designation of the schema. If there is a third slash it specifies the root. file:// would be interpreted as a path relative to the current directory/folder when an xml:base attribute is not supplied (i.e., as in MakeHuman collada files). In collada, this is the directory that the .dae file is located. However, I am far from an expert in this area. You should be able to move the .dae file and the texture folder to any directory on the computer and Blender should be able to find the textures it needs as long as the .dae and textures folder exist within a common parent folder. This certainly works on Windows. No editing of the .dae necessary.

So specifically, did you move a .dae file along with its texture subfolder from the MakeHuman exports directory to another directory and have it fail to open in Kubuntu, or was it an edited collada file that gave you problems?

With the exception of me, most of the other development team members use some flavor of a Linux-based OS and have not reported any problems. We need to know more detail to see what might be going wrong for you.

It is true that all collada files you create will dump their textures into the same textures subfolder so that this folder will grow in size if you create many models with a wide variety of assets. If you are going to move it, you are well-advised to create a new textures folder in the directory to which you move the .dae and to place just the textures you need in it. Does this not work on Kubuntu 15.04?

Aranuvir writes:

3) Referencing a relative path, the "file:" is omitted. I thought, I read this somewhere.

I am not familiar with anything like this for URIs. Of course, relative paths in some shells sort of work like this, but that's a different matter. The file:// is necessary to specify the URI schema.

#6 Updated by Aranuvir # over 2 years ago

  • Target version set to MakeHuman 1.1.0

Still having trouble with textures on dae export. Although I switched to Debian testing. Blender is now 2.77a and MH is 2073 (stable).

Not sure about this problem, but I dare to set it to 1.1.1. Any Linux user who can reproduce this problem?

#7 Updated by Aranuvir # over 2 years ago

  • Target version changed from MakeHuman 1.1.0 to MakeHuman 1.1.1

#8 Updated by Rob Baer over 2 years ago

Because of shader differences between MH and Blender (lightsphere vs phong) and without the automagic of mhx or mhx2 some hand work needs done. Without reading too deeply into this I would refer you to a Wiki piece I put together. See if it applies on Linux side or needs amending?

http://www.makehumancommunity.org/wiki/Moving_Assets_into_Blender

Again, there were some things about path above that we discussed that maybe I'm ignoring so we can continue a discussion if there is a different issue. I have gotten good transfer with both collada and FBX to Blender, but there are shader corrections that make the job SO much harder than using .mhx2. ;-(

#9 Updated by Aranuvir # over 2 years ago

Rob Baer wrote:

Because of shader differences between MH and Blender (lightsphere vs phong) and without the automagic of mhx or mhx2 some hand work needs done. Without reading too deeply into this I would refer you to a Wiki piece I put together. See if it applies on Linux side or needs amending?

http://www.makehumancommunity.org/wiki/Moving_Assets_into_Blender

I know that materials are way from perfect on collada import, but looking at your documentation there seem to be some textures after basic import with collada, in contrast, when I import a model and switch "viewport shading" to material the model stays white as the model on the left side of your docs (figure 1). None of the textures appear under the textures tab.

Here is what Blender is telling me for a full equipped test model:

Image not found: /young_lightskinned_male_diffuse.png.
Image not found: /short04_diffuse.png.
Image not found: /brown_eye.png.
Image not found: /eyebrow001.png.
Image not found: /eyelashes01.png.
Image not found: /teeth.png.
Image not found: /tongue01_diffuse.png.
Image not found: /shoes04_diffuse.png.
Image not found: /shoes04_normal.png.
Image not found: /male_worksuit01_diffuse.png.
Image not found: /male_worksuit01_normal.png.
Couldn't find an image by UID.
Couldn't find an image by UID.
Couldn't find an image by UID.
Couldn't find an image by UID.
Couldn't find an image by UID.
Couldn't find an image by UID.
Couldn't find an image by UID.
Couldn't find an image by UID.
Couldn't find an image by UID.
Collada: Adjusting Blender units to Importset units: 0.100000.
Writing node id='test01-baseObject', name='test01-base'
Writing node id='test01-short04Object', name='test01-short04'
Writing node id='test01-highpolyeyesObject', name='test01-highpolyeyes'
Writing node id='test01-eyebrow001Object', name='test01-eyebrow001'
Writing node id='test01-eyelashes01Object', name='test01-eyelashes01'
Writing node id='test01-teeth_baseObject', name='test01-teeth_base'
Writing node id='test01-tongue01Object', name='test01-tongue01'
Writing node id='test01-shoes04Object', name='test01-shoes04'
Writing node id='test01-male_worksuit01Object', name='test01-male_worksuit01'

There seem to be other users having the same issue: http://www.makehumancommunity.org/forum/viewtopic.php?f=3&t=13541

#10 Updated by Aranuvir # over 2 years ago

Walking through the bug tracker, I'm not sure if this is a problem of the importer... (=Blender).

#11 Updated by Rob Baer over 2 years ago

OK. spent some time on this: Collada export from MH and import into Blender 2.77a on Windows 10 works fine (with shader fixes)

Importing the same collada into Blender 2.76 in Ubuntu 16.04 is broken with the textures not importing properly despite the fact that they are properly in the textures folder. Finally, opening the Blend file from the Windows imported collada file renders fine in Blender 2.76 on Ubuntu 16.04.

Conclusion, this is a collada import problem for Blender ON THE UBUNTU platform (at least version 2.76) [I did not get around to installing 2.77a] This seems to be a problem with properly importing the texture path. I suppose there is an outside chance that it could be a relative path / absolute path issue that doesn't translate as well on the Linux side but this seems far-fetched.

Worthy of a technical note somewhere and perhaps a bug report to Blender, but not an MH issue I think.

#12 Updated by Aranuvir # over 2 years ago

Since a have the issue on Debian with Blender 2.77a I'd state and collada issue on all Debian or even Linux-platforms. Maybe the problem is within the xml parser, see last paragraph on the collada docs you sent me (https://www.khronos.org/collada/wiki/Using_URIs_in_COLLADA#Native_Paths_versus_URIs).

Would you mind to do a last test and edit a dae-file by removing all "file://" under the <library_images> tag, to see if it still imports flawlessly on Windows?

#13 Updated by Rob Baer over 2 years ago

Well here is at least one problem with importing a .dae produced in Windows into a linux OS. The image lines are:
<library_images>
<image id="middleage_lightskinned_male_diffuse2_png" name="middleage_lightskinned_male_diffuse2_png">
<init_from>file://textures\middleage_lightskinned_male_diffuse2.png</init_from>
</image>
<image id="short04_diffuse_png" name="short04_diffuse_png">
<init_from>file://textures\short04_diffuse.png</init_from>
</image>
<image id="brown02_eye_png" name="brown02_eye_png">
<init_from>file://textures\brown02_eye.png</init_from>
</image>
<image id="eyebrow012_png" name="eyebrow012_png">
<init_from>file://textures\eyebrow012.png</init_from>
</image>
<image id="eyelashes01_png" name="eyelashes01_png">
<init_from>file://textures\eyelashes01.png</init_from>
</image>
<image id="teeth_png" name="teeth_png">
<init_from>file://textures\teeth.png</init_from>
</image>
<image id="tongue01_diffuse_png" name="tongue01_diffuse_png">
<init_from>file://textures\tongue01_diffuse.png</init_from>
</image>
<image id="male_worksuit01_diffuse_png" name="male_worksuit01_diffuse_png">
<init_from>file://textures\male_worksuit01_diffuse.png</init_from>
</image>
<image id="male_worksuit01_normal_png" name="male_worksuit01_normal_png">
<init_from>file://textures\male_worksuit01_normal.png</init_from>
</image>
</library_images>

There is code to replace backslashes with forward slashes in MH and apparently it hasn't been called for some reason. Backslashes will never work on othr OSes, but forward slashes SHOULD work on Windows.

Let me check a .dae PRODUCED on the linux side and see what happens. Depending on the results I look at whether the URI descriptor file:// is essential as well. Get back to you.

#14 Updated by Rob Baer over 2 years ago

Okay. If we remove file:// it imports into (at least Blender) fine on both Windows and Linux. I haven't tested other things like Max, Maya, UE, etc.

HOWEVER, it will be also necessary to fix the backslahes written by the exporter when it is used for export from the Windows side. I have not looked, but I do presume it is already using forward slashes during export on the Linux side.

Question:
Paths like: textures/male_worksuit01_normal.png work on Windows, but so do paths like ./textures/male_worksuit01_normal.png I see the latter type of construction used more frequently in the Linux world. Is there any reason to favor one or the other for some OSes? How about OSX?

Code targets for fix:
  • Lines 75-85 of plugins/9_export_collada/dae_materials.py
  • and fix, copyTextureToNewLocation() to use forward slash folder separator on all OSes core/export.py line 152

#15 Updated by Aranuvir # over 2 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 40

Great, thanks for testing :).
Indeed, you're right about the "./" . This is common Unix style, so should be valid for OSX, too. You use it to reference a local file/folder, e.g. if you want to execute the file foobar in the current folder you use ./foobar. If you just write foobar it will search for it in the folders defined by the environment variable PATH and show an error message when it can't be found, even in case the file exists locally.
I omitted the "./* for Windows first, because I wasn't sure if Windows starts to cough on it. So let's take ./texture/foobar.png. Well, many thanks again for testing.
The changes are minor on the files you pointed out. To remove '\', I propose to use formatPath() from getpath.py. Just would like to hear the opinion of Jonas or Joel.

#16 Updated by Aranuvir # over 2 years ago

  • File issue934.patchMagnifier added
  • Status changed from In Progress to Fix exists, needs testing
  • % Done changed from 40 to 90

Providing a patch created with TortoiseHg, hopefully working better than my last ones created with diff. Should format path from \ to / and replaced file:// with ./

If someone could test? (currently I'm on my old netbook again and therefore can't run blender).

#17 Updated by Aranuvir # over 2 years ago

Anyone who already checked this? Any suggestions?

#18 Updated by Joel Palmius about 2 years ago

  • Assignee changed from Jonas Hauquier to Joel Palmius

Did anything happen with this?

#19 Updated by Aranuvir # about 2 years ago

There was I fix that worked for me, thought it needed further testing on other platforms. But the fix was rejected ...

#20 Updated by Joel Palmius about 2 years ago

Can you make a pull request for this against stable?

#22 Updated by Joel Palmius about 2 years ago

Hm, no.

As most of the referenced pull request was merged already in #771, the one you reference is partially superfluous.

Is the problem fixable near term, or should we defer this to 1.1.2?

#23 Updated by Aranuvir # about 2 years ago

#934 and #771 are independent problems, but were both fixed in one pullrequest I had pointed to.
Actually we have three options:
1) Use my quick and dirty approach for a minor bug fix release and opening a new bug report about reworking the exporter library thing later
2) Making changes to the exporter library now, which could lead to problems in all export plug-ins, though it is not very likely
3) Move the whole thing to a later release

Since the issue seems to be limited to Linux, i.e. about 1% of the users, and affects only the collada export, so, combined odds probably <<<< 1% of the users, I'd propose solution 3.

BTW, we could need a platform survey!

#24 Updated by Joel Palmius about 2 years ago

  • Target version changed from MakeHuman 1.1.1 to MakeHuman 1.1.2

I'll simply trust you. It's easier than to think on my own. :-)

#25 Updated by Joel Palmius almost 2 years ago

  • Related to Bug #1148: Test/fix collada pipeline vs various external platforms added

#26 Updated by Aranuvir # almost 2 years ago

Providing a patch to fix the issue, also taking in account one of Jonas objections. Probably there where more but Jonas was not very verbose...

Thorough testing with all other exporters available should be performed.

#27 Updated by Jonas Hauquier almost 2 years ago

I think the patch got lost somewhere. Can we recover it?

#28 Updated by Aranuvir # almost 2 years ago

Recover patch ...

Also available in: Atom