Bug #996

Evaluate storing assets on github instead of tuxfamily

Added by Joel Palmius almost 3 years ago. Updated about 2 years ago.

Status:NewStart date:02/27/2016
Priority:NormalDue date:
Assignee:Joel Palmius% Done:

0%

Category:Unsorted
Target version:MakeHuman 1.1.2

Description

Tuxfamily has limited space and no version control. Github now supports version conrol of large binaries via LFS.

Possibly it'd be convenient to place assets on github instead of tuxfamily.

In this case the download_assets.py script would be need to be modified.

Currently, the most up to date version of all assets is on github anyway, since I put stuff there to manage them while working with the headers (see #992)

download_assets_git.py Magnifier - Git version of download assets (8.25 KB) Joel Palmius, 02/27/2016 04:39 PM

screenshot_1_1457210251.png (17.9 KB) Rob Baer, 03/05/2016 09:37 PM

screenshot_1_1457290094.png (115 KB) Rob Baer, 03/06/2016 07:48 PM

1191
1195

Related issues

Related to MakeHuman - Feature #1010: download_assets_git should alert the user and fail when L... Fix exists, needs testing 04/05/2016

History

#1 Updated by Joel Palmius almost 3 years ago

Initial replacement for download_assets.py

It's only tested on linux yet, and some features of it remains to be implemented. However, I think it covers the basic functionality of the original download_assets.py

I'll play around with it a bit more before committing it to bitbucket.

#2 Updated by Anonymous almost 3 years ago

I think that's a good idea.
A revision system for assets is a very important feature.

#3 Updated by Jonas Hauquier almost 3 years ago

Indeed now that github has LFS.
We probably should create an official mh github account (if there is none yet) for it, and while we're at it make a read-only code clone as well.

#4 Updated by Joel Palmius almost 3 years ago

The "organization" github account I have is here: https://github.com/makehumancommunity

It might make sense to rename it to "makehuman" if we're going to use it for official stuff. But since I also plan to keep adding community and third-part repos there, maybe the "community" part of the name isn't such a disaster.

(It's perfectly possible to rename an organization as long as one is aware of the consequences: https://help.github.com/articles/renaming-an-organization/)

#5 Updated by Joel Palmius almost 3 years ago

I've added the new download script to the stable branch now.

Anyone using this will pretty quickly discover that the git LFS extension is required locally. Otherwise all png files are downloaded as a short text file only containing a LFS file reference.

#6 Updated by Jonas Hauquier almost 3 years ago

This should not go into stable branch.
Stable branch has been feature locked for months, it's not the time to introduce new few tested things there. Especially not since the details have not fully been worked out.

#7 Updated by Rob Baer almost 3 years ago

Joel Palmius wrote:

The "organization" github account I have is here: https://github.com/makehumancommunity

It might make sense to rename it to "makehuman" if we're going to use it for official stuff. But since I also plan to keep adding community and third-part repos there, maybe the "community" part of the name isn't such a disaster.

(It's perfectly possible to rename an organization as long as one is aware of the consequences: https://help.github.com/articles/renaming-an-organization/)

My understanding was that we recently went to great lengths to separate MakeHuman "the organization" from MakeHuman "the community" for reasons related to possible legal benefits and clarity about what we want to claim to own and license. As such, it seems that the more sensible idea is to keep the MakeHuman Community github and add a separate, "official github" for official assets. The latter could naturally take the "makehuman" name rather than renaming "makehumancommunity" to "makehuman".

Did I misunderstand something?

#8 Updated by Joel Palmius almost 3 years ago

Jonas Hauquier wrote:

This should not go into stable branch.
Stable branch has been feature locked for months, it's not the time to introduce new few tested things there. Especially not since the details have not fully been worked out.

I disagree. This is not a feature of the makehuman application. It is a switch of storage space provider and an update to the download routines for reflecting that change.

Whether or not the switch of storage provider is a good idea is another question though. But as it stands now, tuxfamily is behind on the updates.

My understanding was that we recently went to great lengths to separate MakeHuman "the organization" from MakeHuman "the community" for reasons related to possible legal benefits and clarity about what we want to claim to own and license. As such, it seems that the more sensible idea is to keep the MakeHuman Community github and add a separate, "official github" for official assets. The latter could naturally take the "makehuman" name rather than renaming "makehumancommunity" to "makehuman".

Did I misunderstand something?

No, not really. There is a minor fee for high traffic LFS storage ($5/month). Since I already paid that for makehumancommunity I put stuff there out of convenience, not as a statement of anything organizational. I just needed a repo for working with the assets.

After the header change, I guess it's possible to question if the original reason for the split remains. But I'm not advocating reversing it at this time.

#9 Updated by Anonymous almost 3 years ago

My only recomendation is to don't fragment the official repo.
In my opinion the options are:

1) Close the bitbucket account and move all on github (we have to avoid two official repos!)
2) Keep the bitbucket account for code and a github for assets (removing the assets from tuxfamily)

In both cases, even with the new headers, the community contributions can't be on the official repo.
In my personal opinion, the (1) is more clean and tidy.

I think these structural changes are not related to stable vs unstable, and can be done in any moment.

#10 Updated by Jonas Hauquier almost 3 years ago

Joel Palmius wrote:

But as it stands now, tuxfamily is behind on the updates.

In what sense? The assets haven't changed in months, as far as I can recall.
Also, I think official builds and their assets should be separated from community assets.

No, I do not think closing the bitbucket account is a good idea. For starters, nothing should be removed. Don't make the mistake of removing the official authoritative source.

#11 Updated by Joel Palmius almost 3 years ago

Jonas Hauquier wrote:

In what sense? The assets haven't changed in months, as far as I can recall.

Did you not see #992 and the mail conversations? There are plenty of headers in the asset files too.

#12 Updated by Jonas Hauquier almost 3 years ago

I see.
In any case, for stable builds we use archived assets, we can still do the same thing for 1.1

#13 Updated by Joel Palmius almost 3 years ago

Which moves the assets out of version control, and more or less forces a full stop on working on them.

How is it better to store a huge zip file on bitbucket rather than having a functional repo suitable for actually working with the files? The reason we did the zip thing to begin with was because we ran out of space on tuxfamily, not because it was something desirable in itself.

#14 Updated by Jonas Hauquier almost 3 years ago

Joel Palmius wrote:

Which moves the assets out of version control, and more or less forces a full stop on working on them.

That's exactly the idea.

A tagged release is supposed to be final, therefore its assets as well.

For 1.2 the files can be on a repo where they can be versioned and modified. The point is just that this comes at at time where for 1.1 this is probably not useful, or even desirable anymore.
I think we need a bit more time to experiment with this before using it to build a release. Do we already have procedures for using the new assets repo? Eg. we need a way to enforce that a tagged release that does not change anymore, is tied to a fixed set of assets that do not change anymore either.

#15 Updated by Joel Palmius almost 3 years ago

I think that's an absurd limitation. If something like http://www.makehumancommunity.org/forum/viewtopic.php?f=19&t=13103 pops up, we're not allowed to fix it until the next major release (historically a few years off) because the bug happens to be in an asset and not in the code?

How is fixing asset glitches less important than fixing code glitches?

I think the only sane way to address assets glitches is to fix them for the next patch release, in this case 1.1.1. And if we're working with the assets we want them under revision control, not stored as a big zip file.

#16 Updated by Jonas Hauquier almost 3 years ago

Joel Palmius wrote:

I think that's an absurd limitation. If something like http://www.makehumancommunity.org/forum/viewtopic.php?f=19&t=13103 pops up, we're not allowed to fix it until the next major release (historically a few years off) because the bug happens to be in an asset and not in the code?

We're allowed to fix it in the same way as we're allowed to fix it in code: in the next bugfix release (1.1.1)

Updates to assets need to follow the same policy as the code, they can not be changed more liberally. Once a version is finalized for release, its assets must be finalized as well.

With git we would be allowed to have even more frequent updates of the assets, but without specific policies and mechanisms in place to ensure that the right tagged version, and right branch of the assets are checked out with the right build, we will only be shooting ourselves in the foot.
eg. if the code is tagged (and thus final for that release), we need to be 100% sure that the assets that come with it are tagged in such a way that they will always be matched
This is important as we will not be the only ones building the code (eg debian packagers)

Currently, all the script does is a blind pull. There's no way to guarantee in the future that 1.1 builds will be 1.1 builds and not 1.1 code with 1.2 assets. This is a serious problem.
Also I think the build mechanisms, just like the code, will need some time to stabilize across all platforms (eg. the osx build machine currently has no git installed). There must be more testing.

Finally, we have not yet finished the discussion of where to host the official assets, so the only reasonable place for changes is the unstable branch.

#17 Updated by Joel Palmius almost 3 years ago

The script does not just do a "blind pull".

The defaults are set in the head of the script:

class DownloadAssetsGit:

    _git_official_assets_repo = "https://github.com/makehumancommunity/official_assets_11x.git" 
    _git_official_assets_branch = "master" 

The defaults can be overridden with an environment variable:

    def readOverridesFromEnvironment(self):

        if 'GIT_OFFICIAL_ASSETS_REPO' in os.environ:
            self._git_official_assets_repo = os.environ['GIT_OFFICIAL_ASSETS_REPO']

        if 'GIT_OFFICIAL_ASSETS_BRANCH' in os.environ:
            self._git_official_assets_branch = os.environ['GIT_OFFICIAL_ASSETS_BRANCH']

This is no different than

def version():
    return "1.2" 

... in the current download_assets, which now points the script at a non-existing directory.

Since it's obviosly possible to pull the right code from the right branch and repo, it should also be possible to pull the right assets from the right branch and repo.

But if you're so violently against having revision control for a stable release, I can move the management of these things to jenkins instead for the time being.

#18 Updated by Jonas Hauquier almost 3 years ago

Joel Palmius wrote:

But if you're so violently against having revision control for a stable release, I can move the management of these things to jenkins instead for the time being.

It's more that I'd like to test this a bit more before including it in a stable release that we'd like to release sooner rather than later.

I understand that the script allows for configuring a branch and even a repo. But it does not allow configuring a tag, which is what is needed for a release.
For building eg v1.1 (now and at some point in the future), the procedure should be something like

hg pull -r 1.1.0
git pull -r 1.1.0

Once this is possible, and this happens by default when a packager builds the release (as we all know how most fail to read instructions that are more than 2 lines of text), I think we can use it for making releases.

I'd also like to keep the option

download_assets.py debug

as this is very useful for development.

#19 Updated by Jonas Hauquier almost 3 years ago

Perhaps we do not need a repository for every version. It might be more logical to follow the 2-branch strategy that is used for the code.
Combined with tags, this should be enough.

#20 Updated by Anonymous almost 3 years ago

Joel Palmius wrote:

I think that's an absurd limitation. If something like http://www.makehumancommunity.org/forum/viewtopic.php?f=19&t=13103 pops up, we're not allowed to fix it until the next major release (historically a few years off) because the bug happens to be in an asset and not in the code?

I agree.
The old way to store assets on tuxfamily and create static archives on bitbucket was only a workaround for the lack version system for binaries. Now that git handles binaries, I think that we should handle the assets as the rest of source code.

#21 Updated by Jonas Hauquier almost 3 years ago

I am not debating whether or not to do it.
I am debating whether the system, as it was imagined by Joel right now, is exactly how we want to use it.
And more importantly, whether we want to try this out on the stable branch immediately.

I think we should use branches and tags, instead of different repositories and branches.

In my opinion, this is too early to include in stable at this point. Doing so will make all choices final, even if we discover in a month that we would better have organized things differently.

#22 Updated by Joel Palmius almost 3 years ago

The complement of that is that we'll keep being stuck with the dysfunctional static management of assets if we keep that approach.

If we think version control is a good thing to have, we should move towards that before the release.

But I have manually backed out the change where the git approach is used automatically. For now, I'll deal with that in jenkins instead.

Concerning release schedule: It isn't as if we're going to release this weekend no matter which approach we take. Running an rc or two for a few weeks is a good idea anyhow. With our track record, we can envision 1.1 living for the foreseeable future.

The OSX build hasn't run since early fall for example, so it's largely untested.

#23 Updated by Jonas Hauquier almost 3 years ago

Let's try it in unstable, and decide before 1.1 whether we'll use it for the next release.

Meanwhile, do you agree to update the script so that we have?
  • only one repo for official assets, not one repo per version
  • have two branches in that repo, master and stable, to mimick the branches of the code
  • use tags for checking out releases (the same tags as used on hg, eg. v1.1.0)

Perhaps the v1.0 assets can be archived on the repo as well. It would also give us more material to test with.

#24 Updated by Joel Palmius almost 3 years ago

I can live with that. I'll see to the necessary changes, but it might take a few days.

The reason the 1.0.x and 1.1.x assets ended up in different repos was that there wasn't really a clear line of development between them. I guess I could have first imported the 1.0.x assets, branched, then imported the 1.1.x assets. But since they were so different I didn't see any particular gain.

The 1.2.x assets would obviously be an extension of 1.1.x though.

However, since I am building RCs of stable, I will need the build routines available in stable. So atm it'd be cumbersome to drop the stuff from stable and move it to default.

#25 Updated by Rob Baer almost 3 years ago

Default is essentially already at 1.2.x, and Stable is essentially at 1.1.x, not 1.0.2. We have no branch at 1.0.2 which remains static as released. This happened at r1989:10a28f79ca6a.

Default nightly builds are of very limited value until #991 is addressed.

Conversely, stable nightlies (now pointed at the upcoming MH 1.1.0 release candidate) are needed to test outstanding build issues for resolution as per #994. The build server which has been targeting default should probably switched to targeting stable for all OSes instead. This positions us for the RCs without denying (non-mercurial) users much functionality.

I guess what gets/got lost is what happens to stable nightly builds which has been tracking 1.0.2. Do we produce one last stable build with stable updates through r1988, call it MH 1.0.3, and retire the MH 1.0.x series from the build server? A seemingly less sensible alternative would be to 'archive 1.0.2' that is paired with a corresponding maintained asset repository of the same name? My opinion is to freeze all further code AND assets for 1.0.x at this time, and move on to the 2 branch approach with 1.1.x (stable) and 1.2.x (default) for both code and assets.

Because handling of assets is tied both to the "build server transition" and to "asset-versioning in the code itself", standardized workflows for these times could benefit from standard practices.

#26 Updated by Joel Palmius almost 3 years ago

I'd prefer that we steered people away from 1.0.x. The assets in the community repo are incompatible with that version and so on and so forth. The github repo for 1.0.x assets is largely for keeping an archive copy around (with updated headers), not necessarily because there's a plan to using it for building new releases.

Conversely, at this precise moment in time I'd like to steer testers towards running the RCs rather than running 1.2.x. So I'm not in a particular hurry with getting nightly builds for 1.2.x up right now.

However, since I am building RCs of stable, I will need the build routines available in stable.

What I mean here is that I'm building 1.1.x, not the old 1.0.x stable.

#27 Updated by Anonymous almost 3 years ago

The headers of the assets that will be included in 1.1 (and in future releases) MUST be updated since I don't want the copyright of them.
If for some reasons (not clear to me) we want to continue to use the assets on tuxfamily, they must be updated with new headers.
In my opinion the best way is to have the updated assets on git and delete the assets on tuxfamily.

#28 Updated by Jonas Hauquier almost 3 years ago

Let's not rush this. The assets can be updated on tuxfamily as well, but until the final build is made this does not matter too much.
Also remember that the redirect files for 1.0 will have to remain on tuxfamily, so that 1.0 builds remain working.

#29 Updated by Joel Palmius almost 3 years ago

Manuel Bastioni wrote:

The headers of the assets that will be included in 1.1 (and in future releases) MUST be updated since I don't want the copyright of them.

This is already the case. No build of 1.1 currently include any of the old headers. Note that this is also true of the poseunit header.

I modified the jenkins build procedure so that it pulled the assets from git instead of from tuxfamily.

I also deleted the old nightly builds for the relevant platforms.

Thus what's available as 1.1 is currently:

I suggest anyone who's interested in seeing if the git procedure works test these and try to figure out if anything seems to be missing in them.

#30 Updated by Rob Baer almost 3 years ago

At first pass the Windows RC1-PPA build seems as "all there" as mercurial build (e.g, there are missing slider icons #983).

Edit
However, it seems that this build is a zip packaged within a second zip container, so it did not extract quite as I had expected.

#31 Updated by Joel Palmius almost 3 years ago

Rob Baer wrote:

However, it seems that this build is a zip packaged within a second zip container, so it did not extract quite as I had expected.

I don't understand what this means.

When downloading the RC now, I don't find any nested zip file?

#32 Updated by Rob Baer almost 3 years ago

Windows version
Windows 10 MH 1.1.0 RC2

What I downloaded this morning was labeled RC1 and seemed zip-file nested. I just downloaded current available (labeled RC2) and there is no evidence of .zip nesting. Conclusion - it's fixed.

However, when I click makehuman.exe for this morning's RC1 version it starts right up (as if it is signed properly), but when I click the newer RC2 makehuman.exe, it balks with the "unsigned software" warning. [If you ignore the warning it does start, of course.]

Is this intentional / expected?

#33 Updated by Rob Baer almost 3 years ago

Test of Windows build - Win 10

Downloading the RC1 and RC2 versions from today [Help | About] provides the following version information in comparison to HG:

RC1 MH 1.1.0 r1998 db8a 3a15 f612
RC2 MH 1.1.0 r2002 55f4 570d c8ca
HG MH 1.1.0 r2019 55f4 570d c8ca (most recent stable)

It seems the the current Mercurial version UUID matches HG but the revision number does not. Is this a build script problem?

#34 Updated by Joel Palmius almost 3 years ago

Rob Baer wrote:

It seems the the current Mercurial version UUID matches HG but the revision number does not. Is this a build script problem?

No, that's because a merge happened (stable -> default). This bumps the revision number, but not the UUID.

Concerning the big fat warning... I don't get it when trying, and I'm thinking it is pretty random. It's the old "no-one told us about this software, so it's probably a virus" bias that happens with a few select antivirus programs (previously mainly Norton), together with the lack of code signing.

I don't think there's any way around this. There are two FAQ items which we'll have to refer people to if they keep running into it:

#35 Updated by Joel Palmius almost 3 years ago

Rob Baer wrote:

What I downloaded this morning was labeled RC1 and seemed zip-file nested. I just downloaded current available (labeled RC2) and there is no evidence of .zip nesting. Conclusion - it's fixed.

I still don't understand what this means. Is there an error message? Or do you find a zip file inside a zip file?

The build procedure is exactly the same between rc1 and rc2. Only differences are a few more files were included, and a few lines of code changed in one of the plugins.

#36 Updated by Rob Baer almost 3 years ago

As to the zip thing - let's forget it. It seemed to me there was a nested zip, but there was no error of any kind. I probably just confused myself (and then you). Moving on to the more significant ...:

I don't want to drop my query about version stamping yet (although I may be confused here too). According to what I see r2002 and r2019 DO NOT share the same uuid. The RC2 help | about box claims to be r2002 but it displays the uuid for r2019 (55f4 570d c8ca), not r2002. This is confusing to say the least.

RC2 MH 1.1.0 r2002 55f4 570d c8ca -- Doesn't look right!

To me it looks like the revision is behind, not the uuid. r2002 happened 2 weeks ago and had a different uuid. I'm pretty sure your RC2 build is: r2019 55f4 570d c8ca (most recent stable), not r2002 6651 2212 585c (2 weeks old). The uuid tag is correct, but the revision tag is wrong.

Help | About reports it as having the uuid of the former and the revision of the later.

Here are the revision-uuid match ups according to what I see. Am I missing something?

#37 Updated by Joel Palmius almost 3 years ago

Seems your problem description is solid.

My best guess is that there's a stale VERSION file left behind from some previous build. I'll purge the workspace before the next build.

#38 Updated by Joel Palmius almost 3 years ago

Also, this issue has gotten a bit derailed from what it was originally about. I'll leave it for now, but for further discussions which does not specifically touch github etc, maybe we could break those out into separate issues.

#39 Updated by Jonas Hauquier almost 3 years ago

  • Related to Feature #1010: download_assets_git should alert the user and fail when LFS git plugin is not installed. added

#40 Updated by Joel Palmius over 2 years ago

  • Target version changed from MakeHuman 1.1.0 to MakeHuman 1.1.1

As far as I can see, it works, and I've used it for all the RCs.

I'll keep the issue open, but moving to 1.1.1.

(The discussion on if the repos should be moved/renamed remains)

#41 Updated by Joel Palmius about 2 years ago

  • Target version changed from MakeHuman 1.1.1 to MakeHuman 1.1.2

Also available in: Atom