Pre-requisite checker

[luzpaz]

A pre-requisite checker would be a helpful tool for FreeCAD in general, I believe. Maybe it's worth making a FC wide function that can do this for addons in general?
It would help working towards realizing issue 857

We wanted to wait until we have a decent pre-requisite checker, so that a user who installs it via the addon manager and tries to run it without first having installed necessary dependencies, will at least get a helpful error message. This is in the works right now

References:
Mentioned in CFDFoam thread
Discussion going on in Splitting FreeCAD up in to smaller packages thread.
Mantis: Ticket 3032 and 857

What it could check for:

• FEM WB: OpenFOAM @oliveroxtoby & @jaheyns (Other Prereqs)
• FEM WB: Fenics @joha2 (thread)
• FEM WB: CalculiX @berndhahnebach
• FEM WB: Elmer @drhooves AKA m42kus on forum. GSOC 2017
• Graphiz
• Geodata WB: Gdal @microelly2 see forum thread
• ezdxf (available via pypi)
• opencv, cv2 (GeoData WB)
• add other deps.

[luzpaz]

Another example when installing the Geodata WB there is a dependency on the animation WB:
https://forum.freecadweb.org/viewtopic.php?f=4&t=20232#p163690

[luzpaz]

Another example with installing Assembly2
https://forum.freecadweb.org/viewtopic.php?f=22&t=22442&p=174033#p174033

[luzpaz]

@yorikvanhavre any feedback?

[yorikvanhavre]

Yes, we could think about something like that... It could be something simple, such as adding some "PREREQUISITE.md" file to the repo, that would contain a list of other workbenches this one depends on, that the addons manager could read, and install the needed ones if needed...

[luzpaz]

@yorikvanhavre good idea. Sound simple and direct.
@oliveroxtoby and @jaheyns what ideas have you thought about in this regard?

[luzpaz]

@yorikvanhavre just to get the ball rolling:
https://gist.github.com/luzpaz/2722740ce0319219511500c07cfad1f4

Edit: using QGIS metadata.txt format:
https://gist.github.com/luzpaz/0c83375ea503ed46d17b4b47dc35a9d9

[oliveroxtoby]

A simple text file describing dependencies between FreeCAD workbenches sounds fine to me.

When it comes to checking for external software installed on the system, like OpenFOAM, calculix, gmsh, etc, perhaps a function could be mandated in the addons which the manager would call to ask it to check its external dependencies. Or maybe safter just to make this the module's responsibility to do every time it is loaded. I'm not sure how easily it could be automated, especially checking version numbers, etc?

[luzpaz]

Pivy usecase: https://forum.freecadweb.org/viewtopic.php?f=3&t=22453

[looooo]

good idea!
once there was a discussion about adding metadata to the addons in the forum. (Currently I can't find the topic) Maybe it would be a good idea to add dependencies there. If I remeber correctly it was a file called meta.py. Things like [author, version, dependencies, doc-url,...] could be listed there.
Using pip as a crossplatform packagemanager would also solve this issues (partially, because more difficult dependencies can not be installed eg pivy). Something like paraview should simple output a warning if not installed, because such a big library has to be installed with system-package-manager or directly...

[luzpaz]

Examples of QGIS's metadata.txt file within each QGIS plugin:
https://github.com/sourcepole/qgis-openlayers-plugin/blob/master/openlayers/metadata.txt
https://github.com/anitagraser/TimeManager/blob/master/metadata.txt
Though it only has a min version for QGIS

Here is a plugin builder script that is a UI that simplifies the plugin creation process:
http://geoapt.net/pluginbuilder/

[joha2]

For the scipy test I wrote something simple in within the pyrate workbench. There is a paths.txt file where to check and it trys to import scipy afterwards throwing an exception if import fails. Is this the intended meaning of a pre requisite check? https://github.com/joha2/pyrate/blob/master/InitGui.py#L35 Unfortunately I had no better idea at the time being and it would be nice to have such a pre requisite checker. :-)

For the Fenics import/export I use the FEM WB mesh helper functions excessively, so these functions must be available. Up to now there is no Fenics needed, but maybe this will be necessary in the future.

Best wishes
Johannes

[luzpaz]

CFDFoam guys are making some headway: jaheyns/CfdOF@d2f9a1f

[luzpaz]

We could also think about using a metadata.yml format for the file...?

[luzpaz]

Linking an older thread from the forums: The FreeCAD plugin installer

[luzpaz]

Played around with adapting the QGIS metadata approach:
https://gist.github.com/luzpaz/0c83375ea503ed46d17b4b47dc35a9d9

; metadata.txt
[general]
name=ExplodedAssembly
description=Create exploded assembly animations
version=2.4
about=Exploded Assembly adds exploded animation controls to FreeCAD. FreeCAD workbench to create exploded views and animations of assemblies. 
FreeCADMinimumVersion=0.17.xxxxx
author=Javier Martínez (aka @JMG1)
[email protected]
changelog=2.4
    - Fixed #222: hide label if TimeManager is turned off
    - Fixed #156: copy parent style to interpolation layer
    - Fixed #109: save label settings in project
    - Fixed window resizing issues in label options gui
    - Fixed window resizing issues in video export gui
    - Fixed HiDPI issues with arch gui
    2.3.2
    - Sponsored fix for #187 and #211
    - Contributions to testing by @rduivenvoorde
    2.3.1
    - Temporarily fixed #198 sorting of frames wrong for animation?
    - Fixed #197 Export video skips frames by reverting "possible fix for #188, also implements #167? (#189)"
    2.3
    - Fixed #157: Now last animation path is saved in QSettings
    - Made archeology dialog resizable and more readable
    - Added play/pause toggle images 
    - Fixed #172: remove *.PNGw files with "Clear previous frame files in directory"
    - Improvements for WMTS support
tags=exploded-assembly,time,animation,assembly
icon=icon.png
screenshot=http://2.bp.blogspot.com/-Og8hzXXrAS0/VuaVxWhcKEI/AAAAAAAACv4/MCYpnIEPUrgeOrYIxr9ZoGqXdT9_bszjQ/s1600/Captura%2Bde%2Bpantalla%2Bde%2B2016-03-14%2B09%253A32%253A47.png
screecast=https://youtube.com/watch?v=lzYR7I2h7KQ
experimental=False
homepage=http://linuxforanengineer.blogspot.com/
tracker=https://github.com/JMG1/ExplodedAssembly/issues
repository=https://github.com/JMG1/ExplodedAssembly

Perhaps there should be a section appended at the bottom for dependencies like:

[deps]
workbenches=part,assembly,raytracer
python=3.x.x
pylibs=numpy,scipy,
3rdparty=VTK vX.y.z
3rdparty=OCC v7.1.1
FCaddon=assembly2@g3425fdr,part-o-magic@f32sed43

[yorikvanhavre]

I would rather go by parts, because python modules will be a tricky thing to do (different installation methods on different platforms, possible permissions problems, etc). But having one WB depend on another is already something we can work on.

Or, we could start simple, and simply pop up a dialog if some dependencies are unmet (both workbenches and python modules), and let the user sort things by him/herself. Then in a later step we see if some of those can be automated.

Your metadata.txt seems a good solution, is there an existing python parser for it?

[luzpaz]

is there an existing parser for it?

https://github.com/qgis/QGIS-Django/blob/master/qgis-app/plugins/validator.py

FYI a python implementation in the QGIS Official Python Plugin Repository code. BTW, the format is ini file so it's supported by standard libraries.
Per the folks on #QGIS

[yorikvanhavre]

Ok thanks, in any case we can just start simple, with getting the lines that begin with workbenches= and pylibs=

[luzpaz]

@yorikvanhavre FYI QGIS does not have dependency management yet.
Also: http://docs.qgis.org/testing/en/docs/pyqgis_developer_cookbook/plugins.html

[luzpaz]

@yorikvanhavre
Per QGIS regarding 3rd party python lib dependencies:

...you should check and tell the user what to do to install the missing deps. Or, use paver and package them as ext-libs

checkout Paver https://pythonhosted.org/Paver/

Edit: posted to the FC forum

[luzpaz]

The DesignSPHysics folks are taking an interesting approach to installation. Checkout:
https://github.com/ndrs92/DesignSPHysics/blob/master/installer/installer.py

[luzpaz]

Started a boilerplate metadata.txt file at: https://gist.github.com/luzpaz/5e12362f0771eec9e6e82ef0e4d1ba56

; An example for this metadata.txt adapted from QGIS for FreeCAD
; http://docs.qgis.org/testing/en/docs/pyqgis_developer_cookbook/plugins.html#plugin-metadata
; All FreeCAD addons repos should fork this metadata.txt and adapt to their project
; the next section is mandatory

[general]
name=HelloWorld
[email protected]
author=Just Me
FreeCADMinimumVersion=0.17.xxxx
description=This is an example plugin for greeting the world.
    Multiline is allowed:
    lines starting with spaces belong to the same
    field, in this case to the "description" field.
    HTML formatting is not allowed.
about=This paragraph can contain a detailed description
    of the plugin. Multiline is allowed, HTML is not.
version=version 1.2
; tracker can either link to a bugtracker or a forum thread link
tracker=http://bugs.itopen.it
repository=http://www.itopen.it/repo
license=????

; end of mandatory metadata

; start of optional metadata
category=Raster
changelog=The changelog lists the plugin versions
    and their changes as in the example below:
    (also observe the two different styles of changelogs)
    2.4
    - Fixed #222: hide label if TimeManager is turned off
    - Fixed #156: copy parent style to interpolation layer
    - Fixed #109: save label settings in project
    - Fixed window resizing issues in label options gui
    - Fixed window resizing issues in video export gui
    - Fixed HiDPI issues with arch gui
    2.3.2
    - Sponsored fix for #187 and #211
    - Contributions to testing by @rduivenvoorde
    2.3.1
    - Temporarily fixed #198 sorting of frames wrong for animation?
    - Fixed #197 Export video skips frames by reverting "possible fix for #188, also implements #167? (#189)"
    2.3
    - Fixed #157: Now last animation path is saved in QSettings
    - Made archeology dialog resizable and more readable
    - Added play/pause toggle images 
    - Fixed #172: remove *.PNGw files with "Clear previous frame files in directory"
    - Improvements for WMTS support
;   or something like this 
    1.0 - First stable release
    0.9 - All features implemented
    0.8 - First testing release

; Tags are in comma separated value format, spaces are allowed within the
; tag name.
; Tags should be in English language. Please also check for existing tags and
; synonyms before creating a new one.
tags=nurbs,0.17,hello world

; these metadata can be empty, they will eventually become mandatory.
homepage=http://www.itopen.it

; icon size should be: ?????
icon=icon.png
screenshot=screenshot.png

; animated gifs are recommended
screencast=screencast.gif

; experimental flag (applies to the single version)
experimental=True

; deprecated flag (applies to the whole plugin and not only to the uploaded version)
deprecated=False

; if empty, it will be automatically set to major version + .99
FreeCADMaximumVersion=0.17.xxxx

; start of dependency section

[dependencies]
workbenches=part,assembly,raytracer

; specific python version we need to work with?
python=3.x.x

; add 3rd party python libs
pylibs=numpy vX.y.z
pylibs=scipy vX.y.z

; add 3rd party optional python libs
optionalpylibs=scikit vX.y.z

; add 3rd party applications
3rdparty=VTK vX.y.z
3rdparty=OCC v7.1.1

; add dependencies of other FC addons
FCaddon=assembly2@g3425fdr,part-o-magic@f32sed43

; end of dependency section

[joha2]

Is there already code implemented which can read this metadata.txt file? What is the category keyword for? Best wishes
Johannes

[luzpaz]

Is there already code implemented which can read this metadata.txt file?

@joha2 see #29 (comment)
We'd need to adapt it to FreeCAD.

What is the category keyword for ?

We have yet to designate that. Most likely serves as an organization/search function to differentiate plugins/addons from each other.

[yorikvanhavre]

There is now a basic dependency checker implemented in FreeCAD/FreeCAD@456d3b80f
This is how it works:

1. When pressing the "install" button, the addon manager checks if a metadata.txt file exists in the repo's root dir.
2. If yes, the content of that file is downloaded, and searched for lines that begin with workbenches= and pylibs=
3. All names found after workbenches= are searched in FreeCADGui.listWorkbenches().keys(). If not found there, installation will abort (a message will be printed)
4. All names found after pylibs= are tried to import. If it fails, installation will abort (a message will be printed)

[luzpaz]

Sweet!
Question: What if no pylibs are necessary? Then we are aborting erroneously...?

[yorikvanhavre]

Right you are. Fixed in FreeCAD/FreeCAD@58909797a

[joha2]

Nice one! Is there also the possibility to divide pylibs and workbenches in mandatory and optional ones, such that the workbench or the macro could check, whether the optional ones are found and perhaps offer a failsafe solution? E.g. if you need scipy (which is problematic for win64 together with FreeCAD) and you wrote some functionality as failsafe solution for a missing scipy library then it would be nice to have a check whether these optional requirements are met.
Best wishes
Johannes

[yorikvanhavre]

good idea. Done in FreeCAD/FreeCAD@99e400e87

[luzpaz]

Looks like we have a new ini variables optionalpylibs=

Edit: updated the metadata.txt boilerplate

[HoWilgh]

To make it a little bit more complicated... does 'optional' also cover the case that at least one of the options has to be available?
For mesh generation in FEM-wb at least one mesher is needed.For Calculix one can use netgen or gmsh. For CFDFoam one can use gmsh or cfMesh.

[HoWilgh]

BTW: CFDFoam needs gmsh>2.13, gnuplot+python-gnuplot and optional cfMesh.

[luzpaz]

Another reference to addon-manager/pre-requisite checker: https://forum.freecadweb.org/viewtopic.php?f=10&t=23704&p=185205#p185031

[sgrogan]

See starting Here: https://forum.freecadweb.org/viewtopic.php?f=18&t=23548&start=10#p183879
wmayer has given us a template!

[luzpaz]

Created the beginnings of a metadata.txt generator:
https://docs.google.com/forms/d/1-nb1ryX9zEVx1RIuuzMAvTqSr_1tebEpMIsgxAPVTYw/edit?usp=sharing

[joha2]

Please have a look into #71

[luzpaz]

@yorikvanhavre where is the documentation for metadata.txt file?

[chennes]

I've added more complete dependency resolution in FreeCAD/FreeCAD#5339 -- this reads both package.xml and metadata.txt information and consolidates them, and then walks the dependency tree when an Addon is installed and presents the user with a complete list of the package's requirements. At present it does not offer to automatically install them, this is under discussion in the forums. https://forum.freecadweb.org/viewtopic.php?f=22&t=65044

You can now find rudimentary documentation of the metadata.txt file in https://wiki.freecadweb.org/Workbench_creation, and more complete documentation of package.xml in https://wiki.freecadweb.org/Package_Metadata

[chennes]

That PR is now merged, I'm closing this as basically implemented (of course there's always more to do, but the basic feature is there).

[joha2]

@chennes Thanks for merging!