Add MocapNET

animation/MocapNET/doc/DoxygenMainpage.h

@@ -0,0 +1,74 @@
+/**
+@mainpage  MocapNET
+@author Ammar Qammaz a.k.a. AmmarkoV - http://ammar.gr
+
+Ensemble of SNN Encoders for 3D Human Pose Estimation in RGB Images
+
+Website: https://github.com/AmmarkoV/RGBDAcquisition  
+
+@section libIntro Introduction
+
+ We present MocapNET, an ensemble of SNN encoders that estimates the 3D human body pose based on 2D joint estimations extracted from monocular RGB images. MocapNET provides an efficient divide and conquer strategy for supervised learning. It outputs skeletal information directly into the BVH format which can be rendered in real-time or imported without any additional processing in most popular 3D animation software. The proposed architecture achieves 3D human pose estimations at state of the art rates of 400Hz in high-end systems using only CPU processing.
+  
+Please cite the following paper if this work helps your research :
+
+@inproceedings{Qammaz2019,
+  author = {Qammaz, Ammar and Argyros, Antonis A},
+  title = {MocapNET: Ensemble of SNN Encoders for 3D Human Pose Estimation in RGB Images},
+  booktitle = {British Machine Vision Conference (BMVC 2019)},
+  publisher = {BMVA},
+  year = {2019},
+  month = {September},
+  address = {Cardiff, UK},
+  url = {http://users.ics.forth.gr/argyros/res_mocapnet.html},
+  projects =  {CO4ROBOTS,MINGEI},
+  pdflink = {http://users.ics.forth.gr/ argyros/mypapers/2019_09_BMVC_mocapnet.pdf},
+  videolink = {https://youtu.be/fH5e-KMBvM0}
+}
+
+
+
+@section libDesignOverview Overview, System Requirements and Dependencies
+
+MocapNET is a high performance 2D to 3D single person pose estimator. This code base targets recent Linux (Ubuntu 18.04 - 20.04 +) machines, and relies on the Tensorflow C-API and OpenCV. Windows 10 users can try the linux subsystem that has been also reported to work.
+
+Tensorflow is used as the Neural Network framework for our work and OpenCV is used to enable the acquisition of images from webcams or video files as well as to provide an easy visualization method.
+
+We have provided an initialization script that automatically handles most dependencies, as well as download all needed pretrained models. After running it the application should be ready for use.
+
+Any issues not automatically resolved by the script can be reported on the issues section of this repository.
+
+In order to enable a series of easy to use mini-demos with as few dependencies as possible. We have included a MocapNETBenchmark utility which has hardcoded input and output that can run even in a system without OpenCV to give you a performance estimation of our method. If you have OpenCV available you can use our live demo ( MocapNETLiveWebcamDemo binary ) that also contains the 2D joint detector. By giving it the correct parameters you can switch between a cut-down version of OpenPose (--openpose), VNect (--vnect) or our own MobileNet (default) based 2D joint estimator. All of these are automatically downloaded using the initialize.sh script. However in order to achieve higher accuracy estimations you are advised to set up a full OpenPose instance and use it to acquire JSON files with 2D detections that can be subsequently converted to 3D BVH files using the MocapNETJSON binary. They will provide superior accuracy compared to the bundled 2D joint detectors which are provided for faster performance in the live demo, since 2D estimation is the bottleneck of the application. The lowest end machine where this codebase is currently deployed is an Intel Core 2 6600 PC with 4GB of RAM and an NVIDIA GeForce GT740 GPU where MocapNET achieves 50fps and the bundled 2D Joint Detector 10fps for an aggregate 7fps of 3D pose estimation from RGB. Our live demo will try to run the 2D Joint estimation on your GPU and MocapNET 3D estimation on the system CPU to achieve a combined framerate of over 30 fps which in most cases matches or surpasses the acquisition rate of web cameras. Unfortunately there are many GPU compatibility issues with Tensorflow C-API builds since recent versions have dropped CUDA 9.0 support as well as compute capabilities that might be required by your system, you can edit the initialize.sh script and change the variable TENSORFLOW_VERSION according to your needs. If you want CUDA 9.0 you should se it to 1.12.0. If you want CUDA 9.0 and have a card with older compute capabilities (5.2) then choose version 1.11.0. If all else fails you can always recompile the tensorflow C-API to match your specific hardware configuration. You can also use this script that automates building tensorflow r1.15 that might help you, dealing with the Bazel build system and all of its weirdness. Release 1.15 is the final of the 1.x tensorflow tree and is compatible with MocapNET, Tensorflow 2.0 does not yet have C bindings and is thus not yet supported, according to the Tensorflow site, version 2.3 will be the first to re-include C bindings.
+
+If you are interested in generating BVH training data for your research, we have also provided the code that handles randomization and pose perturbation from the CMU dataset. After a successful compilation, dataset generation is accessible using the scripts scripts/createRandomizedDataset.sh and scripts/createTestDataset.sh. All BVH manipulation code is imported from a secondary github project that is automatically downloaded, included and built using the initialize.sh script. These scripts/createRandomizedDataset.sh and scripts/createTestDataset.sh scripts will populate the dataset/ directory with CSV files that contain valid training samples based on the CMU dataset. It is trivial to load these files using python. After loading them using them as training samples in conjunction with a deep learning framework like Keras you can facilitate learning of 2D to 3D BVH.
+
+@section libBuilding Building the library
+
+To compile the library issue :
+
+ sudo apt-get install build-essential cmake libopencv-dev libjpeg-dev libpng-dev libglew-dev libpthread-stubs0-dev
+
+./initialize.sh
+
+After performing changes to the source code, you do not need to rerun the initialization script. You can recompile the code by using :
+
+cd build 
+cmake .. 
+make 
+cd ..
+
+
+@section libUpdating Updating the library
+
+The MocapNET library is under active development, the same thing is true for its dependencies.
+
+In order to update all the relevant parts of the code you can use the update.sh script provided.
+
+./update.sh
+
+If you made changes to the source code that you want to discard and want to revert to the master you can also use the revert.sh script provided
+
+./revert.sh
+
+
+*/

animation/MocapNET/doc/OpenPose.md

@@ -0,0 +1,49 @@
+Workflow for 2-D Video Conversion to a .bvh File on Windows and Linux
+
+	For highest speed and accuracy install OpenPose, in a Linux, Windows or Mac system, using the faster GPU version if a capable GPU is available.  Probably a graphics card with 8GB of VRAM is the minimum needed.  As described later, a Linux system will be needed to run the MocapNET utilities for making conversions to a .bvh file.
+
+	After installing OpenPose, navigate in a terminal to wherever the openpose directory is located.  For example in Windows PowerShell:
+
+cd D:\openpose-1.7.0-binaries-win64-gpu-python3.7-flir-3d_recommended\openpose
+
+	Run OpenPose as shown here for Windows PowerShell:
+
+bin\OpenPoseDemo.exe --video D:\Test4.mp4 --hand --scale_number 8 --scale_gap 0.125  --net_resolution "320x176" --write_json output_jsons/ --model_pose BODY_25B  -number_people_max 1
+
+	The above command specifies slower, higher accuracy scale flags with body bones and additional hand bones.  Add --face to generate face bones if there is enough graphics memory.  A rather low net resolution is needed to prevent Cuda out of video memory errors.   320x176 resolution works on an 8GB VRAM RTX 2070.  If you have more video memory you might be able to set the resolution with --net_resolution -1x368 or higher.  OpenPose can process a crowd but the programs to convert its output to a .bvh file handle one person only.  The --model_pose BODY_25B flag specifies a high resolution body model that can be added as follows:
+
+	Download the Caffe model (100 MB) from:
+
+http://posefs1.perception.cs.cmu.edu/OpenPose/models/pose/1_25BBkg/body_25b/pose_iter_XXXXXX.caffemodel
+
+	Then go to:
+
+https://github.com/CMU-Perceptual-Computing-Lab/openpose_train/blob/master/experimental_models/1_25BBkg/body_25b
+
+	Right-click on the pose_deploy.prototxt file and Save link as..., which will name it as pose_deploy.prototxt.  Save it into an openpose/models/pose/body_25b folder you create.  Also put the Caffe model you downloaded in this folder.  You can now run OpenPose with this higher accuracy model.
+
+The output of the OpenPose run will be in the folder openpose/output_jsons containing .json files that specify the joints’ x and y positions for each frame.  For the next operation change all the .json file names using, for example, Bulk Rename Utility in Windows and rename the resulting folder with a convenient name such as BRUout.  Change sections Name (2) Fixed, and Numbering (10) Mode, Insert at 13, Start 0 and Pad 5.  The exact new names of the .json files should be:
+
+colorFrame_0_00000_keypoints.json, colorFrame_0_00001_keypoints.json, etc.
+
+	
+Install MocapNET
+
+	Run the MocapNET installer in an Ubuntu-like Linux system after installing the dependencies.  MocapNET has utilities to convert the OpenPose .json files into a comma-separated-values (CSV) file, which in turn will be converted into a .bvh file.  If your computer has Windows only, you can try the command-line Windows Subsystem for Linux (WSL) in newer versions of Win 10, though it won’t provide a graphical output.  Otherwise install Ubuntu as a virtual machine in VirtualBox on Windows.  Another solution is to add Ubuntu as a dual-boot option on a Windows computer.
+
+	After installation of MocapNET copy the BRUout folder created above into the MocapNET folder.  Then in a terminal navigate to the MocapNET folder and run:
+
+./convertOpenPoseJSONToCSV --from BRUout  --size 640 480
+
+	The program will run, showing the progress of processing the data and will output a file stored at BRUout/2dJoints_v1.4.csv.  This CSV file can then be further processed with the following command:
+
+./MocapNET2CSV --from BRUout/2dJoints_v1.4.csv  --size 640 480
+
+	After completion the output will be an out.bvh file in the MocapNET folder.  This file can be loaded into Blender, etc. using an import mocap bvh option, and re-targeted to any desired armature for animation.  One good option is to use the MakeHuman program to generate a model with a matching CMU body, hands and face rig.  The MakehHuman MHX2 option provides another way to retarget the .bvh to an armature.
+
+Note that presently the public version of MocapNET supports conversion of body bones only, but hand and face bone support is well underway.
+
+	- end -
+
+
+

animation/MocapNET/doc/ablation_bmvc21/README.md

@@ -0,0 +1,99 @@
+## Towards Holistic Real-time Human 3D Pose Estimation using MocapNETs
+
+### Ablation study experimental results
+
+_Ammar Qammaz_
+<br/>
+<br/>
+
+This is a summary of the ablation experiments that accompanies the following paper:
+
+A. Qammaz and A.A. Argyros, "Towards Holistic Real-time Human 3D Pose Estimation using MocapNETs", In British Machine Vision Conference (BMVC 2021), (to appear), BMVA, UK, November 2021. 
+
+ We conducted all the experiments listed here by acquiring the STB and RHD datasets from the [VAE 3D hands repository](https://github.com/spurra/vae-hands-3d). We used the [HandOnlyTest utility](https://github.com/FORTH-ModelBasedTracker/MocapNET/blob/master/src/MocapNET2/HandOnlyTest/handTest.cpp) to compute standalone 3D hand poses and this [Python utility](https://github.com/FORTH-ModelBasedTracker/MocapNET/blob/master/src/python/compareUtility/compareUtility.py) to compare the 3D output of the proposed method to ground truth and extract results. 
+
+Due to RAM constraints, the largest generated data samples consisted of 7.276.716 M samples. A fully trained ensemble for the left hand took approximately 6 days to train for 7M samples. Using a tenth of the sample size (727K)  we where able to conduct faster experiments to cover a larger variety of combinations, but we also used an intermediate 1/2 sample ratio with 3,638,568 samples also comparing the possible variance that sample size introduces to the training procedure. It should be noted that we generate each of the [3 differently sized datasets *once* using this tool](https://github.com/FORTH-ModelBasedTracker/MocapNET/blob/master/scripts/createRandomHandDataset.sh#L58) in order for all the experiments across a sample size to be directly compareable. The tensorflow [randomized seed](https://www.tensorflow.org/api_docs/python/tf/random/set_seed) is also fixed to 0 for each training to be deterministic and reproducible.
+
+The tables below are all variations on the final proposed setup of the MocapNET3 ensemble that uses *Hierarchical Coordinate Descent (HCD)* optimization as a fine tuning step, *eNSRM* descriptors to encode 2D input data, *SWISH* activations, a dropout of *0.3* or 30% and where each of the encoders regresses *1 degree of freedom(d.o.f.)* . Starting from this configuration we alternate settings to inspect how they affect 3D pose estimation training accuracy on the STB and RHD datasets. 
+
+
+The first set of "small" sample size experiments using 1/10 of the final sample size for different combinations of a 2D Descriptor, comparing the proposed eNSRM with the rich diagonal with the [MocapNET2 NSRM](http://users.ics.forth.gr/~argyros/mypapers/2021_01_ICPR_Qammaz.pdf) and the [Self-normalizing (SELU)](https://arxiv.org/abs/1706.02515) activations of [MocapNET 1](http://users.ics.forth.gr/~argyros/mypapers/2019_09_BMVC_mocapnet.pdf) and [MocapNET 2](http://users.ics.forth.gr/~argyros/mypapers/2021_01_ICPR_Qammaz.pdf). Finally, all runs are tested with and without the Hierarchical Coordinate Descent fine-tuning step introduced in [MocapNET 2](http://users.ics.forth.gr/~argyros/mypapers/2021_01_ICPR_Qammaz.pdf).
+
+<br/>
+
+| Sample Size | Dropout | D.o.f. | HCD | Descriptor | Activation |  STB avg. acc. | RHD avg. acc.  |
+| :-:            | :-:        |         :-: |      :-:     |      :-:     |      :-:     |      :-:     |      :-:     |
+| 727.591 | 0.3 | 1 | &#10003; | NSRM | SWISH | 10.2 mm | 24.9 mm |
+| 727.591 | 0.3 | 1 | &#10003; | eNSRM | SWISH | 11.8 mm | 25.0 mm |
+| 727.591 | 0.3 | 1 | &#10003; | eNSRM | SELU | 10.1 mm | 25.2 mm |
+| 727.591 | 0.3 | 1 | &#10003; | NSRM | SELU | 10.6 mm | 25.1 mm |
+| 727.591 | 0.3 | 1 | &#10007; | NSRM | SWISH | 15.6 mm | 26.1 mm |
+| 727.591 | 0.3 | 1 | &#10007; | eNSRM | SWISH | 18.2 mm | 26.4 mm |
+| 727.591 | 0.3 | 1 | &#10007; | eNSRM | SELU | 18.0 mm | 27.1 mm |
+| 727.591 | 0.3 | 1 | &#10007; | NSRM | SELU | 15.9 mm | 26.5 mm |
+
+
+One of the first observations is that the HCD step plays an important role in improving accuracy, especially on temporarilly cohesive input data such as the STB data.  The results also show that since the HCD improvements improve the results of the neural network, in order to properly compare neural network configurations this is best done when HCD is turned off. We also observe that for this small sample size although the proposed SWISH activation performs better than the [MNET2](http://users.ics.forth.gr/~argyros/mypapers/2021_01_ICPR_Qammaz.pdf) [SELU]((https://arxiv.org/abs/1706.02515) the richer diagonal actually makes the task more difficult for the neural network although the slightly increased error is still recoverable via the HCD step.
+ 
+Another idea we tested is grouping together encoder outputs. Since the internal state of each of the encoders could theoretically be reused across more than one degrees of freedom, especially since many joints feature more than one degrees of freedoms, we attempted to regress 2,3 and 5 degrees of freedom using the same encoder. Testing these configurations without the HCD step reveals a gradually worse accuracy. This is to be expected since this was one of the reasons for the ensemble architecture of encoders with 1 d.o.f. . Our explanation for this behavior is that sharing the capacity of the network between more than one degrees of freedom means less network computations resources dedicated to each of the outputs. It also means that during training the loss function becomes less indicative of each of the outputs making the training procedure gradually harder. 
+
+<br/>
+
+| Sample Size | Dropout | D.o.f. | HCD | Descriptor | Activation |  STB avg. acc. | RHD avg. acc.  |
+| :-:            | :-:        |         :-: |      :-:     |      :-:     |      :-:     |      :-:     |      :-:     |
+| 727.591 | 0.3 | 2 | &#10007; | eNSRM | SWISH | 19.8 mm | 25.0 mm |
+| 727.591 | 0.3 | 3 | &#10007; | eNSRM | SWISH | 25.5 mm | 24.2 mm |
+| 727.591 | 0.3 | 5 | &#10007; | eNSRM | SWISH | 27.0 mm | 27.3 mm |
+
+In order to successfully deepen a densely connected network we have employed residual connections and relatively intense dropout rates. Concerned about wether the high drop-out rate could cause the network encoders (seen in Figure 2 of the paper) to over-rely on the skip connections and thus effectively really utilize only a lower total number of layers, we performed experiments varying the dropout rate to study the effects. We observe that increased dropout improves results especially in the RHD dataset, however the accuracy variance is not indicative of such a side-effect. 
+
+<br/>
+
+| Sample Size | Dropout | D.o.f. | HCD | Descriptor | Activation |  STB avg. acc. | RHD avg. acc.  |
+| :-:            | :-:        |         :-: |      :-:     |      :-:     |      :-:     |      :-:     |      :-:     |
+| 727.591 | 0.1 | 1 | &#10003; | eNSRM | SWISH | 10.1 mm | 25.3 mm |
+| 727.591 | 0.1 | 1 | &#10007; | eNSRM | SWISH | 19.1 mm | 27.2 mm | 
+| 727.591 | 0.2 | 1 | &#10003; | eNSRM | SWISH | 10.9 mm | 24.9 mm |
+| 727.591 | 0.2 | 1 | &#10007; | eNSRM | SWISH | 18.2 mm | 26.7 mm |
+| 727.591 | 0.25 | 1 | &#10003; | eNSRM | SWISH | 10.4 mm | 25.0 mm |
+| 727.591 | 0.25 | 1 | &#10007; | eNSRM | SWISH | 19.3 mm | 26.8 mm |
+
+
+We also performed some ablation experiments using the full sample size proposed in the paper. The following experiments that took approximately 3 weeks to complete, show less variance between the different combinations of design characteristics when HCD is enabled.
+
+<br/>
+
+| Sample Size | Dropout | D.o.f. | HCD | Descriptor | Activation |  STB avg. acc. | RHD avg. acc.  |
+| :-:            | :-:        |         :-: |      :-:     |      :-:     |      :-:     |      :-:     |      :-:     |
+| 7.276.716 | 0.3 | 1 | &#10003; | eNSRM | SWISH | 9.9 mm | 25.3 mm |
+| 7.276.716 | 0.3 | 1 | &#10007; | eNSRM | SWISH | 20.5 mm | 26.8 mm |
+| 7.276.716 | 0.3 | 1 | &#10003; | NSRM | SWISH | 9.8 mm | 25.0 mm |
+| 7.276.716 | 0.3 | 1 | &#10007; | NSRM | SELU | 19.5 mm | 26.3 mm |
+| 7.276.716 | 0.3 | 1 | &#10003; | NSRM | SELU | 9.9 mm | 25.2 mm |
+| 7.276.716 | 0.3 | 1 | &#10007; | NSRM | SELU | 17.6 mm | 27.5 mm |
+
+
+Finally, using a dataset 1/2 the size of our final size we once again perform variations on the proposed configuration and we manage to slightly improve the best results of our method achieving 9.7 mm for the STB and 25.0 mm for the RHD. However, since the BMVC publication policies prohibit updating experimental results during the rebuttal period we did not include these slightly improved figures on the paper. 
+
+<br/>
+
+| Sample Size | Dropout | D.o.f. | HCD | Descriptor | Activation |  STB avg. acc. | RHD avg. acc.  |
+| :-:            | :-:        |         :-: |      :-:     |      :-:     |      :-:     |      :-:     |      :-:     |
+| 3.638.568 | 0.3 | 1 | &#10003; | eNSRM | SWISH | 9.7 mm | 25.0 mm |
+| 3.638.568 | 0.3 | 1 | &#10007; | eNSRM | SWISH | 18.6 mm | 26.7 mm |
+| 3.638.568 | 0.3 | 1 | &#10003; | NSRM | SWISH | 10.0 mm | 24.9 mm |
+| 3.638.568 | 0.3 | 1 | &#10007; | NSRM | SWISH | 17.6 mm | 26.1 mm |
+| 3.638.568 | 0.3 | 1 | &#10003; | NSRM | SELU | 11.5 mm | 25.1 mm |
+| 3.638.568 | 0.3 | 1 | &#10007; | NSRM | SELU | 18.9 mm | 26.3 mm |
+| 3.638.568 | 0.3 | 1 | &#10003; | eNSRM | SELU | 9.9 mm | 25.2 mm |
+| 3.638.568 | 0.3 | 1 | &#10007; | eNSRM | SELU | 17.6 mm | 27.5 mm |
+| 3.638.568 | 0.1 | 1 | &#10003; | eNSRM | SWISH | 10.2 mm | 25.2 mm |
+| 3.638.568 | 0.1 | 1 | &#10007; | eNSRM | SWISH | 17.2 mm | 26.9 mm |
+| 3.638.568 | 0.1 | 1 | &#10003; | eNSRM | SELU | 10.0 mm | 25.4 mm |
+| 3.638.568 | 0.1 | 1 | &#10007; | eNSRM | SELU | 16.3 mm | 27.2 mm |
+| 3.638.568 | 0.1 | 1 | &#10003; | NSRM | SELU | 10.8 mm | 25.2 mm |
+| 3.638.568 | 0.1 | 1 | &#10007; | NSRM | SELU | 19.8 mm | 26.6 mm |
+
+
+To summarize our ablation study results, the hierarchical coordinate descent (HCD) algorithm provides significant pose estimation improvements regardless of the configuration and training of the neural network providing the initial 3D pose estimation. That being said, a network that can arrive closer to the correct solution makes HCD perform even better. A second conclusion from our experimental observations is that a reduced dropout rate does not seem to have a dramatic effect on error thus suggesting that the utilized capacity of the network remains similar under all tested dropout ratios. We observe that a larger number of samples improves network accuracy especially in the case of eNSRM descriptors that feature more complex features and also perform 2D alignment of 2D shapes that are more difficult to "learn". Finally, SELU and SWISH activation functions seem to perform similarly, however due to the slight edge of SWISH experiments and their improved computational performance they seem to be a good choice for the MNET3 ensemble.
+

animation/MocapNET/doc/createDoxygen.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+
+#Switch to this directory
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+cd "$DIR"
+ 
+
+export MOCAPNET_DOXYGEN_INPUT="$DIR/.."
+export MOCAPNET_DOXYGEN_OUTPUT="$DIR/../doc"
+
+echo "Doxygen Input file will be $MOCAPNET_DOXYGEN_INPUT"
+echo "Doxygen Output file will be $MOCAPNET_DOXYGEN_OUTPUT"
+
+
+cd ..
+doxygen doc/doxyfile
+cd doc/latex
+make
+cd .. 
+cd ..
+
+ln -s doc/latex/refman.pdf MocapNETManual.pdf
+
+
+cd "$DIR"
+ 
+
+exit 0