I’m trying to extract SMPL parameters on video in a multishot setting. To this end, tracklets are necessary.

Steps: PHALP setup -> Multishot Human Pose extraction -> select data for training NeRF.

1. PHALP setup

Summary of PHALP

• “Tracking People by Predicting 3D Appearance, Location & Pose” (CVPR 2022)
• Input: monocular video
• Task: Tracking people
• Method
  • lift people in a frame to 3D (3D pose of human, human location in 3D space, 3D appearance)
  • track a person: collect 3D observations over time in a tracklet representation -> build temporal model
    • model predict the future state of a tracklet including 3d location, pose, appearance
    • for the future frame, calculate similarity between pred state of tracklet and single frame observations
  • Association: simple Hungarian matching
• Output: *.pkl file (input for the next step, multishot human pose extraction)
• project page: PHALP project

Installation

Tips

• about branch
  • master
    • pros: compatible to Mshot (CVPR22), relatively easy to install
    • cons: out-dated, hard-coded
  • v1.1
    • pros: fancy code and demos
    • cons: OpenGL and OSMesa would be a little bit tricky, not compatible to Mshot repo.
      • 1. output *.pkl file should be slightly changed
      • 1. this demo code doesn’t generate instance segmentation mask per person, which is necessary to Mshot preprocessing.
• My settings
  • docker container / without conda
  • off-screen
    • Pyrenderer and OpenGL may cause problems, you need to choose EGL or OSMesa
    • I build OSMesa and use PyOpenGL + OSMesa combination. (check ‘Failure Logs…’ under)
  • works on PHALP master and v1.1 branches, and also work for Mshot.
  • Here is pip list output of my environment. req.txt

Failure Logs... (skip this)

Logs - follow this [PHALP](https://github.com/brjathu/PHALP#installation) - it takes quite a time to solve environment in conda... - "('Connection broken: OSError("(104, \'ECONNRESET\')")', OSError("(104, 'ECONNRESET')"))" error - I'm using docker container - to install git+http, 'apt-get install git' - pycocotools error: [link](https://stackoverflow.com/questions/72611914/error-could-not-build-wheels-for-pycocotools-which-is-required-to-install-pypr) - torch.ao is added on 1.10 I tried setting an environment without conda, but failed. issues while installing PHALP: - pip install pyrootutils submitit gdown dill colordict scenedetect pytube - pip install scikit-learn==0.22.2 - OpenGL: ssh env (without display) - install OSMesa [OSMesa doc](https://pyrender.readthedocs.io/en/latest/install/index.html#installmesa) - apt-get install libexpat1-dev - encountered on PosixPath ~~~ error but I couldn't handle it

---

2. OpenPose

Settings

• docker image from the hub: ‘d0ckaaa/openpose’
• execute make commands to install
• download pretrained opencv models by get models script.

---

3. multishot human pose setup

Settings

• follow this multishot
• I use the same docker as the previous one.
• Notes
  • use exactly the same version of smplx (if not, it may cause error)
  • no need to use torch 1.5.
  • change environ variables: PYOPENGL_PLATFORM’ egl → osmesa
  • expand docker’s shared memory size to 2GB
• Data
  • download mshot_data from github official Mshot page (approximately 1GB)
  • download smpl neutral pkl and gmm08 on the official smplify webpage and move to the correct location

Pre-processing

• *.pkl → *.npz
• output contains information for a specific person (create separately)
• these are the initial value of optimization step.

Optimization

• optimize the SMPL parameters using multishot loss

Memo

### Notes. - To extract SMPL by multishot human pose, tracking and identification are needed. - What about the other SMPL algorithms? SMPL extraction models like ROMP, VIBE... - Do they use 2D keypoints? - Regression or optimized based? - How are they inputs like? sequence or only an image? should be cropped? do they work on multiple people? - VIBE: https://arxiv.org/pdf/1912.05656.pdf -