How can I custom COLMAP to use the other deep features? Currently I’m stuck into this, so I’ll take a look at the way how DISK customize their output to COLMAP for the following SfM steps.

These days, I’m jumping in the deep end to get familiar with COLMAP and basics of multiview geometry. I’m trying to use deep features (eg. LoFTR) to COLMAP but failed while sparse reconstruction. I think it’s because of my wrong usage of matches_importer. And I find that Sitcoms3D do SfM using COLMAP and they also replace the basic SIFT features of colmap into ‘DISK’ feature.

So, things that I have to do in this step are:

• briefly see the concept of DISK feature algorithm
• input/output
• how to save and load feature in COLMAP
• try to use DISK official code to check the correct values of colmap db sqlite commands
• then, insert my custom feature algorithm to colmap db by editing sqlite commands

Concept of ‘DISK’ feature algorithm

Paper: “DISK: Learning local features with policy gradient” https://arxiv.org/abs/2006.13566 (NeurIPS 2020)

• first, extract a set of local features FA and FB from each
• and then match them to produce a set of correspondences MA↔B
• To learn how to do this through reinforcement learning, they probabilistically redefine feature distribution and match distribution.
• about reward function & gradient estimator are described in paper… (idk RL..)

Input/output

setup ‘DISK’ link environment and try.

• Although I’m using DISK nested in Sitcoms3D repo link but it may not work. Better to use official code.
  • i just manually download pretrained pth file, and replace detect.py into the official version.
• if you use docker, you may need to increase shared memory size. Default docker setting is 64MB, but out of shared memory problem occured to me, so I added ‘–shm-size=2gb’

run detect.py

• INPUT: frames directory
• OUTPUT: *.h5 file (descriptors and keypoints)

Output shape and values of detect.py

run match.py

• INPUT: *.h5 file directory
• OUTPUT: matches.h5 file (descriptors and keypoints)

run colmap/h5_to_db.py

• INPUT: frame dir, h5 dir, output db file path (end with *.db)
  • but Sitcoms3D and my case need masking, so I use binary mask that generated from Detectrion2 (Mask-RCNN)
• OUTPUT: db file that compatible to COLMAP

check values of db add_matches()’s params

array to blob. Original array contains pair information of matching, idx1 and idx2 of each image’s keypoint id.

Then, use exhaustive_matcher and mapper for sparse reconstruction. These all works well in my case.

How to save and load feature in COLMAP.

Please refer to ‘DISK’ repo.

Step 1. Feature Extraction.

• python detect.py h5_artifacts_destination images_directory
• eg. python detect.py –height 1024 –width 1024 –n 2048 scene/h5 scene/images

Step 2. Match Keypoints

• python match.py h5_artifacts_destination
• eg. python match.py –rt 0.95 –save-threshold 100 scene/h5

Step 3. Convert to COLMAP *.db file

• features are inserted to db without descriptor.
• eg. python colmap/h5_to_db.py –database-path scene/database.db scene/h5 scene/images

Step 4. execute colmap cli commands

• exhaustive_matcher and mapper (sparse reconstruction)

Then, let’s check how to convert h5 to db file and insert my custom features like that.

What kinds of information should be added to *.db file?

h5_to_db.py do this.

• db.create_tables()
  • define tables that conpatible to colmap usage
  • by executing ‘CREATE_{CAMERAS, IMAGES, KEYPOINTS, DESCRIPTORS, MATCHES, TWO_VIEW_GEOMETRIES, NAME_INDEX}_TABLE’ which are defined in colmap/colmap/database.py
• add_keypoints()
  • this method() returns ‘fname_to_id’ (fname_to_id[filename] = image_id)
    • connected to db.add_keypoints(image_id, keypoints) which are defined in colmap/colmap/database.py
  • (optional) create_camera(): add new camera model and intrinsics
• check_masking(h5_path, mask_path)
  • this method() returns ‘mask_keyp_id’ (mask_keyp_id[filename] = keep_inds)
  • read mask files and check if keypoint is valid or not, and keep this information in ‘mask_keyp_id’.
• add_matches(db, h5_path, fname_to_id, mask_keyp_id)
  • get colmap image ids (id_1, id_2) from fname_to_id
  • it calls the method db.add_matches(id_1, id_2, matches.T[masked_matches]) for image pairs, and add masked matching keypoints
    • matches = group[key_2][()]
    • i think i need to check the value. is it scaled or not, or something else…
    • my custom feature pairs should fit this form!
• db.commit(): just commit the data to the *.db file

(ing…)