Везде

RAS MathematicsПрограммирование Programming and Computer Software

  • ISSN (Print) 0132-3474
  • ISSN (Online) 3034-5847

METHOD OF GEOMETRY RECONSTRUCTION FROM A SET OF RGB IMAGES USING DIFFERENTIABLE RENDERING AND VISUAL HULL

You can
PII
S3034584725030044-1
DOI
10.7868/S3034584725030044
Publication type
Article
Status
Published
Authors
A. I. Lysykh
  • Affiliation: ITMO University
D. D. Zhdanov
  • Affiliation: ITMO University
M. I. Sorokin
  • Affiliation: ITMO University
Volume/ Edition
Volume / Issue number 3
Pages
40-53
Abstract
The use of differentiable rendering methods is an up-to-date solution to the problem of geometry reconstruction from a set of RGB images without using expensive equipment. The disadvantage of this class of methods is the possible distortions of the geometry that arise during optimization and high computational complexity. Modern differentiable rendering methods calculate and use two types of gradients: silhouette gradients and normal gradients. Most distortions arising in geometry optimization are caused by modifications of parameters associated with silhouette gradients. The paper considers the possibility of increasing the efficiency of geometry reconstruction methods based on the use of differentiable rendering by dividing the reconstruction process into two stages: initialization and optimization. The first stage of reconstruction involves the creation of a visual shell of the reconstructed object. This stage allows one to automate the process of selecting the original geometry and start the next stage under two conditions: the silhouettes of the object have already been reconstructed from all observation points and the topologies of the reconstructed and true objects are equivalent. The second stage comprises a geometry optimization cycle based on the fulfillment of the above conditions. This cycle consists of four steps: image rendering, loss function calculation, gradient calculation, and geometry optimization. Satisfying the condition of matching the contours of the original and reference geometry eliminates the need to use silhouette gradients. This solution significantly reduces the number of errors that occur during optimization, as well as reduces the computational complexity of the method by eliminating the calculation of the loss function, gradient calculation, and optimization of parameters associated with the silhouettes of objects. The testing and analysis of the results showed an increase in the accuracy of geometry reconstruction with a decrease in grid resolution and a decrease in the total running time of the method in comparison with similar methods, as well as an up to two-fold increase in the speed of optimization steps.
Keywords
дифференцируемый рендеринг восстановление геометрии реконструкция геометрии
Date of publication
02.06.2025
Year of publication
2025
Number of purchasers
0
Views
64

References

  1. 1. Cardenas-Garcia J.F., Yao H.G., Zheng S. 3D reconstruction of objects using stereo imaging // Optics and Lasers in Engineering. 1995. V. 22. № 3. P. 193-213.
  2. 2. Mikhail E.M. Introduction to Modern Photogrammetry // John Williey & Sons. 2001.
  3. 3. He C., Shen Y., Forbes A. Towards higher-dimensional structured light // Light: Science & Applications. 2022. V. 11. № 1. P. 205.
  4. 4. Collis R.T.H. Lidar // Applied optics. 1970. V. 9. № 8. P. 1782-1788.
  5. 5. Zhou Z. et al. Three-Dimensional Geometry Reconstruction Method from Multi-View ISAR Images Utilizing Deep Learning // Remote Sensing. 2023. V. 15. № 7. P. 1882.
  6. 6. Liu Z.N. et al. High-quality textured 3D shape reconstruction with cascaded fully convolutional networks // IEEE Transactions on Visualization and Computer Graphics. 2019. V. 27. № 1. P. 83-97.
  7. 7. Henderson P., Ferrari V. Learning to generate and reconstruct 3d meshes with only 2d supervision // arXiv preprint arXiv:1807.09259. 2018.
  8. 8. Kato H. et al. Differentiable rendering: A survey // arXiv preprint arXiv:2006.12057. 2020.
  9. 9. Periyasamy A.S., Behnke S. Towards 3D Scene Understanding Using Differentiable Rendering // SN Computer Science. 2023. V. 4. № 3. P. 245.
  10. 10. Nicolet B., Jacobson A., Jakob W. Large steps in inverse rendering of geometry // ACM Transactions on Graphics (TOG). 2021. V. 40. № 6. P. 1-13.
  11. 11. Rojas R. The backpropagation algorithm // Neural networks: a systematic introduction. 1996. P. 149-182.
  12. 12. Kato H., Ushiku Y., Harada T. Neural 3d mesh renderer // Proceedings of the IEEE conference on computer vision and pattern recognition. 2018. P. 3907-3916.
  13. 13. Liu S. et al. Soft rasterizer: A differentiable renderer for image-based 3d reasoning // Proceedings of the IEEE/CVF international conference on computer vision. 2019. P. 7708-7717.
  14. 14. Chen W. et al. Learning to predict 3d objects with an interpolation-based differentiable renderer // Advances in neural information processing systems. 2019. V. 32.
  15. 15. Petersen F., Bermano A.H., Deussen O., Cohen-Or D. “Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer”. arXiv:1903.11149. 2019.
  16. 16. Yan X., Yang J., Yumer E., Guo Y., Lee H. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision // NeurIPS, 2016.
  17. 17. Tulsiani S., Zhou T., Efros A.A., Malik J. Multi-view supervision for single-view reconstruction via differentiable ray consistency // CVPR, 2017.
  18. 18. Insafutdinov E., Dosovitskiy A. Unsupervised Learning of Shape and Pose with Differentiable Point Clouds // in NeurIPS, 2018.
  19. 19. Liu S., Saito S., Chen W., Li H. Learning to infer implicit surfaces without 3d supervision // NeurIPS, 2019.
  20. 20. Niemeyer M., Mescheder L., Oechsle M., Geiger A. Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision // CVPR, 2020.
  21. 21. Mildenhall B. et al. Nerf: Representing scenes as neural radiance fields for view synthesis // Communications of the ACM. 2021. V. 65. № 1. P. 99-106.
  22. 22. Yang J., Pavone M., Wang Y. Freenerf: Improving few-shot neural rendering with free frequency regularization // Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023. P. 8254-8263.
  23. 23. Kerbl B. et al. 3d gaussian splatting for real-time radiance field rendering // ACM Trans. Graph. 2023. V. 42. № 4. P. 139:1-139:14.
  24. 24. Guédon A., Lepetit V. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. P. 5354-5363.
  25. 25. Jiang Y. et al. Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization // Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020. P. 1251-1261.
  26. 26. Lombardi S. et al. Neural volumes: Learning dynamic renderable volumes from images // arXiv preprint arXiv:1906.07751. 2019.
  27. 27. Yifan W. et al. Differentiable surface splatting for point-based geometry processing // ACM Transactions on Graphics (TOG). 2019. V. 38. № 6. P. 1-14.
  28. 28. Godard C., Mac Aodha O., Brostow G.J. Unsupervised monocular depth estimation with left-right consistency // Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. P. 270-279.
  29. 29. Kajiya J.T. The rendering equation // Proceedings of the 13th annual conference on Computer graphics and interactive techniques. 1986. P. 143-150.
  30. 30. Li T.M. et al. Differentiable monte carlo ray tracing through edge sampling // ACM Transactions on Graphics (TOG). 2018. V. 37. № 6. P. 1-11.
  31. 31. Zhang C. et al. A differential theory of radiative transfer // ACM Transactions on Graphics (TOG). 2019. V. 38. № 6. P. 1-16.
  32. 32. Zhang Z., Roussel N., Jakob W. Projective sampling for differentiable rendering of geometry // ACM Transactions on Graphics (TOG). 2023. V. 42. № 6. P. 1-14.
  33. 33. Loubet G., Holzschuch N., Jakob W. Reparameterizing discontinuous integrands for differentiable rendering // ACM Transactions on Graphics (TOG). 2019. V. 38. № 6. P. 1-14.
  34. 34. Xu P. et al. Warped-area reparameterization of differential path integrals // ACM Transactions on Graphics (TOG). 2023. V. 42. № 6. P. 1-18.
  35. 35. Loper M.M., Black M.J. OpenDR: An approximate differentiable renderer // Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VII 13. - Springer International Publishing, 2014. P. 154-169.
  36. 36. Laine S. et al. Modular primitives for high-performance differentiable rendering // ACM Transactions on Graphics (ToG). 2020. V. 39. № 6. P. 1-14.
  37. 37. Ravi N. et al. Accelerating 3d deep learning with pytorch3d // arXiv preprint arXiv:2007.08501. 2020.
  38. 38. Gupta K. Neural mesh flow: 3d manifold mesh generation via diffeomorphic flows. University of California, San Diego, 2020.
  39. 39. Palfinger W. Continuous remeshing for inverse rendering // Computer Animation and Virtual Worlds. 2022. V. 33. № 5. P. e2101.
  40. 40. Hoppe H. et al. Mesh optimization // Proceedings of the 20th annual conference on Computer graphics and interactive techniques. 1993. P. 19-26.
  41. 41. Лысых А.И., Жданов Д.Д., Сорокин М.И. Использование визуальной оболочки для реконструкции геометрии по набору RGB-изображений с помощью дифференцируемого рендеринга // Материалы 34-й Международной конференции по компьютерной графике и машинному зрению. Омск: Омский государственный технический университет, 2024. С. 238-249.
  42. 42. Laurentini A. The visual hull concept for silhouette-based image understanding // IEEE Transactions on pattern analysis and machine intelligence. 1994. V. 16. № 2. P. 150-162.
  43. 43. Ren T. et al. Grounded sam: Assembling open-world models for diverse visual tasks // arXiv preprint arXiv:2401.14159. 2024.
  44. 44. Liu S. et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection // European Conference on Computer Vision. Springer, Cham, 2025. P. 38-55.
  45. 45. Ren T. et al. Grounding DINO 1.5: Advance the “Edge” of Open-Set Object Detection // arXiv preprint arXiv:2405.10300. 2024.
  46. 46. Ravi N. et al. Sam 2: Segment anything in images and videos // arXiv preprint arXiv:2408.00714. 2024.
  47. 47. Visvalingam M., Whyatt J.D. Line generalization by repeated elimination of points // Landmarks in Mapping. Routledge, 2017. P. 144-155.
  48. 48. Ramer U. An iterative procedure for the polygonal approximation of plane curves // Computer graphics and image processing. 1972. V. 1. № 3. P. 244-256.
  49. 49. Douglas D.H., Peucker T.K. Algorithms for the reduction of the number of points required to represent a digitized line or its caricature // Cartographica: the international journal for geographic information and geovisualization. 1973. V. 10. № 2. P. 112-122.
  50. 50. Paszke A. et al. Pytorch: An imperative style, high-performance deep learning library // Advances in neural information processing systems. 2019. V. 32.
QR
Translate

Индексирование

Scopus

Scopus

Scopus

Crossref

Scopus

Higher Attestation Commission

At the Ministry of Education and Science of the Russian Federation

Scopus

Scientific Electronic Library