A full list of publications is available in Publication page.

Research Topics

Work in Osaka University:

Work in NAIST:

Plant Structure Modeling

We propose a method for inferring three-dimensional (3D) plant branch structures that are hidden under leaves from multi-view observations. Unlike previous geometric approaches that heavily rely on the visibility of the branches or use parametric branching models, our method makes statistical inferences of branch structures in a probabilistic framework. By inferring the probability of branch existence using a Bayesian extension of image-to-image translation applied to each of multi-view images, our method generates a probabilistic plant 3D model, which represents the 3D branching pattern that cannot be directly observed. Experiments demonstrate the usefulness of the proposed approach in generating convincing branch structures in comparison to prior approaches.

  1. Takuma Doi, Fumio Okura, Toshiki Nagahara, Yasuyuki Matsushita, Yasushi Yagi:
    "Descriptor-free multi-view region matching for instance-wise 3D reconstruction"
    Proc. Asian Conf. on Computer Vision (ACCV'20), (oral, acceptance rate: 8%), Dec 2020.
    (CVF open access) (arXiv)
  2. *Takahiro Isokane, *Fumio Okura, Ayaka Ide, Yasuyuki Matsushita, Yasushi Yagi:
    "Probabilistic plant modeling via multi-view image-to-image translation"
    Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR'18), pp. 2906-2915, Jun 2018.
    (Project page)

Dual Task Gait

The performance of dual task, simultaneously performing two tasks, is a useful measure of a person's cognitive abilities because it creates a heavier load on the brain than single tasks. Large-scale datasets of dual-task behavior are required to quantitatively analyze the relationships among dual-task performance, cognitive functions, and personal attributes such as age. We developed an automatic data collection system for dual-task behavior that can be installed in public spaces or facilities. The system is designed as an entertainment kiosk to attract participants. We used the system to collect a large-scale dataset consisting of more than 70,000 sessions of dual-task behavior, in conjunction with a long-running exhibition in a science museum. The resultant dataset, which includes sensor data such as RGB-D image sequences, can be used for learning- and vision-based investigations of human cognitive functions.

Dual task gait analysis
  1. Fumio Okura, Ikuhisa Mitsugami, Masataka Niwa, Kota Aoki, Chengju Zhou, Yasushi Yagi:
    "Automatic collection of dual-task human behavior for analysis of cognitive function"
    ITE Trans. on Media Technology and Applications, Vol. 6, No. 2, pp. 138-150, Apr 2018.
    (Open access paper)
  2. Chengju Zhou, Ikuhisa Mitsugami, Fumio Okura, Kota Aoki, Yasushi Yagi:
    "Growth assessment of school-age children from dual-task observation"
    ITE Trans. on Media Technology and Applications, Vol. 6, No. 4, pp. 286-296, Oct 2018.
    (Open access paper)

Unifying Color and Texture Transfer for Season Transfer

Recent color transfer methods use local information to learn the transformation from a source to an exemplar image, and then transfer this appearance change to a target image. These solutions achieve very successful results for general mood changes, e.g., changing the appearance of an image from ''sunny'' to ''overcast''. However, such methods have a hard time creating new image content, such as leaves on a bare tree. Texture transfer, on the other hand, can synthesize such content but tends to destroy image structure. We propose the first algorithm that unifies color and texture transfer, outperforming both by leveraging their respective strengths. A key novelty in our approach resides in teasing apart appearance changes that can be modelled simply as changes in color versus those that require new image content to be generated. Our method starts with an analysis phase which evaluates the success of color transfer by comparing the exemplar with the source. This analysis then drives a selective, iterative texture transfer algorithm that simultaneously predicts the success of color transfer on the target and synthesizes new content where needed. We demonstrate our unified algorithm by transferring large temporal changes between photographs, such as change of season - e.g., leaves on bare trees or piles of snow on a street - and flooding.

Unified Color and Texture Transfer for Season Transfer
  1. Fumio Okura, Kenneth Vanhoey, Adrien Bousseau, Alexei A. Efros, George Drettakis:
    "Unifying color and texture transfer for predictive appearance manipulation"
    Computer Graphics Forum (Proc. Eurographics Symposium on Rendering), Vol. 34, Issue 4, pp. 53-63, Jun 2015.
    (Low resolution preprint:pdf, 7MB) (Full resolution preprint:pdf, 96MB)
    (Additional results)

Inconsistency Issues in Indirect Augmented Reality

Indirect augmented reality (IAR) employs a unique approach to achieve high-quality synthesis of the real world and the virtual world, unlike traditional augmented reality (AR), which superimposes virtual objects in real time. IAR uses pre-captured omnidirectional images and offline superimposition of virtual objects for achieving jitter- and drift-free geometric registration as well as high-quality photometric registration. However, one drawback of IAR is the inconsistency between the real world and the pre-captured image. In this paper, we present a new classification of IAR inconsistencies and analyze the effect of these inconsistencies on the IAR experience. Accordingly, we propose a novel IAR system that reflects real-world illumination changes by selecting an appropriate image from among multiple pre-captured images obtained under various illumination conditions. The results of experiments conducted at an actual historical site show that the consideration of real-world illumination changes improves the realism of the IAR experience.

Inconsistency Issues in Indirect Augmented Reality
  1. Fumio Okura, Takayuki Akaguma, Tomokazu Sato, Naokazu Yokoya:
    "Addressing temporal inconsistency in indirect augmented reality"
    Multimedia Tools and Applications, Vol. 76, Issue 2, pp. 2671-2695, Jan 2017. (2014 Impact Factor: 1.346)
    (Low resolution preprint:pdf, 4MB) (Full resolution preprint:pdf, 29MB)

Image-Based Rendering for Mixed Reality World Exploration

This study proposes a framework for photorealistic synthesis of virtual objects and virtualized real-world. We combine the offline rendering of virtual objects and image-based rendering (IBR) to take advantage of the high quality of offline rendering without the computational cost of online CG rendering; i.e., it incurs only the cost of the online computation for IBR. Our IBR implementation reduces the computational costs required to online process by generating structured viewpoints (e.g., at every grid point).

  1. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya:
    "Mixed-reality world exploration using image-based rendering"
    ACM Journal on Computing and Cultural Heritage, Vol. 8, Issue 2, Article No. 9, Mar 2015.
    (Preprint:pdf)
  2. 大倉 史生, 神原 誠之, 横矢 直和:
    "事前レンダリング画像群を用いた自由視点画像生成に基づく写実的な拡張現実画像合成",
    日本バーチャルリアリティ学会研究報告, Vol. 18, No. CS-3, pp. 11-16, Sep 2013. (SIG-MR Award)

Free-Viewpoint Mobile Robot Teleoperation Interface

This study proposes a teleoperation interface where an operator can control a robot from freely configured viewpoints using realistic images of the physical world. The viewpoints generated by the proposed interface provide human operators with intuitive control using a head-mounted display and head tracker, and assist them to grasp the environment surrounding the robot. A state-of-the-art free-viewpoint image generation technique is employed to generate the scene presented to the operator. In addition, an augmented reality technique is used to superimpose a 3D model of the robot onto the generated scenes.

  1. Fumio Okura, Yuko Ueda, Tomokazu Sato, Naokazu Yokoya:
    "Teleoperation of mobile robots by generating augmented free-viewpoint images",
    Proc. 2013 IEEE/RSJ Int'l Conf. on Intelligent Robots and Systems (IROS'13), pp. 665-671, Nov 2013. (Paper:pdf)
  2. 上田 優子, 大倉 史生, 佐藤 智和, 横矢 直和:
    "拡張自由視点画像生成を用いた遠隔移動ロボット操縦インタフェース",
    電子情報通信学会 技術研究報告, MVE2012-73, Jan 2013.

Full Spherical HDR Imaging

This study proposes a method for acquiring full spherical high dynamic range (HDR) images without any missing areas by using two omnidirectional cameras mounted on the top and bottom of an unmanned airship. The full spherical HDR images are generated by combining multiple omnidirectional images that are captured with different shutter speeds. The images generated are intended for uses in immersive panorama and its augmentation with image-based lighting.

  1. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya:
    "Aerial full spherical HDR imaging and display"
    Virtual Reality (Springer), Vol. 18, No. 4, pp. 255-269, Nov 2014.
    (Preprint:pdf) / (Video:youtube) / (Video:wmv,11MB)
  2. 大倉 史生, 神原 誠之, 横矢 直和:
    "無人飛行船に搭載された2台の全方位カメラを用いた不可視領域のない全天球HDRビデオの生成"
    日本バーチャルリアリティ学会論文誌, Vol. 17, No. 3, pp. 139-149, Sep 2012. (Paper:pdf) (in Japanese)

Tone Mapping using Region Segmentation

We propose a tone mapping method particularly for HDR images which have two spatially separated luminance distributions of bright and dark regions. We assume that human does not feel a sense of discomfort, even if luminance values between bright and dark regions is reversed, when these regions are definitely divided according to dimidiated luminance and spatial distributions. Under this assumption, we divide an HDR image into bright and dark regions and apply a different tone mapping function to each region independently.

Tone mapping for dimidiate-luminance HDR images
  1. Masaki Kitaura, Fumio Okura, Masayuki Kanbara, Naokazu Yokoya:
    "Tone mapping for HDR images with dimidiate luminance and spatial distributions of bright and dark regions",
    Proc. SPIE Electronic Imaging, Vol. 8292, pp. 829205-01-829205-11, Jan 2012. (Paper:pdf)

Augmented Immersive Panoramas

We developed an augmented immersive panorama system which enables virtual tourism beyond time and space, where immersive panorama is a display method of omnidirectional panoramic images that enables us to look around from a location, like Google Street View. Our application provides a user with both the views of a remote location and related information using augmented reality techniques. This study deals with the geometric and photometric registration problems to generate high-quality augmented omnidirectional videos automatically. The user can look around the scene from the sky above Heijo Palace Site which is an ancient capital in Nara, Japan.

  1. 大倉 史生, 神原 誠之, 横矢 直和:
    "無人飛行船からの空撮全方位動画像を用いた蓄積再生型拡張テレプレゼンス",
    日本バーチャルリアリティ学会論文誌 (Trans. Virtual Reality Society of Japan), Vol. 16, No. 2, pp. 127-138, Jun 2011. (VRSJ Outstanding Paper Award)
    (Paper:pdf) (in Japanese)
  2. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya:
    "Fly-through Heijo Palace Site: Historical tourism system using augmented telepresence",
    Proc. ACM Multimedia (MM'12) Technical Demo, pp. 1283-1284, Oct 2012.
    (Abstruct:pdf) / (Movie:youtube) / (Movie:wmv,11MB)
  3. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya:
    "Fly-through Heijo Palace Site: augmented telepresence using aerial omnidirectional videos",
    Proc. ACM SIGGRAPH'11 Posters, Aug 2011.
    (Abstruct:pdf) / (Poster:pdf,2MB)

Autopilot Aerial Omnidirectional Imaging

An omnidirectional multi-camera system (OMS) mounted on an unmanned airship captures aerial omnidirectional videos suitable for telepresence, augmented/mixed reality, and urban reconstruction. We developed a simple autopilot aerial imaging system.

Unmanned airship
  1. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya:
    "Augmented telepresence using autopilot airship and omni-directional camera",
    Proc. 9th IEEE Int'l Symp. on Mixed and Augmented Reality (ISMAR'10), pp. 259-260, Oct 2010. (Paper:pdf)
  2. 大倉 史生, 神原 誠之, 横矢 直和:
    "空撮画像を用いた拡張テレプレゼンス ~無人飛行船の自動操縦と全方位カメラによるARシステム~",
    画像の認識・理解シンポジウム(MIRU2010)講演論文集, pp. 1183-1189, Jul 2010.