Enabling Semantic-Rich Location Search on Street View Imagery Using Multilingual POI Data

Leeje Jang, Yao Yi Chiang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Street view imagery is a valuable resource for understanding the physical environment and public health. Open-access street view imagery platforms offer street view imagery along with metadata, such as geographic coordinates and capture details (e.g., the times-tamp of when the photo was taken and camera product type). Meta-data improves the utility of street view imagery for various location-based tasks, including urban studies and geospatial search. Recent research has further improved street view imagery by incorporating additional metadata, including human perceptions, weather conditions, and seasonal details. However, connecting street view imagery with Points of Interest (POI) data, including their names and amenity types, remains challenging. In this paper, we propose MulMapper, an automated system that enhances street view imagery metadata by extracting multilingual text, identifying text alignment, and matching POI attributes to the corresponding entity in the image. We develop two modules on top of one of the existing text spotter models: (1) a ‘text alignment detector’ to capture text alignment types and (2) a ‘character-wise text classification loss’ to overcome long-tail recognition issues, which result from imbalanced data distribution across diverse character sets. The proposed method greatly enhances the accuracy of matching between POIs and street view imagery while also enabling more semantically rich location searches within the images.

Original languageEnglish (US)
Title of host publicationGeoSearch 2024 - Proceedings of the 3rd ACM SIGSPATIAL International Workshop on Searching and Mining Large Collections of Geospatial Data
EditorsHao Li, Abhishek Potnis, Wenwen Li, Dalton Lunga, Martin Werner, Andreas Zufle
PublisherAssociation for Computing Machinery, Inc
Pages29-35
Number of pages7
ISBN (Electronic)9798400711480
StatePublished - Oct 29 2024
Event3rd ACM SIGSPATIAL International Workshop on Searching and Mining Large Collections of Geospatial Data, GeoSearch 2024 - Atlanta, United States
Duration: Oct 29 2024 → …

Publication series

NameGeoSearch 2024 - Proceedings of the 3rd ACM SIGSPATIAL International Workshop on Searching and Mining Large Collections of Geospatial Data

Conference

Conference3rd ACM SIGSPATIAL International Workshop on Searching and Mining Large Collections of Geospatial Data, GeoSearch 2024
Country/TerritoryUnited States
CityAtlanta
Period10/29/24 → …

Bibliographical note

Publisher Copyright:
© 2024 Copyright held by the owner/author(s).

Fingerprint

Dive into the research topics of 'Enabling Semantic-Rich Location Search on Street View Imagery Using Multilingual POI Data'. Together they form a unique fingerprint.

Cite this