Enabling Multimodal User Interactions for Genomics Visualization Creation

Qianwen Wang; Xiao Liu; Man Qing Liang; Sehi L'Yi; Nils Gehlenborg

doi:10.1109/VIS54172.2023.00031

Enabling Multimodal User Interactions for Genomics Visualization Creation

Qianwen Wang, Xiao Liu, Man Qing Liang, Sehi L'Yi, Nils Gehlenborg

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Visualization plays an important role in extracting insights from complex and large-scale genomics data. Traditional graphical user interfaces (GUIs) offer limited flexibility for custom visualizations. Our prior work, Gosling, enables expressive visualization creation using a grammar-based approach, but beginners may face challenges in constructing complex visualizations. To address this, we explore multimodal interactions, including sketches, example images, and natural language inputs, to streamline visualization creation. Specifically, we customize two deep learning models (YOLO v7 and GPT3.5) to interpret user interactions and convert them into Gosling specifications. A workflow is proposed to progressively introduce and integrate multimodal interactions. We then present use cases demonstrating their effectiveness and identify challenges and opportunities for future research.

Original language	English (US)
Title of host publication	Proceedings - 2023 IEEE Visualization Conference - Short Papers, VIS 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	111-115
Number of pages	5
ISBN (Electronic)	9798350325577
DOIs	https://doi.org/10.1109/VIS54172.2023.00031
State	Published - 2023
Externally published	Yes
Event	2023 IEEE Visualization Conference, VIS 2023 - Hybrid, Melbourne, Australia Duration: Oct 22 2023 → Oct 27 2023

Publication series

Name	Proceedings - 2023 IEEE Visualization Conference - Short Papers, VIS 2023

Conference

Conference	2023 IEEE Visualization Conference, VIS 2023
Country/Territory	Australia
City	Hybrid, Melbourne
Period	10/22/23 → 10/27/23

Bibliographical note

Publisher Copyright:
© 2023 IEEE.

Keywords

Human-centered computing
Human-centered computing
Interaction Design
Visualization
Visualization systems and tools

Publisher link

10.1109/VIS54172.2023.00031

Find It

Find It at U of M Libraries

Cite this

Wang, Q., Liu, X., Liang, M. Q., L'Yi, S., & Gehlenborg, N. (2023). Enabling Multimodal User Interactions for Genomics Visualization Creation. In Proceedings - 2023 IEEE Visualization Conference - Short Papers, VIS 2023 (pp. 111-115). (Proceedings - 2023 IEEE Visualization Conference - Short Papers, VIS 2023). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/VIS54172.2023.00031

Enabling Multimodal User Interactions for Genomics Visualization Creation. / Wang, Qianwen; Liu, Xiao; Liang, Man Qing et al.
Proceedings - 2023 IEEE Visualization Conference - Short Papers, VIS 2023. Institute of Electrical and Electronics Engineers Inc., 2023. p. 111-115 (Proceedings - 2023 IEEE Visualization Conference - Short Papers, VIS 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Wang, Q, Liu, X, Liang, MQ, L'Yi, S & Gehlenborg, N 2023, Enabling Multimodal User Interactions for Genomics Visualization Creation. in Proceedings - 2023 IEEE Visualization Conference - Short Papers, VIS 2023. Proceedings - 2023 IEEE Visualization Conference - Short Papers, VIS 2023, Institute of Electrical and Electronics Engineers Inc., pp. 111-115, 2023 IEEE Visualization Conference, VIS 2023, Hybrid, Melbourne, Australia, 10/22/23. https://doi.org/10.1109/VIS54172.2023.00031

Wang Q, Liu X, Liang MQ, L'Yi S, Gehlenborg N. Enabling Multimodal User Interactions for Genomics Visualization Creation. In Proceedings - 2023 IEEE Visualization Conference - Short Papers, VIS 2023. Institute of Electrical and Electronics Engineers Inc. 2023. p. 111-115. (Proceedings - 2023 IEEE Visualization Conference - Short Papers, VIS 2023). doi: 10.1109/VIS54172.2023.00031

@inproceedings{2e13100eb3e947c6b1936140b3a4d717,

title = "Enabling Multimodal User Interactions for Genomics Visualization Creation",

abstract = "Visualization plays an important role in extracting insights from complex and large-scale genomics data. Traditional graphical user interfaces (GUIs) offer limited flexibility for custom visualizations. Our prior work, Gosling, enables expressive visualization creation using a grammar-based approach, but beginners may face challenges in constructing complex visualizations. To address this, we explore multimodal interactions, including sketches, example images, and natural language inputs, to streamline visualization creation. Specifically, we customize two deep learning models (YOLO v7 and GPT3.5) to interpret user interactions and convert them into Gosling specifications. A workflow is proposed to progressively introduce and integrate multimodal interactions. We then present use cases demonstrating their effectiveness and identify challenges and opportunities for future research.",

keywords = "Human-centered computing, Human-centered computing, Interaction Design, Visualization, Visualization systems and tools",

author = "Qianwen Wang and Xiao Liu and Liang, {Man Qing} and Sehi L'Yi and Nils Gehlenborg",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 IEEE Visualization Conference, VIS 2023 ; Conference date: 22-10-2023 Through 27-10-2023",

year = "2023",

doi = "10.1109/VIS54172.2023.00031",

language = "English (US)",

series = "Proceedings - 2023 IEEE Visualization Conference - Short Papers, VIS 2023",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "111--115",

booktitle = "Proceedings - 2023 IEEE Visualization Conference - Short Papers, VIS 2023",

}

TY - GEN

T1 - Enabling Multimodal User Interactions for Genomics Visualization Creation

AU - Wang, Qianwen

AU - Liu, Xiao

AU - Liang, Man Qing

AU - L'Yi, Sehi

AU - Gehlenborg, Nils

PY - 2023

Y1 - 2023

N2 - Visualization plays an important role in extracting insights from complex and large-scale genomics data. Traditional graphical user interfaces (GUIs) offer limited flexibility for custom visualizations. Our prior work, Gosling, enables expressive visualization creation using a grammar-based approach, but beginners may face challenges in constructing complex visualizations. To address this, we explore multimodal interactions, including sketches, example images, and natural language inputs, to streamline visualization creation. Specifically, we customize two deep learning models (YOLO v7 and GPT3.5) to interpret user interactions and convert them into Gosling specifications. A workflow is proposed to progressively introduce and integrate multimodal interactions. We then present use cases demonstrating their effectiveness and identify challenges and opportunities for future research.

AB - Visualization plays an important role in extracting insights from complex and large-scale genomics data. Traditional graphical user interfaces (GUIs) offer limited flexibility for custom visualizations. Our prior work, Gosling, enables expressive visualization creation using a grammar-based approach, but beginners may face challenges in constructing complex visualizations. To address this, we explore multimodal interactions, including sketches, example images, and natural language inputs, to streamline visualization creation. Specifically, we customize two deep learning models (YOLO v7 and GPT3.5) to interpret user interactions and convert them into Gosling specifications. A workflow is proposed to progressively introduce and integrate multimodal interactions. We then present use cases demonstrating their effectiveness and identify challenges and opportunities for future research.

KW - Human-centered computing

KW - Interaction Design

KW - Visualization

KW - Visualization systems and tools

UR - http://www.scopus.com/inward/record.url?scp=85182596464&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85182596464&partnerID=8YFLogxK

U2 - 10.1109/VIS54172.2023.00031

DO - 10.1109/VIS54172.2023.00031

M3 - Conference contribution

AN - SCOPUS:85182596464

T3 - Proceedings - 2023 IEEE Visualization Conference - Short Papers, VIS 2023

SP - 111

EP - 115

BT - Proceedings - 2023 IEEE Visualization Conference - Short Papers, VIS 2023

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2023 IEEE Visualization Conference, VIS 2023

Y2 - 22 October 2023 through 27 October 2023

ER -

Enabling Multimodal User Interactions for Genomics Visualization Creation

Abstract

Publication series

Conference

Bibliographical note

Keywords

Publisher link

Find It

Other files and links

Fingerprint

Cite this