Automated Illustration of Text to Improve Semantic Comprehension

Fox Chapel Area High School
Suvir Prakash Mirchandani; Konrad Urban


Millions of people worldwide suffer from aphasia, a disorder that severely inhibits language comprehension. Medical professionals suggest that individuals with aphasia have a noticeably greater understanding of pictures than of the written or spoken word. Accordingly, we design a text-to-image converter that augments lingual communication, overcoming the highly constrained input strings and predefined output templates of previous work. This project offers four primary contributions. First, we develop an image processing algorithm that finds a simple graphical representation for each noun in the input text by analyzing Hu mo-ments of contours in images from The Noun Project and Bing Images. Next, we construct a da-taset of 700 human-centric action verbs annotated with corresponding body positions. We train support vector machines to match verbs outside the dataset with appropriate body positions. Our system illustrates body positions and emotions with a generic human representation created using iOS’s Core Animation framework. Third, we design an algorithm that maps abstract nouns to concrete ones that can be illustrated easily. To accomplish this, we use spectral clustering to iden-tify 175 abstract noun classes and annotate these classes with representative concrete nouns. Fi-nally, our system parses two datasets of pre-segmented and pre-captioned real-world images (Im-ageClef and Microsoft COCO) to identify graphical patterns that accurately represent semantic relationships between the words in a sentence. Our tests on human subjects establish the system’s effectiveness in communicating text using im-ages. Beyond people with aphasia, our system can assist individuals with Alzheimer’s or Parkin-son’s, travelers located in foreign countries, and children learning how to read.

「為配合國家發展委員會「推動ODF-CNS15251為政府為文件標準格式實施計畫」,以及 提供使用者有文書軟體選擇的權利,本館檔案下載部分文件將公布ODF開放文件格式, 免費開源軟體可至LibreOffice 下載安裝使用,或依貴慣用的軟體開啟文件。」

檔案名稱 檔案大小 格式
Automated Illustration of Text to Improve Semantic Comprehension 365 KB Adobe Reader(Pdf)檔案