Yingzhen (Amber) CAO
Idiom Picker
A Multiplayer Augmented-Reality Based Serious Game
Overview
From November 2022, I joined the Human-Crypto Society Lab (HCSL) at CUHK(SZ) as an undergraduate research assistant. We are passionate about exploring the real-world use of emerging technologies, like AR. Our group found the loneliness among Chinese older adults a severe problem during the Covid-19 pandemic. So we proposed a new high-tech intervention on this problem, which is a multiplayer AR serious game. An experiment will be conducted to assess the effectiveness of AR on improving mental health.
Responsibilities
-
Literature Review
-
User Interviews
-
Experiment Design
-
Game Design and Evaluation
Project Content
-
November 2022-Febraury 2023 (In-Progress)
-
Research Project at the HCSL, CUHK(SZ)
-
Team: Hao He (project leader), Tong Chen (game development), and me (literature review, methodology, game design)
Tools Used
-
Unity
-
Leiniao雷鸟 AR Glass
The Problem
.png)
Loneliness negatively impact older adults' health.
Loneliness is a risk factor of broad-based morbidity, including depressive symptomatology, physical health, and functional limitations.
It becomes increasingly prevalent among Chinese older adults
A study in 2016 reported 26.2% older adults endorsed loneliness. China faces a projected increasing number of older adults who suffer from loneliness during the Covid-19 pandemic.
Research
The Covid-19 pandemic not only triggered a loneliness epdemic but also reduced the effectiveness and practicality of traditional interventions on loneliness. We reviewed many literatures and evaluated previous prominent interventions in the context of the pandemic.
The Covid-19 Pandamic
Everyday Physical Distancing
Medical Resource Shortage
Pandemic Lockdowns
Travel Restrictions
Testing Requirements
REDUCE
Traditional Loneliness Interventions
1. Social Facilitation Interventions
to facilitate social interaction with peers, or others who may be lonely. For example, charity-funded friendship clubs, shared interest topic groups...
2. Health and Social Care Provision
involves of health and social care professionals and enrolment in a formal programme of care, either in a nursing home or community setting
5. Psychological Therapies
to utilise recognised therapeutic approaches delivered by trained therapists or health professionals
6. Leisure/ Skill Development
focused on leisure activities and/or skill development. For example, gardening programmes, computer/Internet use, voluntary work, holidays and sports...
3. Befriending Interventions
a form of social facilitation with the aim of formulating new friendships. Usually one-to-one and often involved volunteers.
4. Animal Interventions
canine or feline animal interventions, which focussed mainly on animal-assisted therapy
-
Most need the help of professionals/volunteers
-
Usually face-to-face
-
Many need special settings
.png)
Brainstorming
1. face to face interventions for loneliness are unapplicable during the pandemic
2. Interventions can be remote, but in what way?
Traditional mediums include phonecalls, social medias...Can we use technology to assist interventions?
Next questions:
1. Which technology we are going to embed in loneliness interventions?
2. How to combine the chosen technology and loneliness interventions effectively?
Inspiration
1. Metaverse and Extended Reality
Extended reality (XR) technologies encompass a wide spectrum of immersive technologies, from augmented reality (AR) to mixed reality (MR) to virtual reality (VR). The pandemic has prompted the adoption of extended reality (XR).
We found a number of virtual platforms that construct metaverse have been proposed that officially support social networking. These platforms have strong collaborative interaction capabilities, such as Rec Room and VRChat.
from VRChat: https://hello.vrchat.com/
VRChat is an online virtual world platform that allows users to interact with others with customized 3D avatars and worlds.
2. Serious Games in virtual environment
Serious games are educational and training games to embed the learning contents into the gamifications. From literature review, we found many practices of appling XR technologies to mimic training and learning scenarios in a virtual way to enhance effectiveness of serious games.
For example. This is a VR serious game was designed to provide the training of road crossing skills to children with ASD in both left-hand and right-hand traffic systems. The designed VR serious game supports multiple platforms and user interaction methods, including the Kinect sensor, touch screen, keyboard, and mouse. The Kinect sensor will be used to detect gestures produced by the player. It is used as the primary mode of user interactions. In the serious game, children will learn the traffic rules and how to cross the road safely. Road scenarios include traffic light crossing and zebra crossing.
Idea Process
.png)
√ Augmented Reality
-
VR requires a headset device which is heavier. AR deives, like glasses, are lighter and more comfortable.
-
VR is more likely to result in motion sickness for older adults than AR.
-
AR devices are more affordable than VR devices
√ Serious Games
-
to address accompanying symptoms of loneliness, like reduced cognitive abilities
-
productive enagagement of older adults is more helpful for decreasing loneliness
-
set personalized learning objectives in serious games to match elders' needs
Our Final Idea
A high-tech loneliness intervention
AR-based Multiplayer Serious Game
Objectives:
1. To reduce loneliness of older adults by facilitating social interactions
2. To practice physical and cognitive abilities of older adults through gameful learning
Game Design
This serious game intends to provide Chinese older adults with
-
"self-identification" via a customized avatar construction process
-
faciliate "social interaction" from the collaboration with their families or friends,
-
train "both physical and cognitive abilities" via learing Chinese Phrases.
1. Tools
Leiniao AR Glass
Unity Development Platform
An Andorid Phone
2. User Interaction Method
The interactions based on leiniao AR glasses and its own software are very simple:
1. Adjust the angle of the phone to select items, buttons... on the visual screen
2. Click the screen to confirm every action
3. Communicate with other players through the microphone of the AR glasses
3. Game System
MULTIPLAYER
This is a game designed for two players. For each player, the displaying content is generally the same except for game sections that two players's tasks are different.
BEFORE
We will help older adults cutomize their own avatar. This avatar will appear in the later game, responding to task success and failure through different facial expressions and body languages.
FIRST
Players should choose their roles between the main player and the partner (usually family members, friends...)
The main player will be required to have a primary test on their previous knowledge of Chinese idioms.
SECOND
We will display idioms (that the main player failed to recognize in the previous test) and their explanations at the top of the screen. Each time, the screen only display a single idiom and its explanation. In the center of the screen, there will be a 6*6 table, the main player needs to find the corresponding characters of the idiom they are learning in this board, and try to remember its explanation.
THIRD
The idiom and its explanation at the top of screen will disappear. The main player need to select the right Chinese characters of the idiom from the central table. After picking up all four characters, the main player is required to choose its right Chinese meaning.
The main player will learn all unknown idioms by repeating the second and third steps for every idiom.
FOURTH
After the completion of idioms learning, players will enter the mastery test. In this stage, the partner will be the tester and the main player will be the subject. Partner will select five idioms to test the main player. After the partner read the meaning of the idiom,the main player should find its characters in a 9*9 large table. This process will be repeated until all five idioms in one round have been tested. The number of rounds needed depends on the main player' baseline test. The main player's test score is recorded as finishing time t and success or failure s/f.
Next Steps
RESEARCH
November 2022
(Complete)
DESIGN
TEST
Jan - Feb 2022
February 2022
REPORT
Feb-Mar 2022
More Coming Soon!