Innovating the Shopping Mall Experience with the Digital Information Wall

Personalized close-up interactivity with engaging distant visual effects

The experience for customers close to the screen should be personalized, while also engaging those standing further away with eye-catching animations.

One challenge with a large-scale interactive screen is accommodating both users who are interacting closely and those who are just passing by. For the former, interactions need to be precise and detailed, while content for the latter should be scaled and abstract to catch their attention.

In this project, I defined two main states: public and private. The public state features flashy, low-resolution animations visible from a distance. When a user taps any element on the screen, a private space is created, focusing on a personalized experience without distractions. As the user interacts further, recommended items gather around them, blurring the line between public and private.

Interactive flow diagrams

The concept was initially explored, designed, and documented in 2D diagrams. While this process helped clarify ideas and organize thoughts, it wasn’t enough to ensure that all project members, including clients, were aligned. Interactive details, in particular, needed to be explored and communicated differently than with conventional web wireframes. This led to the development of coded prototypes.

Using creative coding to communicate interactive design details, focusing on the transition from public to private states

To effectively prototype the dynamic and interactive elements of the Digital Information Wall, I needed a tool that could surpass the limitations of 2D diagrams. Leveraging my experience with creative coding, I initially used Processing, as shown below, to create mockups that tested motion, interactive designs, and creative direction.

However, the complexity of the project required a more robust solution to communicate the transition between public and private states. This led me to learn and use openFrameworks, which allowed for the creation of interactive prototypes that could capture the fluidity and depth of the user experience. The video below shows one of the earlier prototypes developed using openFrameworks.

The transition between public and private states proved to be the most challenging concept to communicate. The prototypes below, though conceptual, conveyed the idea much more clearly. A grid of rectangles in various sizes represents thousands of items dynamically floating. Different colors suggest relationships between the items. When a user touches any of them, a personal state is generated, causing all items to float away from the center of interaction. As the user interacts further, items with similar colors gather around, representing how each interaction invokes recommendations. This version below maintains clarity and emphasizes the transition between the public and private views.

Engineering team spent 4 months to release the first version in Unity 3D

"Given that many of the company's engineers were familiar with Unity 3D, the final product was developed using this platform. Although the 3D perspective was removed, the key difference was that the 2D planes from openFrameworks were still prepared within a three-dimensional environment.

As we connected the product to the shopping mall's e-commerce website and integrated real images, we gained insights into the user experience that were not apparent from the prototypes. For example, we experimented with varying the tile sizes based on the number of people in front of the touchscreens and created scenes where all items of the same color were grouped together.

Although the engineers were overwhelmed, they suggested many creative movements. While I initially preferred simpler rule-based motions, I was pleased with them proposing. Many of their suggestions were incorporated, adding significant variation to the motion.

Expanding the product into wider clients including retail

After the launch of the Digital Information Wall at the shopping mall, many retailers and facilities contacted us to install similar systems. Notable clients include Kiko Milano, the Italian makeup chain. While pragmatic digital cataloging is more optimal on e-commerce websites, many clients recognized the need for a 'wow' effect to make the experience at their brick-and-mortar retail stores stand out.

One challenge we faced was that, to achieve this wow effect, the Digital Information Wall had to be somewhat large-scale, which posed challenges in terms of scaling and maintenance costs. As a result, many clients opted to use this solution at their flagship locations. Public clients, such as municipal convention centers or museums, were also interested in using the wall to showcase photos of local spots.