Enhancing Users' Predictions of Robotic Pouring Behaviors Using Augmented Reality: A Case Study

Accepted and presented at the 33rd IEEE International Conference on Robot and Human Interactive Communication, IEEE RO-MAN 2024 held at the Pasadena Convention Center in Pasadena, California, USA on August 26th to 30th, 2024.

Proceedings

Under Construction…

Summary

This project examined how humans, with their intuitive understanding of fluid dynamics, could assist robots in tasks like pouring, where robots often struggled with complex calculations. The study proposed using visualization tools, such as augmented reality (AR), to allow users to predict and correct robotic pouring trajectories before execution. A human-participant study investigated whether people could adjust robotic pouring behaviors to reduce spills using AR headsets and traditional 2D displays. The results showed that visualization tools, particularly AR, helped improve pouring outcomes and reduce spills, highlighting the potential for human-robot collaboration in fluid manipulation tasks. The following video is a short presentation presented as a workshop paper for VAM:HRI 2023.

Motivation

Robots face challenges in understanding the complexity of fluid dynamics, particularly when pouring fluids, which can result in less accurate outcomes. On the other hand, humans, with years of experience, can almost intuitively predict the outcome of their own pouring actions. This raises the questions:

1) Can people predict the outcome of a robot's pouring actions?

2) Can we improve their predictions with visualizations tools?

Pouring Parameters

Rotation Velocity - refers to the percentage of the robot's maximum rotational velocity of the robot's wrist. Participants can control the speed by selecting the speed setting (slow, medium, fast)

Rotation Angle Limit - refers to the limit on how far the wrist can rotate. Users can set the angle rotation by clicking one of the 4 options (90, 120, 150, 180)

Horizontal & Vertical Position - refers to the robot's end effector horizontal and vertical position. By clicking either the increase or decrease buttons, the robot moves along the horizontal and vertical axis approximately 1.5 cm respectively.

Source Containers

Ten common source containers, varying in rim size (wide or narrow), were selected. The research coordinator randomly chose half of the containers to require adjustments to one of four pouring parameters. Container (B) needed a speed change, (E) required an angle limit adjustment, (F) needed a horizontal position change, and (H) and (J) required vertical position changes, with J being an obvious choice due to proximity. Containers A, C, D, G, and I required no adjustments, though some could pour without spilling even if vertical adjustments were made.

(N - narrow) (W - wide)

  1. Soda Can (N)

  2. Storage Container (W)

  3. Blue Cup (W)

  4. Plastic Water Bottle (N)

  5. Sports Bottle (N)

  6. Tumbler (N)

  7. Wine Glass (W)

  8. Root beer Bottle (N)

  9. Paper cup (W)

  10. Mug (W)

Table for Curated Pouring Configurations with parameter correction

Hardware & Software

We used a 6-DoF Kinova Gen3 Lite robot with a 2-finger gripper controlled with Robot Operating System (ROS) and MoveIt running on Ubuntu 18.04. Python and C++ scripts filtered the robotic data before delivery to the AR device and RViz for visualization. Augmented visuals were developed with Unity 2020.3.44f1 and deployed onto a Hololens2 which rendered the data over the physical environment.

Vuforia tracked the robot's position and orientation through a cylindrical target-image fixed to the robot. A digital-twin of the robot provided by Kinova is placed with the target-image to match the real robot setup with C# scripts that control data exchange between the robot and AR device that occurred over a shared Wi-fi network using ROS-TCP Connector. Scaled digital twins of the source containers were created in modeling software and imported to both Unity and RViz

The experiment took place in an isolated lab space that included a robotic manipulator and a target container fixed to a standard table. The amount of water filled in each source container was approximately half the container. A desktop computer connected to the robot was stationed next to the table with a tablet displaying the UI panel to control the robot pouring behavior along with a separate window to record participants' questionnaire responses.

Visual Conditions (Dependent Variable)

Results

Conclusions

In this work, we investigated how visualizing a robot's planned actions could improve human predictions of pouring outcomes. Pouring, intuitive for people but difficult for robots, was used as the task. Participants first predicted the outcome of various pouring behaviors, then adjusted the robot's parameters to reduce spillage. Visualizing the robot's future pouring trajectory in augmented reality (AR) significantly helped participants minimize spills by identifying key faults like pour angle and horizontal positioning. However, predicting spills remained challenging.