Koc University Human-Human Interaction Behaviour Pattern Recognition Dataset

This repository contains labelled haptic interaction data collected from Human-Human dyads in a joint object manipulation scenario. We conducted an experimental study to generate data that can be used to identify human-human haptic interaction patterns and learn models for capturing salient characteristics of dyadic interactions.

In order to identify human interaction patterns, we have developed an application where two human subjects interact in a virtual environment through the haptic channel using SensAble Phantom Premium haptic devices. The application requires the subjects to coordinate their actions in order to move the rectangular object together in a 2D maze-like environment:

Figure 1. Experimental Setup
Based on our interpretations of user interactions, we identified a set of interaction patterns that were observed frequently in our dyadic object manipulation task. Regarding these patterns, we proposed a human interaction behavior taxonomy. In the proposed taxonomy, the first layer presents a very general categorization of any physical interaction involving multiple agents. In this layer, an interaction-based perspective is adopted to classify the task as being either harmonious, conflicting, or neutral. The second layer is concerned with the “intentions” of the agents. In this sense, it is not related to the resulting motion of the object itself, but is rather responsible for defining whether the agents’ motion plans agree or not. Finally, the last layer describes interaction patterns that are commonly encountered in our task. The proposed technique allows the recognition of interaction behaviours, which constitute the leaves of the following taxonomy tree:
Figure 2. Taxonomy of interaction patterns in dyadic object manipulation

During the experiment, the subjects are presented with two different scenes to observe interaction patterns in both translational and rotational motion. The first scene, which will be called the straight scene from now on, depicts a horizontal path, whereas the second scene, called the bifurcated scene, presents a fork-shaped path for the users to follow. In order to elicit different interaction patterns, we presented the subjects with different manipulation scenarios, in which conflicts between partners are artificially invoked by providing each agent with different visual information about the location of the target configuration.

Please click on the following visuals to get more information on the experimental scenarios in each scene:

Video 1. Experimental Scenarios in the Straight Scene

Video 2. Experimental Scenarios in the Bifurcated Scene


40 subjects (6 female and 34 male), aged between 21 and 29, participated in our study. The subjects were randomly divided into two groups to form dyads that should work as partners during the experiment. Hence, this dataset consists of data collected from 20 human-human dyads. For each dyad, the data is recorded at 1 kHz and stored as a Matlab struct.

Raw Interaction Data


Due to large data size, the videos are not available online. However, you can reconstruct the simulated trials by running this Matlab code.

Labelled Feature Sets

The feature sets are stored as Matlab .mat files, each of which is a 6×1 cell array. Cells 1 to 6 respectively contain instances from classes C1 to C6 as depicted in Fig. 2 above.

Please note that this dataset is available for research purposes only. If you are interested in using the dataset, please do so by citing the following paper:

Cigil E. Madan, Ayse Kucukyilmaz, T. Metin Sezgin, and Cagatay Basdogan. Recognition of Haptic Interaction Patterns in Dyadic Joint Object Manipulation. IEEE Transactions on Haptics, 2015. vol.8, no.1, pp.54,66, Jan.-March 2015. [ Bibtex ] doi: 10.1109/TOH.2014.2384049

The experimental procedure, and the details of how data is collected can be found in the aforementioned paper. May you have any queries, please direct them to Ayse Kucukyilmaz via e-mail:a.kucukyilmaz [at] imperial.ac.uk

The dataset is collected at Koc University (Robotics and Mechatronics Laboratory and Intelligent User Interfaces Laboratory). The copyright of the data remains with this institution.