With the development and changes of our society, more and more territory emergency occurred, thus human detection and recognition have gradually become the main research techniques for security and surveillance systems. Aiming for the recognition and classification of human activities, non-contact anthropometric technology can combine human biometric recognition with sensor detection. The detection, recognition and classification of human activity status can better adapt to different application environments in the case of long distance or non carrying. Specifically, radar technology has a lot of advantages such as it will not be influenced by light intensity so that radar can detect targets in low visibility weather and it can penetrate block clothes and so on. Therefore, there is great research value and application significance in human activity detection and feature extraction by using radar technology.
Due to the development of various anti detection technologies, higher requirements for human activity recognition technology are put forward. Therefore, it is necessary to recognize and depict the behavior of the monitored object and the characteristic parameters of the motion more precisely in the realization of human detection. In recent years, the human activity analysis has been focused on the features of micro-Doppler motion. Characteristics of different micro forms such as stride length, stride frequency, and swing angle can be used as a valid basis for moving target classification and identification , and exist with a unique form. As a carrier of micro-Doppler motion features, the micro-Doppler signals have individual characteristics in the time-frequency analysis for radar signals, such micro-Doppler motion features can be effectively used to recognize and estimate human activity. Thus, this paper will use continuous wave radar, with micro-Doppler signal as the research object, in three aspects, namely, time-frequency analysis, feature extraction and classification to complete the classification and recognition of human activities.
The traditional Fourier transform is not adapted to the analysis of non-stationary signals, but, in the time-frequency domain, statistical pattern classification of non-stationary signal features is possible. Many techniques can be used for estimation of components of a signal, such as short-term Rényi entropy of its time-frequency distribution , wavelets, wavelet packets and the matching pursuit method , and adaptive kernel design current method . Recently, based on the imaging algorithm, the S-method approach is presented for the extraction of micro-Doppler features. The proposed S-method can reduce the influence of cross-terms for the echo of each range cell, the effectiveness is confirmed, and the method has low computational complexity in practical use [5,6].
After the concept of micro-motion proposed by Chen in , human detection and recognition based on radar technology have great improvement. In [8,9,10], the authors used a continuous wave radar to target human activities, which consist of walking, running, walking while holding a stick, crawling, boxing while moving forward, boxing while standing in place, and sitting still. They introduced deep learning algorithms, directly to a raw micro-Doppler spectrogram for both human detection and activity classification problem. The accuracy results were 97.6% for human detection and 90.9% for human activity classification. Professor Narayanan and his team in [11,12,13] investigated the Hilbert–Huang transform (HHT) method, which was adaptive to nonlinear and non-stationary signals. This study presented simulations of simple human activity, which were subsequently validated using experimental data obtained from both an S-band radar and a W-band millimetre wave (mm-wave) radar. The classification accuracies were obtained at distances of up to 90 m between the human and the radar. However, the existence of endpoint effect limits the practical application of HHT algorithm. It still lacks the value research of the characteristics of human body signals after the completion of the HHT analysis. Orovic et al. in [14,15] proposed a human gait classification technique utilizing the arm positive and negative Doppler frequencies and their relative time of occurrence, by using Hermite multi-window processing technique. Meanwhile, the distribution concentration was shown by using Hermite functions of a few first orders, and it preserved favourable properties of the standard S-method. In order to obtain more echo signals of human body, some researchers embark on equipment improvement. A method of using high frequency bands or increasing radar numbers is proposed. In , the dimension of the UWB (Ultra-Wide Band) echoes was reduced using Principal Component Analysis (PCA). In , recognition performance between people undertaking the same activity was assessed on a set of experimental data collected via a continuous wave radar operating at X-band using a Naïve Bayesian classifier and a shape-similarity-spectrum classifier. In , a feature based on the whole matrix U derived from the SVD (Singular Value Decomposition) of the spectrograms has been proposed. It has been shown that high classification accuracy above 98% can be achieved when multistatic data are combined using separated classifiers at each radar node. In , the authors focused on the distinction between unarmed and potentially armed personnel, and it has been shown that some feature combinations can provide reduced classification errors when combining data from multistatic aspect angles. On the other side, other combinations were robust in providing low errors even with only monostatic data.
For the multi-signal cross interference phenomenon of multi-window processing, the adaptive S-method distribution algorithm is proposed based on a standard S-method, which can guarantee the suppression of multi-signal cross interference based on time-frequency aggregation, thus achieving the purpose of multi signal separation.
Adopting processing windows of different lengths according to the different signal segments can improve signal aggregation degree, eliminate adjacent signals cross terms and reduce peak oscillation phenomenon at the same time and be helpful to obtain more pure effective motion signals.
The pendulum arm swing peak matching algorithm of the swing arm is proposed in the process of feature extraction. The algorithm can effectively filter the interference signal by extracting and matching the effective peak value of the swing arm envelope. On this basis, it is more effective to extract relevant feature values.
The usage of software radio technology to build the system prototype realizes the classification of human movement state. By comparing the experimental results in different environments, the higher accuracy of the system is verified, and the stronger anti-interference performance of the system is verified by interference experiments.
2. Experimental Setup
The point scattering model is the simplest model of target electromagnetic scattering excitation. The target can be defined as a three-dimensional reflection function with point scattering characterization, while the occlusion effect can also be achieved in the point scattering pattern. Then, we will analyze human activity process based on point scattering form and extract micro-Doppler motion performance. Mainly starting from non-uniform motion of the human body and swing arm, the scattering point model is used for theoretical derivation of micro-Doppler characteristics induced by the motion, which provides a theoretical basis for subsequent signal analysis.
In the process of motion, the head and trunk of human body can be considered as a whole because one can keep constant posture in the movement process. In other words, the motion of this part can be simplified as human body center mass motion. The center of mass will have a slight fluctuation in the process of moving, so the motion of the center of mass is modeled as radial horizontal motion. The process is shown in Figure 1a.
When the body is walking, the upper limbs mainly move back and forth periodically around the articulatio humeri and the articulation cubiti, the lower limbs mainly move back and forth around the articulatio coxae and the articulation genus, respectively. It is known from the literature that the motion rules of the human articulatio humeri can be expressed by sinusoidal motion, and the freedom degree of the elbow swing of the human body is related to the speed of walking, which can be defined as a fixed value in the same motion mode. Combined with the experimental scene, the arm swing process is considered as the radial sinusoidal vibration of scattering point, which is shown in Figure 1b.
The frequency of terahertz radar ranges from 0.1 THz to 10 THz, which is situated between microwaves and infrared waves . High-resolution range profiles and Doppler signatures can be achieved easily due to the high frequency of terahertz waves. As a result, a terahertz wave has advantages with regards to target detection and recognition . With the development of terahertz radar systems, terahertz imaging , and detection , there has been much interest in the study of terahertz radar. However, the research on practical applications is rare and gesture recognition using terahertz radar is still an unexplored field.
A great deal of the research into hand-gesture recognition is based on computer vision and contact-based gesture classifications . The performance of vision-based approaches depends strongly on lighting conditions . Contact-based gesture recognition demands individuals to be accustomed with the usage of the interface device, which is not adaptable to new users . As a result, the application of vision-based and contact-based gesture recognition has many limits. In contrast, terahertz radar can not only provide full-time observation of targets , but can also work without wearable devices. In addition, terahertz radar can be used for speed and distance detection . It is applicable to the recognition of hand gestures by detecting changes in distance and speed. In recent years, centimeter-wave radar (frequency in the 3–30 GHz range) has been used in gesture recognition systems . However, large-scale motion has usually been studied, which is confined to low resolution. Very recently, a gesture recognition system Soli was designed for some specific usage scenarios based on millimeter-wave radar with a short range (frequency in 60 GHz). Compared with the lower frequency radar system, the terahertz radar has a higher carrier frequency. It’s easy for terahertz radar to achieve wider bandwidth and provide better range resolution, which can precisely capture a change of hand gesture.
Information for tracking, Jet Engine Modulation (JEM), polarization, Doppler shifts, HRRP, and radar images are usually utilized to perform target recognition [11,12,13,14,15]. In this paper, we focus on gesture recognition for terahertz radar using multi-modal signals. Multi-modal signals in terahertz systems include HRRP and Doppler signatures. A range profile of terahertz radar is actually a one-dimensional terahertz radar image. Since terahertz radar has sufficient bandwidth, the shape of the returned wave from a target can easily describe the geometric shape and structure of a target . As a result, a change in targets will definitely lead to a change in range profiles. In addition to the change in the target itself, aspect changes are shown in dynamic gestures. Furthermore, a range profile of a single aspect is sensitive to aspect changes . Therefore, range profiles have been widely used in the target recognition. However, most of the previous studies focused on the target itself. HRRP sequences in continuous time reflect movement characteristics, but gesture recognition is rarely discussed. On the other hand, terahertz radar systems used the theory of Doppler speed detection to measure the offset of a frequency. Doppler signatures obtained from terahertz radar are the velocity information of the target motion, which can be used in the hand-gesture recognition field . Multi-modal signals, including HRRP and Doppler signatures in terahertz radar systems, provide target information for both image and velocity. This property allows terahertz radar signals to carry much more information than single sensors, such as camera, infrared sensors, data gloves, and so on. Since gesture recognition has its advantages, thanks to the characteristics of terahertz waves, gesture recognition represents a promising future development for terahertz radar systems. To our knowledge, there have been no reports regarding gesture recognition in the terahertz region.
Target classification algorithms include Dynamic Time Warping, Hidden Markov model (HMM), Random Forest, Adaptive Boosting (AdaBoost), and so on [19,20,21,22,23]. The Dynamic Time Warping (DTW) algorithm allows two temporal sequences to be aligned in terms of length, and also allows similarities between them to be measured. Therefore, DTW is frequently used in gesture recognition. In , hand gesture data was acquired using a multisensor system and the use of DTW as a fusion processor was studied. In , a modified DTW algorithm is designed for gesture recognition using an inertial-sensor based digital pen. In , gesture signals acquired from a depth camera were accurately classified using a novel DTW algorithm. In , the efficiency of different dynamic time warping methods was compared, based on a database of 2160 simple gestures.
Inspired by the preceding works, gesture recognition using terahertz radar is proposed in this paper. HRRP sequences of hand gestures, in continuous time, are obtained from the received signals of a terahertz radar system. The location parameter features of scattering centers are extracted from HRRP sequences using the Relax algorithm . On the other hand, Doppler signatures are extracted using time-frequency analyses using a terahertz radar. Since two characteristics of hand gestures are obtained, the local distance measure of DTW is extended to deal with vectors at each time point. Furthermore, DTW distance fusion is added to accomplish decision-making. A terahertz radar dataset is built in order to verify the effectiveness of our proposed method. The dataset is composed of 10 different classes of gesture signals performed by five people. The total number of gesture signal samples is 1050. The recognition rates are eventually acquired, based on the dynamic hand gesture dataset. In the experiments, we demonstrate that the recognition scheme proves to be effective in the terahertz region. We also conducted a comparison experiment with project Soli. The experimental results prove the contribution of high resolution of our terahertz radar system and the effectiveness of our method.
The rest of the paper is structured as follows. The terahertz radar system is introduced, and multi-modal signals, including HRRP and Doppler signatures, are achieved in the terahertz region in Section 2. In Section 3, a recognition approach is proposed and a recognition scheme is presented. In Section 4, the experimental data and data preprocessing are discussed. In Section 5, some experimental results are presented. Finally, conclusions are given in Section 6.