视频中的连续手语识别和手语动作合成，分别对应了手语动作的视觉信号和手语语义两者相互转换的过程。这两个问题不仅具有重要的社会意义，同时对于理解手语视频这一类特殊人体运动视频中，视觉信号的时空演化以及手语语义的表达方式，进而推动视频特征学习和时间序列建模，具有重要的研究意义。 连续手语识别中的主要困难，是缺乏视频中具体手语动作与手语语义的明确对应关系。如何实现动作和语义的对应关系挖掘，使模型具有对视频中特定视觉模式的表达和解析能力，是解决连续手语识别的关键。本文针对现有识别方法，在序列学习框架设计和模型优化方法上进行了改进。手语动作合成是一个视频序列上的编码—解码过程。现有的视频合成的方法缺乏对手语表达的细节捕捉和描述，因此 手语表意上存在严重的问题。为了保证合成结果表达语义的准确性，需要模型从手语语义表达的多信道角度出发，保证手型、姿态等各信道信息在时间序列上的准确、同步演化。 本文将手语视频中的连续手语识别和手语动作合成问题，视为机器学习中的典型应用开展研究。本文以循环卷积神经网络为基础模型结构，对上述问题提出了新的解决方案，并通过公开数据集验证了方法的性能。本文的主要贡献有： 提出了基于深度循环卷积神经网络结构的连续手语识别模型，该模型相较于现有方法，在建模上具有对手语时序上下文关系和长时间依赖的表达优势；构建了基于联结时序分类的最大似然估计问题，实现了端到端模型对序列层面对应关系的学习。我们在公开数据集上验证了算法的有效性。 提出了一种连续手语识别模型的迭代优化框架。我们构造了模型特征提取部分的最大似然估计方法，并通过优化目标推导将其转化为强监督的分类任务，充分挖掘了手语动作与单词层面手语语义的对应关系。算法通过对特征提取和序列学习模块的迭代优化，显著提升了模型的识别性能。 构建了手语动作的层次化合成方法。在图像合成模型部分提出了兴趣区域的时空编码模块，提升表意关键区域的合成效果。提出了一种基于动作进度的时序状态编码方式，使手语表达的各信道在合成序列中保持信息同步。实验表明我们的合成结果在外观真实性和表意准确性方面具有明显的优势。 关键词：连续手语识别；手语动作合成；循环卷积神经网络；联结时序分类

 Continuous sign language recognition and gesture synthesis are the counterparts of the translation between visual signals and sign language meanings. Progress in these two fields is not only of great importance to human society, but also helpful for us to better understand the ways, in which spatio-temporal signals evolve, and the semantic meanings are represented in sign language videos. Research on continuous sign language recognition and gesture synthesis can contribute to new progress in learning on representations and sequential structures for video sequences. As for continuous sign language recognition, the main difficulty is lack of prior knowledge on matching between gesticulating and sign language gloss. The key to tackle this problem is to exploit the dependencies of video segments and sign language words, and to make model capable of representing and capturing certain visual patterns in videos. This thesis improves the methods for continuous sign language recognition by developing a new sequence learning framework, and by proposing a new optimization approach. Gesture synthesis is a typical encoding-decoding process on video sequences. Most methods for video synthesis fail to describe the visual details in sign language gesticulating, resulting in deteriorating performance on conveying semantic meanings. To synthesize videos with correct meanings, the synthesis model should be built to describe the precise evolution and synchronization for multiple visual cues in sign language. This thesis proposes the methods for problems on continuous sign language recognition and gesture synthesis, from the perspective of machine learning. We take recurrent convolutional neural networks as the fundamental structure of our models. We propose new approaches to tackle these two problems, and demonstrate the effectiveness of our methods on publicly available databases. Our contributions include: We propose an end-to-end continuous sign language recognition model based on the architecture of recurrent convolutional neural networks, which shows superior capability in modeling the semantic contexts and long-term dependencies in sign language. We introduce the connectionist temporal classification algorithm to build the maximum likelihood estimation problem for training the end-to-end recognition model, and experimental results demonstrate the effectiveness of our method. We propose an iterative training algorithm for the recognition model. We build the maximum likelihood estimation approach for the feature extraction module, and then transform it into a supervised learning problem of gesture classification. We use this training process to fully exploit the dependencies between visual cues and semantic meanings in gloss-level. This optimization process runs on feature extraction and sequence learning modules alternatively, and notably improves the recognition performance of our model. We propose a hierarchical architecture for sign language gesture synthesis. We propose a spatio-temporal decoder for regions of interest in image synthesis module, to improve the visual details in crucial regions for conveying semantic meanings. We develop a new encoding approach for temporal states based on the concept of action progress, which is to instruct our model to synchronize the multiple visual cues in the synthesis process. Our synthesized results present realistic and precise gesticulating compared to the state-of-the-art approaches. Key Words: continuous sign language recognition; gesture synthesis; recurrent convolutional neural networks; connectionist temporal classification