图像内容的学习与理解是人工智能必须经历的一个过程。本论文以自然图像为研究对象，通过研究图像中包含的物体类别，挖掘目标的属性以及目标与目标之间的相互关系，利用计算机程序和人工智能方法完成图像内容的语义解释，使计算机系统自动获得图像所表达的信息内容。而语言作为人类交流的重要工具，也是智能的一个重要标志，因此，研究者们也希望计算机能够像人类一样，不仅能学习和理解图像的内容，还能够像人类一样能够用自然语言描述出来，能够像人类一样会推理，会像人类一样正确回答问题。这和人工智能领域的一个梦想是一致的：让计算机能够看到并理解我们周围丰富的视觉世界，并赋予它们以自然语言与人类沟通的能力。 图像内容的智能学习与理解作为人工智能领域与计算机视觉领域的重要交叉学科，自然也受到研究者们广泛的研究。本论文结合这一课题，主要针对两个子任务——图像的文本描述与视觉问答任务，这两个任务将图像内容的理解同人类语言联系起来，这也是走向真正人工智能必须经历的一步。本论文主要研究完成上述两个任务的方法。主要内容和贡献归纳如下： (1)基于门控递归单元的图像文本描述算法。针对传统的非深度算法存在生成的图像描述语句句式单一、长短固定、描述内容极其有限的缺点，本论文利用现在流行的强大的深度网络来实现图像的文本描述。首先，利用深度卷积神经网络对图像进行编码，提取更具判别性和表达性的图像全局特征；其次，利用门控递归单元作为多模态融合学习和语句生成器模块，该模型不仅可以生成长度可变、样式丰富的自然语句，而且能够充分挖掘自然语言和图像之间的多模态映射关系。在此基础上，我们通过加深递归单元的深度，使其能够更好的模拟图像和自然语言的非线性关系。该算法在三个主要图像-文本描述数据集上进行了验证，实验结果表明，该算法能够很好的实现从图像到文本的“翻译”。 (2)基于图像全局-局部特征及注意力机制的图像文本描述算法。由于基于图像全局特征的图像文本描述算法仅能学习整幅图像和整个描述语句的多模态映射关系，显然，这种映射有些粗糙，因此，我们需要进一步寻找图像局部区域与自然语言基本单元之间的细粒度映射关系。该算法利用视觉注意力机制来挖掘这种对应关系，每次生成的单词用来选取相应的图像区域。然而，现有利用注意力机制的方法仅仅用到图像局部特征，抛弃了图像全局特征，而图像全局特征蕴藏着图像的全局信息，是对局部特征的重要补充。同时，局部特征会受到图像尺度变化的影响。因此，该算法将图像全局特征和局部特征通过多模态融合模块进行融合。此外，为了进一步提升现有方法中语言模型和多模态融合模块的性能，我们采用门控反馈策略加深长-短时记忆网络的层数。实验结果表明，该算法在图像文本描述任务中表现优异。 (3)基于属性及注意力机制的视觉到语言任务算法。为了减轻图像信息和自然语言间的跨模态语义鸿沟问题，该算法将图像属性信息作为图像与语言间的“桥梁”。该算法主要包含两个层级的注意力网络——基于语义指导的注意力网络和基于文本指导的注意力网络。前者用于突出和图像属性相关的区域以及和图像区域相关的属性，后者用于寻找自然语句和图像局部之间的映射关系。该算法有两个分支，分别用于图像文本描述和视觉问答任务。相关实验也在这两类数据集上进行，实验结果表明，该算法提高了图像文本描述和视觉问答的精度。 关键词：图像文本描述，视觉问答，多模态学习，递归神经网络

 Imagery content learning and understanding is an important part of artificial intelligence. On the one hand, taking imagery as research objects, the computer system automatically obtains the content information expressed by the image by using computer programs and artificial intelligence methods to study the types of objects contained in the image, mine the target's attributes and the relationship between the objects. On the other hand, natural language is an important tool for human communication, which is also an important symbol of intelligence. Therefore, we also hope that computers can not only learn and understand the content of imagery, but also be able to describe, reason and answer questions exactly with natural language just like human beings. This is consistent with a dream in the artificial intelligence field---let computers understand the rich visual world around can communicate with us using natural language. Imagery content learning and understanding has become an important interdisciplinary subject in the field of artificial intelligence and computer vision, and it has naturally been widely studied by researchers. This dissertation aims at this topic and mainly focuses on two subtasks—image captioning and visual question answering tasks, which connect the imagery content with the human language. This is also a step that must be taken to move towards real artificial intelligence. This dissertation mainly studies the methods to accomplish the above two tasks. The main contents and contributions are summed up as follows: (1)Multi-modal gated recurrent unit for image caption generation. The traditional non-deep learning algorithms for image captioning have some shourtages: 1) the descriptions genrated by these traditional methods habe a fixed length, 2) the sentences are less variety and 3) th sentences generated by these methods cannot describe the images very accurate. The proposed methods uses the powerful and popular deep neural network to realize the image captioning. First, the image is encoded by a deep convolutional neural network which can extract more discriminative and expressive image global features. The gated recurrent unit is used as a multimodal embedding and sentence generator module. This model can not only generate length variable, rich style natural sentences, but also can fully exploit the multi-modal mapping relationship between natural language and images. Furthermore, to better simulate the nonlinear relationship between images and natural language, we increase the depth of recurrent units. The algorithm is validated on the three main image captioning datasets. The experimental results show that the algorithm can well realize the "translation" from image to text. (2)3G structure for image captioning. As the image captioning algorithms based on global image features can only learn the multi-modal mapping relationship between the entire image and the entire description statement, it is clear that this kind of mapping too rough, we need to further find the fine-grained mapping relationship between the local regions of the image and the natural language elements. The proposed algorithm uses the visual attention mechanism to mine this correspondence, the generated word is used to select the image regions at each time-step. At the same time, the existing methods of using attention mechanisms only use image local features, but abandon the global features of the image. However, the global features of the image contain global information of the image, which is an important supplement to the local features. Furthermore, the local features will be suffer from the issue of object scaling. The proposed algorithm fuses the global and local image features through a multimodal fusion module. In addition, in order to further improve the performance of the multi-modal fusion and language module in the existing methods, we have used the gated feedback strategy to increase the deepth of the long-short term memory network. The experimental results show that the proposed algorithm performs well in image captioning task. (3)Vision-to-language tasks based on attributes and attention mechanism. In order to alleviate the problem of cross-modal semantic gap between image and natural language, the proposed algorithm uses image attribute information as a "bridge" between image and language. The algorithm mainly consists of two levels of attention network---semantic-guided attention network and text-guided attention network. The former is used to highlight the regions related to image attributes and the attributes related to the image regions. The latter is used to find the mapping relationship between natural sentences and image parts. The algorithm has two branches, which are used for image captioning and visual question answering tasks, respectively. The related experiments are also performed on these two types of datasets. The results show that the algorithm improves the accuracy of image captioning and visual question answering. Key Words: Image Captioning, Visual Question Answering, Multi-modal Learning, Recurrent Neural Network