强化学习是解决顺序决策问题的一类重要的机器学习技术，经过几十年的发展已经成功应用于自动控制、机器人、推荐和检索等诸多领域。近年来，使用迁移学习方法来解决强化学习任务的研究表明，从源任务中学到的知识可以用来更好地解决类似的目标任务。然而，当进行迁移学习的源任务与目标任务不相似时，会导致负迁移的发生，但是很少有研究关注如何防止负迁移，因此该问题仍然是一个开放性的问题。目前大部分迁移学习方法假设相似的源任务是由人来选择的；只有少量方法基于任务相似性度量选择最相似的源任务，但是这些方法往往都有比较严格的前提条件；此外，目前还没有一种明确的方法可以根据一个或多个指标确定负迁移何时发生。本文针对强化学习迁移中的负迁移问题，研究如何选择合适的源任务，从不同角度提出了相应的解决方法。其主要贡献可以概括如下： 1.针对强化学习迁移中如何选择合适的源任务这一问题，提出了两种新颖的基于整体模型的马尔可夫决策过程(Markov Decision Process，MDP)间的距离度量方法。具体的，(1)这两种度量都是基于状态间距离的。为此，本文提出了同构的MDP(Homogeneous MDP)的概念以及计算同构MDP的状态间距离的方法；(2)这两种度量方法分别使用康托洛维奇度量(Kantorovich Metric)和豪斯多夫度量(Hausdorff Metric)来组合两个MDP的状态间的距离，以此作为两个MDP间的距离。这两种度量能够用于强化学习迁移中，根据任务间的距离选择合适的源任务。此外，本文还提出了两种相应的值函数迁移方法。在通用的实验场景上的实验结果表明，本文提出的度量能够有效地找到相似任务，避免负迁移；同时，所提出的度量和迁移方法能够显著提升算法的学习性能。 2.针对强化学习迁移中如何判断迁移学习是否有效这一问题，提出了一种基于深度神经网络模型的识别一对迁移任务的正负迁移性能的方法。本文研究一类视频强化学习任务(Video Reinforcement Learning Tasks)，该类任务的特征可由其任务描述图像表示，不同任务间的相关性(或差异)能够体现在图像中。在这类任务上，将预测迁移性能(正迁移或负迁移)的问题形式化为二分类问题；然后采用一个深度卷积神经网络自动学习两个任务的高层特征表示，并用一个全连接网络预测其迁移性能。在通用的实验场景上构造数据集并进行实验，结果表明，本文所提出的方法可以准确地预测迁移性能，并且显著优于对比方法。 3.针对课程学习中如何构造合适的课程这一问题，提出了一类基于源任务自动创建和任务相似度的构建迁移学习任务序列的方法。其主要创新点有：(1)基于面向对象的强化学习任务表示(Object-Oriented Representation)，提出了三种简化任务的算子，用于修改目标任务生成源任务集合。(2)针对使用不同算子修改的任务，提出了相应的任务相似性度量方法，这些度量都是基于面向对象的表示，根据对象、状态等的相似性和差异性进行定义；同时，结合任务的难易程度定义了任务迁移潜能。(3)提出了两种基于迁移潜能的任务序列自动构造方法。在通用的实验场景上的实验结果表明，本文所提出的方法能够构造较好的任务序列，显著提升目标任务的学习速度，并且优于现有的最新算法。 关键词：源任务选择，强化学习迁移，距离度量，迁移性能预测，源任务生成，课程学习

 Reinforcement learning (RL) is an important machine learning technology for solving sequential decision-making problems. After decades of development, it has been successfully applied to many fields such as automatic control, robot, recommendation systems and information retrieval. Recent research in using transfer learning methods to solve RL tasks has shown that knowledge learned from one source task may be reused to solve a similar target task better. However, when a source task is dissimilar to the target task, it will lead to negative transfer. But few studies have focused on how to avoid negative transfer. Most of the existing transfer learning methods assume that similar source tasks have been selected by humans; only few methods choose the most similar source tasks based on task similarity measures, but these methods often have strict preconditions; moreover, there is no well-defined method that can determine when negative transfer may occur. This paper studies how to choose the appropriate source task and proposes corresponding solutions from different perspectives to avoid negative transfer. The main contributions can be summarized as follows: 1.We propose two novel metrics for measuring the distance between two Markov decision processes (MDPs) based on their whole models. Specifically, (1) the two metrics are based on the distance between states. We define the notion of homogeneous MDPs and propose a method for computing the distances between states in different MDPs. (2) After computing all the distances between states, we apply the Kantorovich Metric and the Hausdorff Metric, respectively, to composite them to compute the distance between two MDPs. The two metrics can be used to select the appropriate source tasks according to the distances between tasks for transfer learning in reinforcement learning, and we propose two transfer methods which transfer value functions of selected source tasks to the target task. Experimental results on a benchmark show that our metrics are effective in finding similar tasks to avoid negative transfer in transfer learning, and significantly improve the performance of the baseline algorithms with our transfer methods. 2.We propose a method, which employs a deep neural network to identify the positive/negative transfer performance of a pair of transfer learning tasks, to solve the problem “to transfer or not to transfer” for video RL tasks. The task descriptions of video RL tasks can be represented as images, and the relateness between such tasks can be reflected in the images. Specifically, we formalise the transfer performance prediction problem as a binary classification problem, and adopt a siamese convolutional neural network (CNN) to learn task features from the images and a fully connected network to predict whether transfer is useful between a pair of tasks. The experimental results on benchmarks show that our method can accurately predict the transfer performance and significantly outperform the baseline methods as well as a method which is most related to ours. 3.To construct an appropriate curricula for curriculum learning, we propose new methods to construct task sequences for transfer learning based on automatic source task creation and task similarity measures. The main contributions are as follows: (1) We propose three operators to modify the target task to generate source task sets based on the object-oriented representation of RL tasks. (2)For tasks modified by different operators, the corresponding task similarity measures are proposed. These measures are based on the object-oriented representation and are defined according to the similarities and differences between objects, states and etc. We also define transfer potential considering both similarity between tasks and difficulty of tasks. (3) We propose automatic curriculum construction methods based on the transfer potential measures. Experimental results on a benchmark show that the proposed methods can construct good task sequences, significantly improve the learning speed in the target task, and are superior to the latest existing method. Keywords: Source Task Selection, Transfer Learning in Reinforcement Learning, Distance Metrics, Transfer Performance Prediction, Source Task Creation, Curriculum Learning