关于

👨‍🎓关于我

Hello, 你好啊！欢迎来到我的个人博客网页👋我是郑立，在这里你可以找到一些关于我的信息。目前在中国科学技术大学精密机械与精密仪器系在读博士，我的研究重点是机器人操作(Robot Manipulate)和深度强化学习(DRL)。

📝个人资料

姓名： 郑立

毕业院校： 中国科学技术大学

出生年月： 1998-11

电话： 15255195207

籍贯： 安徽六安

邮箱： zlsy@mail.ustc.edu.cn

专业： 电子信息

期望工作地： 合肥、杭州、苏州

🎓教育背景

2022.09 - 至今	中国科学技术大学	电子信息	博士
2020.09 - 2022.06	中国科学技术大学	仪器科学与技术	硕士
2016.09 - 2020.06	大连理工大学	海洋技术	本科

🔬科学研究

☞ 研究 1 一种基于高效收敛深度强化学习的动态环境下机械手轨迹规划方法

1.一种基于高效收敛深度强化学习的动态环境下机械手轨迹规划方法

An Efficiently Convergent Deep Reinforcement Learning-Based Trajectory Planning Method for Manipulators in Dynamic Environments 🔗论文链接 📃PDF 📊海报

Abstract

Recently, deep reinforcement learning (DRL)-based trajectory planning methods have been designed for manipulator trajectory planning, given their potential in solving the problem of multidimensional spatial trajectory planning. However, many DRL models that have been proposed for manipulators working in dynamic environments face difficulties in obtaining the optimal strategy, thereby preventing them from reaching convergence because of massive ineffective exploration and sparse rewards. In this paper, we solve the inefficient convergence problem at the two levels of the action selection strategy and reward functions. First, this paper designs a dynamic action selection strategy that has a high probability of providing positive samples in the pre-training period by using a variable guide item and effectively reduces invalid exploration. Second, this study proposes a combinatorial reward function that combines the artificial potential field method with a time-energy function, thereby greatly improving the efficiency and stability of DRL-based methods for manipulators trajectory planning in dynamic working environments. Extensive experiments are conducted using the *CoppeliaSim* simulation model with a freely moving obstacle and the 6-DOF manipulator. The results show that the proposed dynamic action selection strategy and combinatorial reward function can improve the convergence rate on the DDPG, TD3, and SAC DRL algorithms by up to 3-5 times. Furthermore, the mean value of the reward function increases by up to 1.47-2.70 times, and the standard deviation decreases by 27.56% to 56.60%.

摘要

最近，基于深度强化学习（DRL）的轨迹规划方法被设计用于机械手轨迹规划，因为它们在解决多维空间轨迹规划问题方面具有潜力。然而，许多为在动态环境中工作的操纵者提出的DRL模型在获得最优策略方面面临困难，从而由于大量无效的探索和稀疏的奖励而阻止它们达到收敛。在本文中，我们解决了行动选择策略和奖励函数两个层面的低效收敛问题。首先，设计了一种动态动作选择策略，利用变量引导项在训练前提供正样本的概率较高，有效减少了无效探索;其次，提出一种将人工势场法与时能函数相结合的组合奖励函数，大大提高了基于DRL的机械手在动态工作环境下轨迹规划方法的效率和稳定性。使用CoppeliaSim模拟模型和自由移动的障碍物和6自由度机械手进行了广泛的实验。结果表明，所提出的动态动作选择策略和组合奖励函数能够将DDPG、TD3和SAC DRL算法的收敛率提高3-5倍。此外，奖励函数的平均值增加了1.47-2.70倍，标准差降低了27.56%至56.60%。

切换语言

< >

☞ 研究 2 基于深度视觉引导和深度强化学习算法的配电带电作业机器人多钉孔装配任务

2.基于深度视觉引导和深度强化学习算法的配电带电作业机器人多钉孔装配任务

Deep Visual-guided and Deep Reinforcement Learning Algorithm Based for Multip-Peg-in-Hole Assembly Task of Power Distribution Live-line Operation Robot 🔗论文链接 📰PDF 📊海报

Abstract

The inspection and maintenance of power distribution network are crucial for efficiently delivering electricity to consumers. Due to the high voltage of power distribution network lines, manual live-line operations are difficult, risky, and inefficient. This paper researches a Power Distribution Network Live-line Operation Robot (PDLOR) with autonomous tool assembly capabilities to replace humans in various high-risk electrical maintenance tasks. To address the challenges of tool assembly in dynamic and unstructured work environments for PDLOR, we propose a framework consisting of deep visual-guided coarse localization and prior knowledge and fuzzy logic driven deep deterministic policy gradient (PKFD-DPG) high-precision assembly algorithm. First, we propose a multiscale identification and localization network based on YOLOv5, which enables the peg-hole close quickly and reduces ineffective exploration. Second, we design a main-auxiliary combined reward system, where the main-line reward uses the hindsight experience replay mechanism, and the auxiliary reward is based on fuzzy logic inference mechanism, addressing ineffective exploration and sparse reward in the learning process. In addition, we validate the effectiveness and advantages of the proposed algorithm through simulations and physical experiments, and also compare its performance with other assembly algorithms. The experimental results show that, for single-tool assembly tasks, the success rate of PKFD-DPG is 15.2% higher than the DDPG with functionized reward functions and 51.7% higher than the PD force control method; for multip-tools assembly tasks, the success rate of PKFD-DPG method is 17% and 53.4% higher than the other methods.

摘要

配电网的检查和维护对有效地向用户供电至关重要。配电网线路电压高，人工带电操作难度大、风险大、效率低。本文研究了一种具有自动化工具装配能力的配电网活线操作机器人(PDLOR)，以取代人类完成各种高风险的电气维护任务。为了解决PDLOR在动态和非结构化工作环境下的刀具装配问题，提出了一种由深度视觉引导的粗定位和先验知识与模糊逻辑驱动的深度确定性策略梯度(PKFD-DPG)高精度装配算法组成的框架。首先，我们提出了一种基于YOLOv5的多尺度识别和定位网络，使钉孔快速闭合，减少无效勘探。其次，我们设计了一个主辅结合的奖励系统，其中主线奖励采用后见之明的经验重播机制，辅助奖励基于模糊逻辑推理机制，解决了学习过程中无效探索和奖励稀疏的问题。此外，通过仿真和物理实验验证了所提算法的有效性和优越性，并与其他装配算法进行了性能比较。实验结果表明，对于单刀具装配任务，PKFD-DPG的成功率比带功能化奖励函数的DDPG高15.2%，比PD力控制方法高51.7%;对于多工具装配任务，PKFD-DPG方法的成功率分别为17%和53.4%。

切换语言

< >

☞ 研究 3 稀疏奖励机器人操作任务中贝叶斯强化学习的不确定性

3.稀疏奖励机器人操作任务中贝叶斯强化学习的不确定性

Uncertainty in Bayesian Reinforcement Learning for Robot Manipulation Tasks with Sparse Rewards 🔗论文链接 📰PDF 📊海报

Abstract

This paper aims to explore the application of Bayesian deep reinforcement learning (BDRL) in robot manipulation tasks with sparse rewards, focusing on addressing the uncertainty in complex and sparsely rewarded environments. Conventional deep reinforcement learning (DRL) algorithms still face significant challenges in the context of robot manipulation tasks. To address this issue, this paper proposes a general algorithm framework called BDRL that combines reinforcement learning algorithms with Bayesian networks to quantify the model uncertainty, aleatoric uncertainty in neural networks, and uncertainty in the reward function. The effectiveness and generality of the proposed algorithm are validated through simulation experiments on multiple sets of different sparsely rewarded tasks, employing various advanced DRL algorithms. The research results demonstrate that the DRL algorithm based on the Bayesian network mechanism significantly improves the convergence speed of the algorithms in sparse reward tasks by accurately estimating the model uncertainty.

摘要

本研究旨在探讨贝叶斯深度强化学习(BDRL)在稀疏奖励机器人操作任务中的应用，重点解决复杂和稀疏奖励环境下的不确定性问题。传统的深度强化学习(DRL)算法在机器人操作任务中仍然面临重大挑战。为了解决这一问题，本文提出了一种称为BDRL的通用算法框架，该框架将强化学习算法与贝叶斯网络相结合，以量化模型不确定性、神经网络中的任意不确定性和奖励函数中的不确定性。采用多种先进的DRL算法，通过多组不同稀疏奖励任务的仿真实验，验证了该算法的有效性和通用性。研究结果表明，基于贝叶斯网络机制的DRL算法通过对模型不确定性的准确估计，显著提高了算法在稀疏奖励任务中的收敛速度。

切换语言

< >

⚙️工程项目

配网带电作业机器人的人机智能融合控制技术及带电接火作业工具应用研究 📰新闻链接

针对配网带电作业的环境复杂性和任务多样性难题，研究持重臂和灵巧臂结合的轻量化双臂协作机器人机构及其控制技术，研制绝缘作业工具及其快换装置。研究作业现场环境立体感知与场景模型三维重构，构建机器人作业虚拟仿真平台，开发基于视觉-力反馈融合的双臂协作机器人临场感遥操作系统。研究作业目标精准识别与锁定技术，研究在给定约束条件下以任务为导向的机器学习训练方法，实现机器人自主避障导航规划和部分自主作业功能。研制出配网带电作业机器人系统，实现 10kV 配网线路带电拆接引、导线清障、更换跌落式熔断器、安装验电接地环、安装故障指示器、修剪树枝等不少于 6 种典型配网带电作业项目，并开展实际场景应用验证。

< >

🛠️技能技术

• 编程语言： Python， C/C++， C#，Lua， Matlab
• 仿真计算： CoppeliaSim，Isaac Gym, Unity3D，Mujoco，ROS，MATLAB ToolBox，SOLIDWORKS，COMSOL 等
• 硬件平台： 掌握移动式、协作机械臂机器人搭建、控制
• 算法框架： 掌握Pytorch，TensorFlow深度学习框架使用，掌握ElegantRL、Tianshou、Stable-baselines3等多种深度强化学习算法库使用
• 英语能力： 英语 CET-6，具有较强的文献调研能力和写作功底
• 其他能力： 高中数学教师资格证、普通话二乙证等

🏆奖励荣誉

• 3次国家励志奖学金和5次硕博一等学业奖学金
• 大连理工大学 2020 年优秀毕业生
• 大连市高等数学竞赛奖
• 中国科学技术大学 2022 年“中国科大庆峰杯”优秀双创作品奖
• 第八届安徽省“互联网+”大学生创新创业大赛高教主赛道银奖
• 中国科学技术大学“科技活动周”优秀科普工作者
• 中科大本科生《机械设计》课程优秀助教

🎨爱好乐趣

• 做饭： 红烧肉，地锅鸡，火锅鸡，豆角焖面，黄豆焖猪蹄，麻辣小龙虾，烤鱼，各式火锅等等