Detecting Abnormal Behaviors of Workers at Ship Working Fields via Asynchronous Interaction Aggregation Network
-
摘要: 港船作业区域人员的异常行为识别可为智能航运的管控与决策提供重要数据支撑，有利于推动智慧港口和智能船舶的发展。基于异步交互聚合网络开展了面向港船工作环境下的人员异常行为识别研究。基于YOLO模型对港船图像进行卷积操作，利用特征金字塔优化卷积结果得到图像序列中每一帧的人员位置，结合联合学习检测和嵌入范式输出港船图像序列中的人、物体特征信息以及时序信息；利用异步交互聚合网络中的交互聚合结构更新特征池的多维度特征信息，以识别港区与船舶工作环境下的人员异常行为。实验结果表明：提出的港船作业区域人员异常行为识别方法的平均识别精度为91%，在港区工作环境下的人员异常行为识别精度为85%，在船舶驾驶台环境下，提出的异常行为识别框架对船员的不安全行为识别精度达到97%。所提出的识别框架在不同港船作业区域环境中都能获得较好的精度，验证了其有效性和可靠性。Abstract: Identify abnormal behaviors of workers at ship working fields provides important information for intelligent shipping management and decision-making, which is conducive to promoting the development of smart ports and intelligent ships. To achieve this, an abnormal behavior recognition framework is proposed based on a novel asynchronous interaction aggregation (AIA) model. The proposed model implements the convolution operation on the maritime surveillance videos by using the YOLO algorithm. The convolution results are optimized using the feature pyramid to locate the human in each image. A method of joint learning of detection and an embedding model are then integrated to extract the spatial-temporal features of workers and objects. Furthermore, the proposed AIA model utilizes an interaction aggregation module that update multi-dimensional feature information in the feature pool to detect abnormal behaviors of workers at ship working fields. The results show that the average recognition accuracy of the proposed method is 91%, and the recognition accuracy is 85% at the working fields. For the ship bridge monitoring, the recognition accuracy of unsafe behaviors of crews can reach up to 97%. Based on its validity and reliability, the proposed framework can achieve good accuracy in a variety of ship working fields.
-
图 1 港船环境下人员异常行为识别流程图
Figure 1. Flow chart for detecting abnormal behaviors of workers at ship working fields
图 3 不同场景下的异常行为动作识别效果图
Figure 3. The proposed framework performance on recognizing abnormal behavior from different video clips
图 4 SlowFast算法对视频3序列识别结果
Figure 4. The recognition results of SlowFast for video 3 clips
表 1 港船人员异常行为的视频片段信息
Table 1. Details for the collected video clips involved with anomaly behavior
视频序列 帧率/ (帧/s) 分辨率 时长/s 实验场景 1 25 926x522 10 港口环境, 名工作人员 2 25 814x458 10 港口环境, 多名工作人员 3 25 720x480 7 复杂港口环境, 多名工作人员 4 25 704x576 10 船舶驾驶台，多名工作人员 
下载: 导出CSV
表 2 视频1序列的异常行为识别结果
Table 2. Abnormal behavior recognition results for video 1 clips
单位: % 人员 J1 J2 J3 J4 #1 100 97 100 76
下载: 导出CSV
表 3 视频2序列的异常行为识别结果
Table 3. Abnormal behavior recognition results for video 2 clips
单位: % 人员 J1 J2 J3 J4 #1 100 97 #2 88 100 100 #3 79 100 100 90
下载: 导出CSV
表 4 视频3序列的异常行为识别结果
Table 4. Abnormal behavior recognition results for video 3 clips
单位: % 人员 J1 J2 J5 J3 #1 97 97 #2 75 75 #3 89 100 72 #4 87 97 99 100 #5 62 62 #6 70 70 69
下载: 导出CSV
表 5 视频4序列的异常行为识别结果
Table 5. Abnormal behavior recognition results for video 4 clips
单位: % 人员 J1 J2 J5 J6 #1 100 97 #2 100 86 100 #3 100 89 100 #4 100 100
下载: 导出CSV
-
[1] 张永锋, 龚建伟, 殷明. 新冠肺炎疫情对中国港航业的影响及其对策[J]. 交通运输工程学报, 2020, 20(3): 159-167. https://www.cnki.com.cn/Article/CJFDTOTAL-JYGC202003019.htmZHANG Y F, GONG J W, YIN M. Influences and response measures of COVID-19 epidemic on shipping and port industry in china[J]. Journal of Traffic and Transportation Engineering, 2020, 20(3): 159-167. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JYGC202003019.htm [2] 周翔宇, 吴兆麟, 王凤武, 等. 自主船舶的定义及其自主水平的界定[J]. 交通运输工程学报, 2019, 19(6): 149-162. https://www.cnki.com.cn/Article/CJFDTOTAL-JYGC201906016.htmZHOU X Y, WU Z L, WANG F W, et al. Definition of autonomous ship and its autonomy level[J]. Journal of Traffic and Transportation Engineering, 2019, 19(6): 149-162. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JYGC201906016.htm [3] 郑松, 吴晓林, 王飞跃, 等. 平行系统方法在自动化集装箱码头中的应用研究[J]. 自动化学报, 2019, 45(3): 490-504. https://www.cnki.com.cn/Article/CJFDTOTAL-MOTO201903005.htmZHENG S, WU X L, WANG F Y, et al. Applying the parallel systems approach to automatic container terminal[J]. Acta Automatica Sinica, 2019, 45(3): 490-504(. in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-MOTO201903005.htm [4] ANDREA Z, JACOPO C, RICCARDO V, et al. Predicting intentions from motion: the subject-adversarial adaptation approach[J]. International Journal of Computer Vision, 2020, 128(1): 220-239. doi: 10.1007/s11263-019-01234-9 [5] WANG H, KLÄSER A, SCHMID C, et al. Dense trajectories and motion boundary descriptors for action recognition[J]. International Journal of Computer Vision, 2013, 103 (1): 60-79. doi: 10.1007/s11263-012-0594-8 [6] 秦宇龙, 王永雄, 胡川飞, 等. 结合注意力与多尺度时空信息的行为识别算法[J]. 小型微型计算机系统, 2021, 42(9): 1802-1809. doi: 10.3969/j.issn.1000-1220.2021.09.002QIN Y L, WANG Y X, HU C F, et al. Action recognition algorithm based on attention and multiscale channels separation spatiotemporal information[J]. Journal of Chinese Computer Systems, 2021, 42(9): 1802-1809. (in Chinese) doi: 10.3969/j.issn.1000-1220.2021.09.002 [7] YANG L H, LIU J, WANG Y M, et al. Online updating extended belief rule-based system for sensor-based activity recognition[J]. Expert Systems with Applications, 2021(186): 1-14. [8] 谭等泰, 李世超, 常文文, 等. 多特征融合的行为识别模型[J]. 中国图像图形学报, 2020, 25(12): 2541-2552. https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB202012007.htmTAN D T, LI S C, CHANG W W, et al. Multi-feature fusion behavior recognition model[J]. Journal of Image and Graphics, 2020, 25(12): 2541-2552. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB202012007.htm [9] ALWANDO E H P, CHEN Y T, FANG W H. CNN-Based multiple path search for action tube detection in videos[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(1): 104-116. doi: 10.1109/TCSVT.2018.2887283 [10] SHU X B, ZHANG L Y, SUN Y L, et al. Host-Parasite: graph LSTM-in-LSTM for group activity recognition[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 32(2): 663-674. doi: 10.1109/TNNLS.2020.2978942 [11] WANG P C, LI W Q, GAO Z M, et al. Action recognition from depth maps using deep convolutional neural networks[J]. IEEE Transactions on Human-Machine Systems, 2016, 46(4): 498-509. doi: 10.1109/THMS.2015.2504550 [12] 陈影玉, 杨神化, 索永峰. 船舶行为异常检测研究进展[J]. 交通信息与安全, 2020, 38(5): 1-11. doi: 10.3963/j.jssn.1674-4861.2020.05.001CHEN Y Y, YANG S H, SUO Y F. Research progress of ship behavior anomaly detection[J]. Journal of Transport Information and Safety, 2020, 38(5): 1-11(. in Chinese) doi: 10.3963/j.jssn.1674-4861.2020.05.001 [13] CHEN X Q, QI L, YANG Y S, et al. Video-based detection infrastructure enhancement for automated ship recognition and behavior analysis[J]. Journal of Advanced Transportation, 2020(2020): 1-12. [14] WEI Z K, XIE X L, ZHANG X J. AIS trajectory simplification algorithm considering ship behaviours[J]. Ocean Engineering, 2020(216): 1-10. [15] XUE J, CHEN Z J, PAPADIMITRIOU E, et al. Influence of environmental factors on human-like decision-making for intelligent ship[J]. Ocean Engineering, 2019(186): 1-14. [16] TANG J J, XIA J, MU X Z, et al. Asynchronous interaction aggregation for action detection[C]. European Conference on Computer Vision-ECCV 2020, Lecture Notes in Computer Science, Glasgow, UK: Springer, 2020. [17] CIPOLLA R, GAL Y, KENDALL A. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics[C]. 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA: IEEE, 2018. [18] GU C H, SUN C, ROSS D A, et al. AVA: A video dataset of spatio-temporally localized atomic visual actions[C]. 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA: IEEE, 2018. [19] FEICHTENHOFER C, FAN H Q, MALIK J, et al. SlowFast networks for video recognition[C]. 2019 International Conference on Computer Vision (ICCV), Seoul, Korea: IEEE, 2019. -
点击查看大图
计量
- 文章访问数: 1284
- HTML全文浏览量: 616
- PDF下载量: 92
- 被引次数: 0