2020-07-28 18:29:12,564 - mmaction - INFO - Environment info: ------------------------------------------------------------ sys.platform: linux Python: 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) [GCC 7.3.0] CUDA available: True CUDA_HOME: /mnt/lustre/share/cuda-9.0/ NVCC: Cuda compilation tools, release 9.0, V9.0.176 GPU 0,1: GeForce GTX 1080 Ti GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4) PyTorch: 1.1.0+cuda90_cudnn7.6.3_lms PyTorch compiling details: PyTorch built with: - GCC 5.3 - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications - Intel(R) MKL-DNN v0.18.1 (Git Hash 7de7e5d02bf687f971e7668963649728356e0c20) - OpenMP 201307 (a.k.a. OpenMP 4.0) - NNPACK is enabled - CUDA Runtime 9.0 - NVCC architecture flags: -gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_50,code=compute_50 - CuDNN 7.6.3 - Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=True, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=True, USE_NNPACK=True, USE_OPENMP=ON, TorchVision: 0.2.2 OpenCV: 4.2.0 MMCV: 1.0.3 MMAction2: 0.5.0+7457e5f ------------------------------------------------------------ 2020-07-28 18:29:12,565 - mmaction - INFO - Distributed training: True 2020-07-28 18:29:12,565 - mmaction - INFO - Config: /mnt/lustre/duanhaodong/mmaction2/configs/kennyloc/bmn/bmn_fine.py # model settings model = dict( type='BMN', temporal_dim=100, boundary_ratio=0.5, num_samples=32, num_samples_per_bin=3, feat_dim=400, soft_nms_alpha=0.4, soft_nms_low_threshold=0.5, soft_nms_high_threshold=0.9, post_process_top_k=100) # model training and testing settings train_cfg = None test_cfg = dict(average_clips='score') # dataset settings dataset_type = 'ActivityNetDataset' data_root = 'data/ActivityNet/feat_kenny_fine/' data_root_val = 'data/ActivityNet/feat_kenny_fine/' ann_file_train = 'data/ActivityNet/anet_anno_train.json' ann_file_val = 'data/ActivityNet/anet_anno_val_kenny.json' ann_file_test = 'data/ActivityNet/anet_anno_val_kenny.json' test_pipeline = [ dict(type='LoadLocalizationFeature'), dict( type='Collect', keys=['raw_feature'], meta_name='video_meta', meta_keys=[ 'video_name', 'duration_second', 'duration_frame', 'annotations', 'feature_frame' ]), dict(type='ToTensor', keys=['raw_feature']), ] train_pipeline = [ dict(type='LoadLocalizationFeature'), dict(type='GenerateLocalizationLabels'), dict( type='Collect', keys=['raw_feature', 'gt_bbox'], meta_name='video_meta', meta_keys=['video_name']), dict(type='ToTensor', keys=['raw_feature']), dict( type='ToDataContainer', fields=[dict(key='gt_bbox', stack=False, cpu_only=True)]) ] val_pipeline = [ dict(type='LoadLocalizationFeature'), dict(type='GenerateLocalizationLabels'), dict( type='Collect', keys=['raw_feature', 'gt_bbox'], meta_name='video_meta', meta_keys=[ 'video_name', 'duration_second', 'duration_frame', 'annotations', 'feature_frame' ]), dict(type='ToTensor', keys=['raw_feature']), dict( type='ToDataContainer', fields=[dict(key='gt_bbox', stack=False, cpu_only=True)]) ] data = dict( videos_per_gpu=8, workers_per_gpu=8, train_dataloader=dict(drop_last=True), val_dataloader=dict(videos_per_gpu=1), test=dict( type=dataset_type, ann_file=ann_file_test, pipeline=test_pipeline, data_prefix=data_root_val), val=dict( type=dataset_type, ann_file=ann_file_val, pipeline=val_pipeline, data_prefix=data_root_val), train=dict( type=dataset_type, ann_file=ann_file_train, pipeline=train_pipeline, data_prefix=data_root)) # optimizer optimizer = dict( type='Adam', lr=0.001, weight_decay=0.0001) # this lr is used for 2 gpus optimizer_config = dict(grad_clip=None) # learning policy lr_config = dict(policy='step', step=7) total_epochs = 9 checkpoint_config = dict(interval=1) evaluation = dict(interval=1, metrics=['AR@AN']) log_config = dict(interval=50, hooks=[dict(type='TextLoggerHook')]) # runtime settings dist_params = dict(backend='nccl') log_level = 'INFO' work_dir = './work_dirs/bmn_fine/' load_from = None resume_from = None workflow = [('train', 1)] output_config = dict(out=f'{work_dir}/results.json', output_format='json') 2020-07-28 18:29:20,699 - mmaction - INFO - Start running, host: duanhaodong@SH-IDC1-10-5-30-146, work_dir: /mnt/lustre/duanhaodong/mmaction2/work_dirs/bmn_fine 2020-07-28 18:29:20,699 - mmaction - INFO - workflow: [('train', 1)], max: 9 epochs 2020-07-28 18:32:52,239 - mmaction - INFO - Epoch [1][50/603] lr: 1.000e-03, eta: 6:13:52, time: 4.172, data_time: 1.141, memory: 5438, loss: 1.7564 2020-07-28 18:35:27,398 - mmaction - INFO - Epoch [1][100/603] lr: 1.000e-03, eta: 5:22:56, time: 3.103, data_time: 0.060, memory: 5438, loss: 1.5422 2020-07-28 18:38:02,643 - mmaction - INFO - Epoch [1][150/603] lr: 1.000e-03, eta: 5:04:17, time: 3.105, data_time: 0.060, memory: 5438, loss: 1.5418 2020-07-28 18:40:37,828 - mmaction - INFO - Epoch [1][200/603] lr: 1.000e-03, eta: 4:53:39, time: 3.104, data_time: 0.060, memory: 5438, loss: 1.4905 2020-07-28 18:43:13,054 - mmaction - INFO - Epoch [1][250/603] lr: 1.000e-03, eta: 4:46:15, time: 3.104, data_time: 0.060, memory: 5438, loss: 1.4234 2020-07-28 18:45:48,412 - mmaction - INFO - Epoch [1][300/603] lr: 1.000e-03, eta: 4:40:29, time: 3.107, data_time: 0.060, memory: 5438, loss: 1.4812 2020-07-28 18:48:23,647 - mmaction - INFO - Epoch [1][350/603] lr: 1.000e-03, eta: 4:35:36, time: 3.105, data_time: 0.060, memory: 5438, loss: 1.3850 2020-07-28 18:50:58,843 - mmaction - INFO - Epoch [1][400/603] lr: 1.000e-03, eta: 4:31:17, time: 3.104, data_time: 0.060, memory: 5438, loss: 1.3973 2020-07-28 18:53:34,046 - mmaction - INFO - Epoch [1][450/603] lr: 1.000e-03, eta: 4:27:21, time: 3.104, data_time: 0.060, memory: 5438, loss: 1.4393 2020-07-28 18:56:09,231 - mmaction - INFO - Epoch [1][500/603] lr: 1.000e-03, eta: 4:23:41, time: 3.104, data_time: 0.060, memory: 5438, loss: 1.3944 2020-07-28 18:58:44,411 - mmaction - INFO - Epoch [1][550/603] lr: 1.000e-03, eta: 4:20:12, time: 3.103, data_time: 0.060, memory: 5438, loss: 1.3971 2020-07-28 19:01:19,781 - mmaction - INFO - Epoch [1][600/603] lr: 1.000e-03, eta: 4:16:55, time: 3.107, data_time: 0.060, memory: 5438, loss: 1.3913 2020-07-28 19:01:27,432 - mmaction - INFO - Saving checkpoint at 1 epochs 2020-07-28 19:05:07,342 - mmaction - INFO - Epoch [2][50/603] lr: 1.000e-03, eta: 4:19:39, time: 4.297, data_time: 1.259, memory: 5438, loss: 1.3898 2020-07-28 19:07:42,553 - mmaction - INFO - Epoch [2][100/603] lr: 1.000e-03, eta: 4:16:02, time: 3.104, data_time: 0.060, memory: 5438, loss: 1.3613 2020-07-28 19:10:17,960 - mmaction - INFO - Epoch [2][150/603] lr: 1.000e-03, eta: 4:12:35, time: 3.108, data_time: 0.061, memory: 5438, loss: 1.3803 2020-07-28 19:12:53,153 - mmaction - INFO - Epoch [2][200/603] lr: 1.000e-03, eta: 4:09:13, time: 3.104, data_time: 0.060, memory: 5438, loss: 1.3431 2020-07-28 19:15:28,472 - mmaction - INFO - Epoch [2][250/603] lr: 1.000e-03, eta: 4:05:57, time: 3.106, data_time: 0.060, memory: 5438, loss: 1.3448 2020-07-28 19:18:03,614 - mmaction - INFO - Epoch [2][300/603] lr: 1.000e-03, eta: 4:02:45, time: 3.103, data_time: 0.060, memory: 5438, loss: 1.3654 2020-07-28 19:20:38,839 - mmaction - INFO - Epoch [2][350/603] lr: 1.000e-03, eta: 3:59:37, time: 3.104, data_time: 0.060, memory: 5438, loss: 1.3784 2020-07-28 19:23:14,164 - mmaction - INFO - Epoch [2][400/603] lr: 1.000e-03, eta: 3:56:32, time: 3.107, data_time: 0.060, memory: 5438, loss: 1.3613 2020-07-28 19:25:49,323 - mmaction - INFO - Epoch [2][450/603] lr: 1.000e-03, eta: 3:53:30, time: 3.103, data_time: 0.060, memory: 5438, loss: 1.3733 2020-07-28 19:28:24,476 - mmaction - INFO - Epoch [2][500/603] lr: 1.000e-03, eta: 3:50:30, time: 3.103, data_time: 0.060, memory: 5438, loss: 1.3753 2020-07-28 19:30:59,657 - mmaction - INFO - Epoch [2][550/603] lr: 1.000e-03, eta: 3:47:33, time: 3.104, data_time: 0.060, memory: 5438, loss: 1.3570 2020-07-28 19:33:34,919 - mmaction - INFO - Epoch [2][600/603] lr: 1.000e-03, eta: 3:44:37, time: 3.105, data_time: 0.060, memory: 5438, loss: 1.3540 2020-07-28 19:33:41,766 - mmaction - INFO - Saving checkpoint at 2 epochs 2020-07-28 19:37:15,763 - mmaction - INFO - Epoch [3][50/603] lr: 1.000e-03, eta: 3:43:58, time: 4.163, data_time: 1.116, memory: 5438, loss: 1.3292 2020-07-28 19:39:50,963 - mmaction - INFO - Epoch [3][100/603] lr: 1.000e-03, eta: 3:40:58, time: 3.104, data_time: 0.061, memory: 5438, loss: 1.3094 2020-07-28 19:42:26,225 - mmaction - INFO - Epoch [3][150/603] lr: 1.000e-03, eta: 3:38:01, time: 3.105, data_time: 0.060, memory: 5438, loss: 1.3524 2020-07-28 19:45:01,594 - mmaction - INFO - Epoch [3][200/603] lr: 1.000e-03, eta: 3:35:05, time: 3.107, data_time: 0.060, memory: 5438, loss: 1.3223 2020-07-28 19:47:36,749 - mmaction - INFO - Epoch [3][250/603] lr: 1.000e-03, eta: 3:32:10, time: 3.103, data_time: 0.060, memory: 5438, loss: 1.3553 2020-07-28 19:50:11,933 - mmaction - INFO - Epoch [3][300/603] lr: 1.000e-03, eta: 3:29:16, time: 3.104, data_time: 0.060, memory: 5438, loss: 1.3111 2020-07-28 19:52:47,082 - mmaction - INFO - Epoch [3][350/603] lr: 1.000e-03, eta: 3:26:24, time: 3.103, data_time: 0.060, memory: 5438, loss: 1.3589 2020-07-28 19:55:22,267 - mmaction - INFO - Epoch [3][400/603] lr: 1.000e-03, eta: 3:23:32, time: 3.104, data_time: 0.060, memory: 5438, loss: 1.3144 2020-07-28 19:57:57,349 - mmaction - INFO - Epoch [3][450/603] lr: 1.000e-03, eta: 3:20:42, time: 3.102, data_time: 0.060, memory: 5438, loss: 1.3241 2020-07-28 20:00:32,664 - mmaction - INFO - Epoch [3][500/603] lr: 1.000e-03, eta: 3:17:53, time: 3.106, data_time: 0.060, memory: 5438, loss: 1.3215 2020-07-28 20:03:07,715 - mmaction - INFO - Epoch [3][550/603] lr: 1.000e-03, eta: 3:15:04, time: 3.101, data_time: 0.060, memory: 5438, loss: 1.3142 2020-07-28 20:05:42,840 - mmaction - INFO - Epoch [3][600/603] lr: 1.000e-03, eta: 3:12:16, time: 3.102, data_time: 0.060, memory: 5438, loss: 1.3574 2020-07-28 20:05:50,459 - mmaction - INFO - Saving checkpoint at 3 epochs 2020-07-28 20:09:24,434 - mmaction - INFO - Epoch [4][50/603] lr: 1.000e-03, eta: 3:10:44, time: 4.178, data_time: 1.145, memory: 5438, loss: 1.3103 2020-07-28 20:11:59,524 - mmaction - INFO - Epoch [4][100/603] lr: 1.000e-03, eta: 3:07:54, time: 3.102, data_time: 0.061, memory: 5438, loss: 1.2857 2020-07-28 20:14:34,600 - mmaction - INFO - Epoch [4][150/603] lr: 1.000e-03, eta: 3:05:04, time: 3.102, data_time: 0.060, memory: 5438, loss: 1.2955 2020-07-28 20:17:09,728 - mmaction - INFO - Epoch [4][200/603] lr: 1.000e-03, eta: 3:02:16, time: 3.103, data_time: 0.060, memory: 5438, loss: 1.2876 2020-07-28 20:19:44,833 - mmaction - INFO - Epoch [4][250/603] lr: 1.000e-03, eta: 2:59:28, time: 3.102, data_time: 0.060, memory: 5438, loss: 1.2997 2020-07-28 20:22:19,924 - mmaction - INFO - Epoch [4][300/603] lr: 1.000e-03, eta: 2:56:40, time: 3.102, data_time: 0.060, memory: 5438, loss: 1.2822 2020-07-28 20:24:54,961 - mmaction - INFO - Epoch [4][350/603] lr: 1.000e-03, eta: 2:53:53, time: 3.101, data_time: 0.060, memory: 5438, loss: 1.3449 2020-07-28 20:27:29,993 - mmaction - INFO - Epoch [4][400/603] lr: 1.000e-03, eta: 2:51:07, time: 3.101, data_time: 0.060, memory: 5438, loss: 1.3345 2020-07-28 20:30:05,012 - mmaction - INFO - Epoch [4][450/603] lr: 1.000e-03, eta: 2:48:21, time: 3.100, data_time: 0.060, memory: 5438, loss: 1.3211 2020-07-28 20:32:40,112 - mmaction - INFO - Epoch [4][500/603] lr: 1.000e-03, eta: 2:45:36, time: 3.102, data_time: 0.060, memory: 5438, loss: 1.3150 2020-07-28 20:35:15,179 - mmaction - INFO - Epoch [4][550/603] lr: 1.000e-03, eta: 2:42:51, time: 3.101, data_time: 0.060, memory: 5438, loss: 1.3243 2020-07-28 20:37:50,260 - mmaction - INFO - Epoch [4][600/603] lr: 1.000e-03, eta: 2:40:07, time: 3.102, data_time: 0.060, memory: 5438, loss: 1.3748 2020-07-28 20:37:57,113 - mmaction - INFO - Saving checkpoint at 4 epochs 2020-07-28 20:41:29,629 - mmaction - INFO - Epoch [5][50/603] lr: 1.000e-03, eta: 2:38:04, time: 4.134, data_time: 1.102, memory: 5438, loss: 1.3040 2020-07-28 20:44:04,630 - mmaction - INFO - Epoch [5][100/603] lr: 1.000e-03, eta: 2:35:18, time: 3.100, data_time: 0.060, memory: 5438, loss: 1.2723 2020-07-28 20:46:39,649 - mmaction - INFO - Epoch [5][150/603] lr: 1.000e-03, eta: 2:32:33, time: 3.100, data_time: 0.060, memory: 5438, loss: 1.3221 2020-07-28 20:49:14,679 - mmaction - INFO - Epoch [5][200/603] lr: 1.000e-03, eta: 2:29:48, time: 3.101, data_time: 0.060, memory: 5438, loss: 1.2830 2020-07-28 20:51:49,681 - mmaction - INFO - Epoch [5][250/603] lr: 1.000e-03, eta: 2:27:03, time: 3.100, data_time: 0.060, memory: 5438, loss: 1.3019 2020-07-28 20:54:24,655 - mmaction - INFO - Epoch [5][300/603] lr: 1.000e-03, eta: 2:24:19, time: 3.100, data_time: 0.060, memory: 5438, loss: 1.3391 2020-07-28 20:56:59,666 - mmaction - INFO - Epoch [5][350/603] lr: 1.000e-03, eta: 2:21:35, time: 3.100, data_time: 0.060, memory: 5438, loss: 1.2711 2020-07-28 20:59:34,711 - mmaction - INFO - Epoch [5][400/603] lr: 1.000e-03, eta: 2:18:52, time: 3.101, data_time: 0.060, memory: 5438, loss: 1.3008 2020-07-28 21:02:09,949 - mmaction - INFO - Epoch [5][450/603] lr: 1.000e-03, eta: 2:16:09, time: 3.105, data_time: 0.060, memory: 5438, loss: 1.3098 2020-07-28 21:04:45,172 - mmaction - INFO - Epoch [5][500/603] lr: 1.000e-03, eta: 2:13:26, time: 3.104, data_time: 0.060, memory: 5438, loss: 1.3039 2020-07-28 21:07:20,175 - mmaction - INFO - Epoch [5][550/603] lr: 1.000e-03, eta: 2:10:44, time: 3.100, data_time: 0.060, memory: 5438, loss: 1.2726 2020-07-28 21:09:55,221 - mmaction - INFO - Epoch [5][600/603] lr: 1.000e-03, eta: 2:08:01, time: 3.101, data_time: 0.060, memory: 5438, loss: 1.3066 2020-07-28 21:10:02,838 - mmaction - INFO - Saving checkpoint at 5 epochs 2020-07-28 21:13:36,947 - mmaction - INFO - Epoch [6][50/603] lr: 1.000e-03, eta: 2:05:44, time: 4.181, data_time: 1.146, memory: 5438, loss: 1.2497 2020-07-28 21:16:11,898 - mmaction - INFO - Epoch [6][100/603] lr: 1.000e-03, eta: 2:03:01, time: 3.099, data_time: 0.060, memory: 5438, loss: 1.2498 2020-07-28 21:18:46,914 - mmaction - INFO - Epoch [6][150/603] lr: 1.000e-03, eta: 2:00:18, time: 3.100, data_time: 0.060, memory: 5438, loss: 1.2930 2020-07-28 21:21:21,973 - mmaction - INFO - Epoch [6][200/603] lr: 1.000e-03, eta: 1:57:35, time: 3.101, data_time: 0.061, memory: 5438, loss: 1.2989 2020-07-28 21:23:56,949 - mmaction - INFO - Epoch [6][250/603] lr: 1.000e-03, eta: 1:54:53, time: 3.099, data_time: 0.060, memory: 5438, loss: 1.2724 2020-07-28 21:26:31,939 - mmaction - INFO - Epoch [6][300/603] lr: 1.000e-03, eta: 1:52:10, time: 3.100, data_time: 0.060, memory: 5438, loss: 1.3150 2020-07-28 21:29:06,973 - mmaction - INFO - Epoch [6][350/603] lr: 1.000e-03, eta: 1:49:28, time: 3.101, data_time: 0.060, memory: 5438, loss: 1.2979 2020-07-28 21:31:41,875 - mmaction - INFO - Epoch [6][400/603] lr: 1.000e-03, eta: 1:46:47, time: 3.098, data_time: 0.060, memory: 5438, loss: 1.2783 2020-07-28 21:34:16,829 - mmaction - INFO - Epoch [6][450/603] lr: 1.000e-03, eta: 1:44:05, time: 3.099, data_time: 0.060, memory: 5438, loss: 1.3365 2020-07-28 21:36:51,789 - mmaction - INFO - Epoch [6][500/603] lr: 1.000e-03, eta: 1:41:23, time: 3.099, data_time: 0.060, memory: 5438, loss: 1.2789 2020-07-28 21:39:26,733 - mmaction - INFO - Epoch [6][550/603] lr: 1.000e-03, eta: 1:38:42, time: 3.099, data_time: 0.060, memory: 5438, loss: 1.2988 2020-07-28 21:42:01,664 - mmaction - INFO - Epoch [6][600/603] lr: 1.000e-03, eta: 1:36:01, time: 3.099, data_time: 0.060, memory: 5438, loss: 1.2849 2020-07-28 21:42:08,585 - mmaction - INFO - Saving checkpoint at 6 epochs 2020-07-28 21:45:41,784 - mmaction - INFO - Epoch [7][50/603] lr: 1.000e-03, eta: 1:33:31, time: 4.148, data_time: 1.111, memory: 5438, loss: 1.3004 2020-07-28 21:48:16,729 - mmaction - INFO - Epoch [7][100/603] lr: 1.000e-03, eta: 1:30:50, time: 3.099, data_time: 0.060, memory: 5438, loss: 1.2510 2020-07-28 21:50:51,692 - mmaction - INFO - Epoch [7][150/603] lr: 1.000e-03, eta: 1:28:08, time: 3.099, data_time: 0.060, memory: 5438, loss: 1.2650 2020-07-28 21:53:26,850 - mmaction - INFO - Epoch [7][200/603] lr: 1.000e-03, eta: 1:25:27, time: 3.103, data_time: 0.060, memory: 5438, loss: 1.3166 2020-07-28 21:56:01,811 - mmaction - INFO - Epoch [7][250/603] lr: 1.000e-03, eta: 1:22:46, time: 3.099, data_time: 0.060, memory: 5438, loss: 1.2562 2020-07-28 21:58:36,774 - mmaction - INFO - Epoch [7][300/603] lr: 1.000e-03, eta: 1:20:05, time: 3.099, data_time: 0.060, memory: 5438, loss: 1.2777 2020-07-28 22:01:12,022 - mmaction - INFO - Epoch [7][350/603] lr: 1.000e-03, eta: 1:17:24, time: 3.105, data_time: 0.060, memory: 5438, loss: 1.2819 2020-07-28 22:03:46,947 - mmaction - INFO - Epoch [7][400/603] lr: 1.000e-03, eta: 1:14:44, time: 3.099, data_time: 0.060, memory: 5438, loss: 1.3051 2020-07-28 22:06:21,855 - mmaction - INFO - Epoch [7][450/603] lr: 1.000e-03, eta: 1:12:03, time: 3.098, data_time: 0.060, memory: 5438, loss: 1.2919 2020-07-28 22:08:56,753 - mmaction - INFO - Epoch [7][500/603] lr: 1.000e-03, eta: 1:09:23, time: 3.098, data_time: 0.060, memory: 5438, loss: 1.2784 2020-07-28 22:11:31,842 - mmaction - INFO - Epoch [7][550/603] lr: 1.000e-03, eta: 1:06:42, time: 3.102, data_time: 0.060, memory: 5438, loss: 1.2908 2020-07-28 22:14:06,768 - mmaction - INFO - Epoch [7][600/603] lr: 1.000e-03, eta: 1:04:02, time: 3.099, data_time: 0.060, memory: 5438, loss: 1.2658 2020-07-28 22:14:14,424 - mmaction - INFO - Saving checkpoint at 7 epochs 2020-07-28 22:17:48,889 - mmaction - INFO - Epoch [8][50/603] lr: 1.000e-04, eta: 1:01:25, time: 4.190, data_time: 1.145, memory: 5438, loss: 1.2311 2020-07-28 22:20:23,835 - mmaction - INFO - Epoch [8][100/603] lr: 1.000e-04, eta: 0:58:44, time: 3.099, data_time: 0.060, memory: 5438, loss: 1.2593 2020-07-28 22:22:58,800 - mmaction - INFO - Epoch [8][150/603] lr: 1.000e-04, eta: 0:56:04, time: 3.099, data_time: 0.060, memory: 5438, loss: 1.2583 2020-07-28 22:25:33,674 - mmaction - INFO - Epoch [8][200/603] lr: 1.000e-04, eta: 0:53:24, time: 3.098, data_time: 0.060, memory: 5438, loss: 1.2332 2020-07-28 22:28:08,613 - mmaction - INFO - Epoch [8][250/603] lr: 1.000e-04, eta: 0:50:44, time: 3.099, data_time: 0.059, memory: 5438, loss: 1.2592 2020-07-28 22:30:43,503 - mmaction - INFO - Epoch [8][300/603] lr: 1.000e-04, eta: 0:48:03, time: 3.098, data_time: 0.059, memory: 5438, loss: 1.2278 2020-07-28 22:33:18,754 - mmaction - INFO - Epoch [8][350/603] lr: 1.000e-04, eta: 0:45:24, time: 3.105, data_time: 0.060, memory: 5438, loss: 1.2202 2020-07-28 22:35:53,937 - mmaction - INFO - Epoch [8][400/603] lr: 1.000e-04, eta: 0:42:44, time: 3.104, data_time: 0.060, memory: 5438, loss: 1.2329 2020-07-28 22:38:28,978 - mmaction - INFO - Epoch [8][450/603] lr: 1.000e-04, eta: 0:40:04, time: 3.101, data_time: 0.060, memory: 5438, loss: 1.2133 2020-07-28 22:41:04,231 - mmaction - INFO - Epoch [8][500/603] lr: 1.000e-04, eta: 0:37:24, time: 3.105, data_time: 0.059, memory: 5438, loss: 1.2135 2020-07-28 22:43:39,353 - mmaction - INFO - Epoch [8][550/603] lr: 1.000e-04, eta: 0:34:45, time: 3.102, data_time: 0.060, memory: 5438, loss: 1.2106 2020-07-28 22:46:14,465 - mmaction - INFO - Epoch [8][600/603] lr: 1.000e-04, eta: 0:32:05, time: 3.102, data_time: 0.060, memory: 5438, loss: 1.2459 2020-07-28 22:46:21,978 - mmaction - INFO - Saving checkpoint at 8 epochs 2020-07-28 22:50:02,436 - mmaction - INFO - Epoch [9][50/603] lr: 1.000e-04, eta: 0:29:22, time: 4.305, data_time: 1.275, memory: 5438, loss: 1.2062 2020-07-28 22:52:37,505 - mmaction - INFO - Epoch [9][100/603] lr: 1.000e-04, eta: 0:26:43, time: 3.101, data_time: 0.060, memory: 5438, loss: 1.2174 2020-07-28 22:55:12,585 - mmaction - INFO - Epoch [9][150/603] lr: 1.000e-04, eta: 0:24:03, time: 3.102, data_time: 0.060, memory: 5438, loss: 1.2192 2020-07-28 22:57:47,592 - mmaction - INFO - Epoch [9][200/603] lr: 1.000e-04, eta: 0:21:23, time: 3.100, data_time: 0.060, memory: 5438, loss: 1.2338 2020-07-28 23:00:22,815 - mmaction - INFO - Epoch [9][250/603] lr: 1.000e-04, eta: 0:18:44, time: 3.104, data_time: 0.060, memory: 5438, loss: 1.1760 2020-07-28 23:02:57,815 - mmaction - INFO - Epoch [9][300/603] lr: 1.000e-04, eta: 0:16:04, time: 3.100, data_time: 0.060, memory: 5438, loss: 1.2100 2020-07-28 23:05:32,785 - mmaction - INFO - Epoch [9][350/603] lr: 1.000e-04, eta: 0:13:25, time: 3.099, data_time: 0.060, memory: 5438, loss: 1.2120 2020-07-28 23:08:07,840 - mmaction - INFO - Epoch [9][400/603] lr: 1.000e-04, eta: 0:10:45, time: 3.101, data_time: 0.060, memory: 5438, loss: 1.2352 2020-07-28 23:10:42,782 - mmaction - INFO - Epoch [9][450/603] lr: 1.000e-04, eta: 0:08:06, time: 3.099, data_time: 0.060, memory: 5438, loss: 1.2350 2020-07-28 23:13:17,753 - mmaction - INFO - Epoch [9][500/603] lr: 1.000e-04, eta: 0:05:27, time: 3.098, data_time: 0.060, memory: 5438, loss: 1.1725 2020-07-28 23:15:52,943 - mmaction - INFO - Epoch [9][550/603] lr: 1.000e-04, eta: 0:02:48, time: 3.105, data_time: 0.062, memory: 5438, loss: 1.2512 2020-07-28 23:18:27,859 - mmaction - INFO - Epoch [9][600/603] lr: 1.000e-04, eta: 0:00:09, time: 3.098, data_time: 0.060, memory: 5438, loss: 1.1785 2020-07-28 23:18:35,531 - mmaction - INFO - Saving checkpoint at 9 epochs