2020-07-27 20:17:48,353 - mmaction - INFO - Environment info:
------------------------------------------------------------
sys.platform: linux
Python: 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) [GCC 7.3.0]
CUDA available: True
CUDA_HOME: /mnt/lustre/share/cuda-9.0/
NVCC: Cuda compilation tools, release 9.0, V9.0.176
GPU 0,1,2,3,4,5,6,7: GeForce GTX 1080 Ti
GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4)
PyTorch: 1.1.0+cuda90_cudnn7.6.3_lms
PyTorch compiling details: PyTorch built with:
  - GCC 5.3
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v0.18.1 (Git Hash 7de7e5d02bf687f971e7668963649728356e0c20)
  - OpenMP 201307 (a.k.a. OpenMP 4.0)
  - NNPACK is enabled
  - CUDA Runtime 9.0
  - NVCC architecture flags: -gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_50,code=compute_50
  - CuDNN 7.6.3
  - Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS=  -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=True, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=True, USE_NNPACK=True, USE_OPENMP=ON, 

TorchVision: 0.2.2
OpenCV: 4.2.0
MMCV: 1.0.3
MMAction2: 0.5.0+7457e5f
------------------------------------------------------------

2020-07-27 20:17:48,354 - mmaction - INFO - Distributed training: True
2020-07-27 20:17:48,354 - mmaction - INFO - Config: /mnt/lustre/duanhaodong/mmaction2/configs/kennyloc/bmn/bmn_400x100_2x8_9e_activitynet_feature.py
# model settings
model = dict(
    type='BMN',
    temporal_dim=100,
    boundary_ratio=0.5,
    num_samples=32,
    num_samples_per_bin=3,
    feat_dim=400,
    soft_nms_alpha=0.4,
    soft_nms_low_threshold=0.5,
    soft_nms_high_threshold=0.9,
    post_process_top_k=100)
# model training and testing settings
train_cfg = None
test_cfg = dict(average_clips='score')
# dataset settings
dataset_type = 'ActivityNetDataset'
data_root = 'data/ActivityNet/feat_kenny/'
data_root_val = 'data/ActivityNet/feat_kenny/'
ann_file_train = 'data/ActivityNet/anet_anno_train.json'
ann_file_val = 'data/ActivityNet/anet_anno_val_kenny.json'
ann_file_test = 'data/ActivityNet/anet_anno_val_kenny.json'

test_pipeline = [
    dict(type='LoadLocalizationFeature'),
    dict(
        type='Collect',
        keys=['raw_feature'],
        meta_name='video_meta',
        meta_keys=[
            'video_name', 'duration_second', 'duration_frame', 'annotations',
            'feature_frame'
        ]),
    dict(type='ToTensor', keys=['raw_feature']),
]
train_pipeline = [
    dict(type='LoadLocalizationFeature'),
    dict(type='GenerateLocalizationLabels'),
    dict(
        type='Collect',
        keys=['raw_feature', 'gt_bbox'],
        meta_name='video_meta',
        meta_keys=['video_name']),
    dict(type='ToTensor', keys=['raw_feature']),
    dict(
        type='ToDataContainer',
        fields=[dict(key='gt_bbox', stack=False, cpu_only=True)])
]
val_pipeline = [
    dict(type='LoadLocalizationFeature'),
    dict(type='GenerateLocalizationLabels'),
    dict(
        type='Collect',
        keys=['raw_feature', 'gt_bbox'],
        meta_name='video_meta',
        meta_keys=[
            'video_name', 'duration_second', 'duration_frame', 'annotations',
            'feature_frame'
        ]),
    dict(type='ToTensor', keys=['raw_feature']),
    dict(
        type='ToDataContainer',
        fields=[dict(key='gt_bbox', stack=False, cpu_only=True)])
]
data = dict(
    videos_per_gpu=8,
    workers_per_gpu=8,
    train_dataloader=dict(drop_last=True),
    val_dataloader=dict(videos_per_gpu=1),
    test=dict(
        type=dataset_type,
        ann_file=ann_file_test,
        pipeline=test_pipeline,
        data_prefix=data_root_val),
    val=dict(
        type=dataset_type,
        ann_file=ann_file_val,
        pipeline=val_pipeline,
        data_prefix=data_root_val),
    train=dict(
        type=dataset_type,
        ann_file=ann_file_train,
        pipeline=train_pipeline,
        data_prefix=data_root))

# optimizer
optimizer = dict(
    type='Adam', lr=0.004, weight_decay=0.0001)  # this lr is used for 2 gpus
optimizer_config = dict(grad_clip=None)
# learning policy
lr_config = dict(policy='step', step=7)

total_epochs = 9
checkpoint_config = dict(interval=1)
evaluation = dict(interval=1, metrics=['AR@AN'])
log_config = dict(interval=50, hooks=[dict(type='TextLoggerHook')])
# runtime settings
dist_params = dict(backend='nccl')
log_level = 'INFO'
work_dir = './work_dirs/bmn_400x100_2x8_9e_activitynet_feature/'
load_from = None
resume_from = None
workflow = [('train', 1)]
output_config = dict(out=f'{work_dir}/results.json', output_format='json')

2020-07-27 20:18:00,517 - mmaction - INFO - Start running, host: duanhaodong@SH-IDC1-10-5-30-200, work_dir: /mnt/lustre/duanhaodong/mmaction2/work_dirs/bmn_400x100_2x8_9e_activitynet_feature
2020-07-27 20:18:00,518 - mmaction - INFO - workflow: [('train', 1)], max: 9 epochs
2020-07-27 20:23:33,498 - mmaction - INFO - Epoch [1][50/150]	lr: 4.000e-03, eta: 2:22:13, time: 6.564, data_time: 2.692, memory: 5438, loss: 1.7708
2020-07-27 20:27:59,152 - mmaction - INFO - Epoch [1][100/150]	lr: 4.000e-03, eta: 2:03:40, time: 5.309, data_time: 0.097, memory: 5438, loss: 1.4418
2020-07-27 20:32:32,347 - mmaction - INFO - Epoch [1][150/150]	lr: 4.000e-03, eta: 1:55:30, time: 5.454, data_time: 0.101, memory: 5438, loss: 1.4168
2020-07-27 20:32:33,526 - mmaction - INFO - Saving checkpoint at 1 epochs
2020-07-27 20:38:30,955 - mmaction - INFO - Epoch [2][50/150]	lr: 4.000e-03, eta: 1:56:45, time: 7.040, data_time: 2.736, memory: 5438, loss: 1.3885
2020-07-27 20:43:02,872 - mmaction - INFO - Epoch [2][100/150]	lr: 4.000e-03, eta: 1:49:16, time: 5.436, data_time: 0.104, memory: 5438, loss: 1.3736
2020-07-27 20:47:36,859 - mmaction - INFO - Epoch [2][150/150]	lr: 4.000e-03, eta: 1:42:54, time: 5.479, data_time: 0.106, memory: 5438, loss: 1.3744
2020-07-27 20:47:38,162 - mmaction - INFO - Saving checkpoint at 2 epochs
2020-07-27 20:53:40,394 - mmaction - INFO - Epoch [3][50/150]	lr: 4.000e-03, eta: 1:40:59, time: 7.131, data_time: 2.819, memory: 5438, loss: 1.3464
2020-07-27 20:58:11,695 - mmaction - INFO - Epoch [3][100/150]	lr: 4.000e-03, eta: 1:34:41, time: 5.428, data_time: 0.108, memory: 5438, loss: 1.3565
2020-07-27 21:02:46,338 - mmaction - INFO - Epoch [3][150/150]	lr: 4.000e-03, eta: 1:28:53, time: 5.489, data_time: 0.106, memory: 5438, loss: 1.3485
2020-07-27 21:02:48,650 - mmaction - INFO - Saving checkpoint at 3 epochs
2020-07-27 21:08:51,397 - mmaction - INFO - Epoch [4][50/150]	lr: 4.000e-03, eta: 1:25:40, time: 7.142, data_time: 2.870, memory: 5438, loss: 1.3236
2020-07-27 21:13:22,861 - mmaction - INFO - Epoch [4][100/150]	lr: 4.000e-03, eta: 1:19:52, time: 5.430, data_time: 0.107, memory: 5438, loss: 1.3349
2020-07-27 21:17:56,931 - mmaction - INFO - Epoch [4][150/150]	lr: 4.000e-03, eta: 1:14:21, time: 5.481, data_time: 0.114, memory: 5438, loss: 1.3586
2020-07-27 21:17:58,134 - mmaction - INFO - Saving checkpoint at 4 epochs
2020-07-27 21:23:58,817 - mmaction - INFO - Epoch [5][50/150]	lr: 4.000e-03, eta: 1:10:26, time: 7.102, data_time: 2.810, memory: 5438, loss: 1.3308
2020-07-27 21:28:31,113 - mmaction - INFO - Epoch [5][100/150]	lr: 4.000e-03, eta: 1:04:56, time: 5.445, data_time: 0.106, memory: 5438, loss: 1.3284
2020-07-27 21:33:05,332 - mmaction - INFO - Epoch [5][150/150]	lr: 4.000e-03, eta: 0:59:36, time: 5.485, data_time: 0.107, memory: 5438, loss: 1.3325
2020-07-27 21:33:06,862 - mmaction - INFO - Saving checkpoint at 5 epochs
2020-07-27 21:39:01,597 - mmaction - INFO - Epoch [6][50/150]	lr: 4.000e-03, eta: 0:55:13, time: 6.985, data_time: 2.660, memory: 5438, loss: 1.3141
2020-07-27 21:43:33,970 - mmaction - INFO - Epoch [6][100/150]	lr: 4.000e-03, eta: 0:49:55, time: 5.442, data_time: 0.105, memory: 5438, loss: 1.3392
2020-07-27 21:48:08,655 - mmaction - INFO - Epoch [6][150/150]	lr: 4.000e-03, eta: 0:44:43, time: 5.497, data_time: 0.110, memory: 5438, loss: 1.3223
2020-07-27 21:48:09,828 - mmaction - INFO - Saving checkpoint at 6 epochs
2020-07-27 21:54:01,430 - mmaction - INFO - Epoch [7][50/150]	lr: 4.000e-03, eta: 0:40:05, time: 6.921, data_time: 2.587, memory: 5438, loss: 1.3200
2020-07-27 21:58:34,909 - mmaction - INFO - Epoch [7][100/150]	lr: 4.000e-03, eta: 0:34:55, time: 5.469, data_time: 0.105, memory: 5438, loss: 1.3169
2020-07-27 22:03:10,070 - mmaction - INFO - Epoch [7][150/150]	lr: 4.000e-03, eta: 0:29:49, time: 5.505, data_time: 0.106, memory: 5438, loss: 1.3176
2020-07-27 22:03:12,494 - mmaction - INFO - Saving checkpoint at 7 epochs
2020-07-27 22:09:16,550 - mmaction - INFO - Epoch [8][50/150]	lr: 4.000e-04, eta: 0:25:04, time: 7.172, data_time: 2.873, memory: 5438, loss: 1.2970
2020-07-27 22:13:49,335 - mmaction - INFO - Epoch [8][100/150]	lr: 4.000e-04, eta: 0:19:58, time: 5.449, data_time: 0.104, memory: 5438, loss: 1.2834
2020-07-27 22:18:23,888 - mmaction - INFO - Epoch [8][150/150]	lr: 4.000e-04, eta: 0:14:55, time: 5.492, data_time: 0.111, memory: 5438, loss: 1.2678
2020-07-27 22:18:25,111 - mmaction - INFO - Saving checkpoint at 8 epochs
2020-07-27 22:24:22,201 - mmaction - INFO - Epoch [9][50/150]	lr: 4.000e-04, eta: 0:10:01, time: 7.031, data_time: 2.698, memory: 5438, loss: 1.2831
2020-07-27 22:28:55,120 - mmaction - INFO - Epoch [9][100/150]	lr: 4.000e-04, eta: 0:04:59, time: 5.455, data_time: 0.105, memory: 5438, loss: 1.2656
2020-07-27 22:33:27,889 - mmaction - INFO - Epoch [9][150/150]	lr: 4.000e-04, eta: 0:00:00, time: 5.457, data_time: 0.108, memory: 5438, loss: 1.2650
2020-07-27 22:33:29,149 - mmaction - INFO - Saving checkpoint at 9 epochs