lnp_ml/models/pretrain/mpnn/pretrain.log

nohup: ignoring input
2026-08-07 06:43:53.431 | INFO | lnp_ml.config:<module>:11 - PROJ_ROOT path is: /home/gongruyi/lnp_ml
2026-08-07 06:43:53.749 | INFO | __main__:main:273 - Using device: cuda | seed: 42
2026-08-07 06:43:53.749 | INFO | __main__:main:277 - Loading train data from /home/gongruyi/lnp_ml/data/processed/train_pretrain.parquet
2026-08-07 06:43:54.598 | INFO | __main__:main:281 - Loading val data from /home/gongruyi/lnp_ml/data/processed/val_pretrain.parquet
2026-08-07 06:43:54.622 | INFO | __main__:main:285 - Train samples: 8236, Val samples: 1454
2026-08-07 06:43:54.623 | INFO | __main__:main:301 - Auto-detecting MPNN ensemble from /home/gongruyi/lnp_ml/models/mpnn/all_amine_split_for_LiON
2026-08-07 06:43:54.624 | INFO | __main__:main:303 - Found 5 MPNN models
2026-08-07 06:43:54.624 | INFO | __main__:main:308 - Creating model (use_mpnn=True, use_moe=False, use_llm=False)...
2026-08-07 06:43:54.661 | INFO | __main__:main:346 - Model parameters: 3,961,386 total, 3,961,386 trainable
2026-08-07 06:43:54.663 | INFO | __main__:warmup_cache:64 - Warming up RDKit cache for 7493 unique SMILES...
Cache warmup: 100%|██████████| 30/30 [05:13<00:00, 10.44s/it]
2026-08-07 06:49:07.718 | SUCCESS | __main__:warmup_cache:70 - Cache warmup complete. Cached 7493 SMILES.
2026-08-07 06:49:07.718 | INFO | __main__:main:353 - Starting pretraining on external data (delivery only)...
2026-08-07 06:50:01.521 | INFO | __main__:pretrain:180 - Epoch 1/50 | Train Loss: 0.8207 | Val Loss: 0.7837
2026-08-07 06:50:01.535 | INFO | __main__:pretrain:196 - -> New best model (val_loss=0.7837)
2026-08-07 06:50:06.487 | INFO | __main__:pretrain:180 - Epoch 2/50 | Train Loss: 0.7038 | Val Loss: 0.7153
2026-08-07 06:50:06.504 | INFO | __main__:pretrain:196 - -> New best model (val_loss=0.7153)
2026-08-07 06:50:11.434 | INFO | __main__:pretrain:180 - Epoch 3/50 | Train Loss: 0.6420 | Val Loss: 0.7234
2026-08-07 06:50:16.402 | INFO | __main__:pretrain:180 - Epoch 4/50 | Train Loss: 0.6071 | Val Loss: 0.6935
2026-08-07 06:50:16.412 | INFO | __main__:pretrain:196 - -> New best model (val_loss=0.6935)
2026-08-07 06:50:21.421 | INFO | __main__:pretrain:180 - Epoch 5/50 | Train Loss: 0.5664 | Val Loss: 0.6741
2026-08-07 06:50:21.430 | INFO | __main__:pretrain:196 - -> New best model (val_loss=0.6741)
2026-08-07 06:50:26.315 | INFO | __main__:pretrain:180 - Epoch 6/50 | Train Loss: 0.5478 | Val Loss: 0.6921
2026-08-07 06:50:31.353 | INFO | __main__:pretrain:180 - Epoch 7/50 | Train Loss: 0.5226 | Val Loss: 0.6984
2026-08-07 06:50:36.286 | INFO | __main__:pretrain:180 - Epoch 8/50 | Train Loss: 0.5069 | Val Loss: 0.6875
2026-08-07 06:50:41.297 | INFO | __main__:pretrain:180 - Epoch 9/50 | Train Loss: 0.4883 | Val Loss: 0.6951
2026-08-07 06:50:46.293 | INFO | __main__:pretrain:180 - Epoch 10/50 | Train Loss: 0.4736 | Val Loss: 0.6682
2026-08-07 06:50:46.302 | INFO | __main__:pretrain:196 - -> New best model (val_loss=0.6682)
2026-08-07 06:50:51.252 | INFO | __main__:pretrain:180 - Epoch 11/50 | Train Loss: 0.4711 | Val Loss: 0.7055
2026-08-07 06:50:56.306 | INFO | __main__:pretrain:180 - Epoch 12/50 | Train Loss: 0.4520 | Val Loss: 0.7177
2026-08-07 06:51:01.242 | INFO | __main__:pretrain:180 - Epoch 13/50 | Train Loss: 0.4523 | Val Loss: 0.6767
2026-08-07 06:51:06.104 | INFO | __main__:pretrain:180 - Epoch 14/50 | Train Loss: 0.4501 | Val Loss: 1.1059
2026-08-07 06:51:11.134 | INFO | __main__:pretrain:180 - Epoch 15/50 | Train Loss: 0.4275 | Val Loss: 0.7007
2026-08-07 06:51:16.206 | INFO | __main__:pretrain:180 - Epoch 16/50 | Train Loss: 0.4120 | Val Loss: 0.7132
2026-08-07 06:51:21.334 | INFO | __main__:pretrain:180 - Epoch 17/50 | Train Loss: 0.3953 | Val Loss: 0.6577
2026-08-07 06:51:21.344 | INFO | __main__:pretrain:196 - -> New best model (val_loss=0.6577)
2026-08-07 06:51:26.251 | INFO | __main__:pretrain:180 - Epoch 18/50 | Train Loss: 0.3738 | Val Loss: 0.6736
2026-08-07 06:51:31.209 | INFO | __main__:pretrain:180 - Epoch 19/50 | Train Loss: 0.3710 | Val Loss: 0.7037
2026-08-07 06:51:36.091 | INFO | __main__:pretrain:180 - Epoch 20/50 | Train Loss: 0.3622 | Val Loss: 0.6836
2026-08-07 06:51:41.027 | INFO | __main__:pretrain:180 - Epoch 21/50 | Train Loss: 0.3631 | Val Loss: 0.6451
2026-08-07 06:51:41.037 | INFO | __main__:pretrain:196 - -> New best model (val_loss=0.6451)
2026-08-07 06:51:45.910 | INFO | __main__:pretrain:180 - Epoch 22/50 | Train Loss: 0.3614 | Val Loss: 0.6778
2026-08-07 06:51:50.816 | INFO | __main__:pretrain:180 - Epoch 23/50 | Train Loss: 0.3487 | Val Loss: 0.7331
2026-08-07 06:51:55.740 | INFO | __main__:pretrain:180 - Epoch 24/50 | Train Loss: 0.3468 | Val Loss: 0.6992
2026-08-07 06:52:00.645 | INFO | __main__:pretrain:180 - Epoch 25/50 | Train Loss: 0.3405 | Val Loss: 0.6879
2026-08-07 06:52:05.487 | INFO | __main__:pretrain:180 - Epoch 26/50 | Train Loss: 0.3542 | Val Loss: 0.6829
2026-08-07 06:52:10.628 | INFO | __main__:pretrain:180 - Epoch 27/50 | Train Loss: 0.3358 | Val Loss: 0.6680
2026-08-07 06:52:15.570 | INFO | __main__:pretrain:180 - Epoch 28/50 | Train Loss: 0.3220 | Val Loss: 0.7306
2026-08-07 06:52:20.457 | INFO | __main__:pretrain:180 - Epoch 29/50 | Train Loss: 0.3064 | Val Loss: 0.9027
2026-08-07 06:52:25.410 | INFO | __main__:pretrain:180 - Epoch 30/50 | Train Loss: 0.3101 | Val Loss: 0.6749
2026-08-07 06:52:30.355 | INFO | __main__:pretrain:180 - Epoch 31/50 | Train Loss: 0.3163 | Val Loss: 0.7611
2026-08-07 06:52:30.355 | INFO | __main__:pretrain:200 - Early stopping at epoch 31
2026-08-07 06:52:30.406 | SUCCESS | __main__:main:401 - Saved pretrain checkpoint to models/pretrain/mpnn/pretrain_delivery.pt
2026-08-07 06:52:30.407 | SUCCESS | __main__:main:407 - Saved pretrain history to models/pretrain/mpnn/pretrain_history.json
2026-08-07 06:52:30.973 | SUCCESS | __main__:main:416 - Saved loss curves plot to models/pretrain/mpnn/pretrain_loss_curves.png
2026-08-07 06:52:30.973 | SUCCESS | __main__:main:418 - Pretraining complete! Best val_loss: 0.6451