Loading…
23
My first fine-tune crashed at 3am, but the log saved me
Last Tuesday I finally got my dataset prepped for a small language model fine-tune, 2,400 examples about repair manuals. At 3 in the morning the training loss spiked to 4.7 and then everything froze. I almost scrapped the whole run, but I checked the wandb log and saw the batch size was chewing through my 8GB GPU in Austin. Dropped it from 8 to 4, restarted, and it finished clean by breakfast. Has anyone else had a loss spike that turned out to be just a memory issue instead of a model problem?
1 comments
Log in to join the discussion
Log In1 Comment
jason119d ago
Did you check your GPU memory during the run or just after it crashed? Because that spike could also be from the log writing itself if wandb was buffering a lot of data on the same card. Next time try watching nvidia-smi in a second terminal while it trains, you'll usually see the memory creep up right before things go sideways.
4