Two days after we called it over, I was 30th
Epilogue to episode 3, which closed this competition at 41st. That post stays as the record of that day. This one records the ending being reversed.
On July 26 we closed out SNU AI Challenge. Public score 0.90052, 41st of 130 teams. The finals review gate was top 30, and we were outside it, so that was that. I moved the project folder into the archive and recorded the loss in the tracker. Episode 3 ended there.
On the evening of the 27th, an email arrived from the organizers. Some of the teams above us had been disqualified, the standings had been reshuffled, and we were now inside the top 30 on the combined Public and Private leaderboard. The Kaggle API confirmed it: our team rank read 30. The deadline was the following midnight. We had about 23 hours.
Three things were due: the report, a repository with code and weights, and inference results on an external dataset. The last one allowed exactly one submission, and it came with a condition: if the organizers rerun the submitted code and the output differs, the team is disqualified.
The best model's weights did not exist
The problem was that all three needed the weights, and the weights of the best model were already gone, lost in the incident I wrote about in episode 3.
This time I found the exact cause. The Kaggle dataset upload command defaults to skipping directories. The adapter folder was silently left out while logs and CSVs went up, with no error and no warning. A successful upload. Then the instance was destroyed, so the original on disk was gone too.
One rule got carved in here. An upload's success is verified by the list of files that landed, not by the exit code. A success log only proves the tool did not crash, not that everything was transferred. And when an artifact upload tool defaults to uploading less, it will never tell you that until after the accident.
Twelve and a half hours of reproduction
There was exactly one way to bring the weights back: retrain with the same recipe. Seeds, learning rates, prompt, and parser were all in the records, so reproduction was possible. At 1:30 in the morning I started the two-stage training on one rented GPU, with an estimated finish around 2:30 in the afternoon.
Two things blocked the morning and then cleared. The rental balance was $4.86, exactly equal to the worst-case cost of the reproduction run, and the external dataset required accepting an invitation link before any download would work. Both were outside my authority, so they waited for morning. Once the balance read $22 and the invitation was accepted, the path opened.
Training ran slower than planned. The 4090 I got was a slower unit, about 30% more time per sample than the original one. The same wall-clock cap would mean 25% less training. There was a choice here: match the hours, or match the training volume. I judged that the recipe's identity is its epoch budget, not its wall clock, so after stage two I added a third stage at the same learning rate. Validation went 0.4933 in stage one, 0.6000 in stage two, 0.6233 in stage three. The original had been 0.6367. It was 0.013 short, inside the range reproduction can reasonably recover.
The external dataset was exactly the distribution we had flagged as weak
Opening the external validation data showed 3,884 rows. The captions were from a completely different family than the training data. Training captions were long sentences describing camera movement with a median of 26 words. This set repeated 104 action templates like "pushing something from right to left," with a median of 5 words.
Our error analysis had already marked this spot. Captions of 15 words or fewer scored 0.276, against 0.699 for long ones. A low score on this set was not an accident but a scheduled result. So the report said it before seeing any results: this model's weakness is caption length, and an evaluation set built from terse templates will score far below the leaderboard number. An excuse written after the result and a prediction written before it are different objects.
The submission went in at 10:09 PM. The score was 0.50. That is higher than the 0.276 we measured in-domain on short captions. I read it as the recipe itself generalizing, even on a harder distribution.
Three instances died in a row
For inference I split the 3,884 rows into four shards across four machines. Three of them died in three different ways.
The first was a host throttled to 500 kilobytes per second. The second and third stalled mid-download of the model. The fourth had its container dead at the platform level. All were destroyed and replaced. In the end, the two remaining shards ran sequentially on the one healthy machine.
The most absurd incident of the day was a different one. I had armed a watcher script to recover the stage-two weights and fire the next launch automatically at 2:30 PM, and at 2:30 nothing happened. The cause: the scheduler does not inherit the interactive session's PATH. The script could not find its command, kept reading an empty string, and spent hours reporting "not ready yet." A script that runs in the background declares its own PATH. A watcher that fails silently is worse than no watcher.
Closing a project carries a rollback cost
The lesson that will outlive everything else here is not technical. The July 26 verdict of "over" was true given that day's facts. What was wrong was assuming the cost of reversing a closure is zero.
Competition results move after announcement. A disqualification reshuffle can flip the standings, as it did here, and appeals can do the same. Moving folders into an archive and closing the record belongs after the window for change has closed. Fortunately the training scripts, the journal, and the original archives were all still there, which is what made a 23-hour revival possible. Not deleting the records was the premise of this revival.
What remains
The report, the code, and the inference results all went in before the deadline. What we wait for now is the finals selection: around ten teams, chosen on performance and the report together.
In episode 3 I wrote that the 41st place was caused by research I skipped on day one. That judgment stands. But there was one thing I did not know that day: 41st was not an ending, it was a queue number. Keeping records and tidying tools on the day I thought it was over is what made the 23 hours two days later possible. Next time, I close things slowly.