14 days of tuning gained 0.05. Two days of research gained 0.08.

This is a postmortem of a finished competition, the third in a series that records three losses in sequence.

SNU AI Challenge 2026 was a seventeen-day campaign. It was a multimodal sequential-reasoning task run on Kaggle. I finished with a Public score of 0.90052, 41st out of 130 teams, twelve submissions, and about $176 in spending. I did not clear the top-30 gate, so I was not eligible to submit a report.

This competition differs from the first two. I did well here. My opening score was 0.138 and I ended at 0.90052. But when I look at where that gain came from, it is unsettling.

Two weeks gave me 0.05; two days gave me 0.08. I did not invent a new training method in those two days. I just searched.

For fourteen days, I did not ask the right question

The variables I was changing were learning rate, data augmentation, image resolution, and LoRA rank. I ran thirteen versions. The validation score climbed slowly from 0.51 to 0.56, then attached itself to an asymptote.

There was one question I did not ask in that stretch: “Is the base model the problem?”

I asked it on July 23. I found an open checkpoint strong for this task type, a model released with reinforcement learning already built into it. It took two days to port it, and it set the campaign high.

In the same campaign, I tried to run my own reinforcement learning five times: DPO three times and GRPO twice. None had any effect. With my available compute and time, reinforcement-learning training was not feasible.

That leaves a simple summary. I failed five times trying to build a capability, and it took two days to find an artifact where that capability was already baked in.

For a competition constrained by compute and time, this has to become a rule. For expensive capabilities such as reinforcement learning, reasoning specialization, and domain pretraining, do not reproduce them; import them. This research should also be the first question in first-day research. I did it on day sixteen. That was the biggest mistake of this campaign.

When I plateau, I change categories, not variables

I turned the tool I extracted from this failure into a table and carried it into the next competition.

Rule: once a plateau lasts more than a week, I require a weekly review one layer up. I force myself to answer four questions. Is the variable I am changing the best one, what is the category above it, what does current practice use, is this a technical problem or a resource problem, and is the answer among constants I have never touched, such as the base model, objective function, data, or evaluation method?

In a plateau, people respond by working harder. Pushing harder on a curve that has reached its asymptote does not move the asymptote. I need to change the curve.

Three near-mistakes: undertrained false negatives

In this campaign, I nearly declared “this axis does not work” three times. All three judgments would have been wrong.

1. Video input mode: validation score 0.057. On inspection, an out-of-memory error had left only 17 actual training samples. After lowering the resolution and running it fairly, it reached 0.347.
2. Vision-attention change: validation score 0.29. On inspection, it was 3.4 times slower, so it trained on 384 samples, just 0.04 epochs of the full dataset. When run properly, it reached 0.3567, tied with the control. This was a genuine no-effect result.
3. The version with a 19-hour cap: it was cut off at 0.74 epochs. I nearly confused saturated training with being behind from the start.

One rule came from this: before judging the score, inspect the training volume. I check the actual number of samples seen, the out-of-memory counter, and measured time per sample first. I run A/B tests only as matched pairs with training volume, seed, and data order aligned.

I also record the statements separately. “This axis does not work” and “I cannot properly evaluate this axis with my money and time” are different statements. The latter needs to be revisited when resources become available. If I write it as the former, I close it permanently.

Money leaked through operations, not experiments

I spent $176. When I dissected the spending, there were periods where infrastructure incidents cost more than experiments.

• A monitoring script died, leaving an instance running idle
• Time spent retrying against a dead model host
• A boot script placed in a temporary directory and failing to run three times in a row
• Choosing a latest-generation GPU architecture without a kernel image and crashing (twice)
• Choosing 40GB based on listed VRAM, only to find it was not actually available (twice)

None of those were experiments. They were operations. I attached a checklist to every competition that uses paid GPUs.

• Verify that the GPU architecture is compatible with my torch build. The newest generation may lack a kernel.
• Measure actual VRAM immediately after instance creation. Do not trust server filters or listings.
• Put boot scripts in a persistent path, then verify that they ran after boot.
• Within ten minutes of launch, check utilization and log progress. Do not run blind.
• Put the watcher at the OS level, and monitor the watcher itself.
• Treat artifact upload as the completion signal. Destroy the instance only after confirming the upload.
• Reserve a 15% reliability tax in the budget in advance. A cheap host costs more when downtime is included.

I repeated two mistakes twice. An incident I learn from once will recur unless I turn it into a filter in code.

I lost the weights of my highest-performing model

I lost the adapter for the model that set the campaign high. I destroyed the instance without verifying the upload.

The reason this hurts becomes clear in deadline week. One standard play on the final day is checkpoint averaging. Averaging the weights of the top two or three models can deliver 0.5% to 2 percentage points for about $1. That play requires the checkpoints to remain.

Preserve every checkpoint permanently. Destroy only after confirming the upload. It needs to become a habit, not a rule.

On firing only twelve submissions

While leading teams fired 20 to 40 shots, I fired twelve. I had a rule: submit only when validation score exceeded the current high by a 0.005 margin.

That looks like a weakness, but it is also the final line of defense. Overfitting to the leaderboard causes a drop during private rescoring. Treating submission slots as a tool for raising scores leads into that trap.

Submission slots are not a scoring tool; they are a measurement budget. N shots per day means “the server will score N hypotheses per day.” I do not spend them on measurements that will not change the result. But an expiring quota is a loss, so in deadline week I recover it by asking the server about candidates that validation could not decide.

That recovery worked twice. A version that failed the validation gate set a new server record. Two runs over a 300-item validation set had a deviation of 0.02. For differences within that noise, only the server can judge.

The validation score is a compass, not a court.

The campaign high came in the final 20 hours

I removed the “finish by maintaining the status quo” branch from the plan. That was my best judgment in this campaign.

Attacks that cost nothing do not stop before the deadline. Local analysis, free compute, research, and designing the next hypothesis keep running because they do not cost money. I fire paid shots one at a time when a measured signal appears. Without a signal, I stay quiet; with one, I do not hesitate.

These two rules appear to conflict. One says conserve, and the other says do not stop. In practice, they apply to different layers. Refusing to surrender is a matter of posture; economy is a matter of spending. I can conserve spending while maintaining posture.

The final 20 hours of the free lane produced checkpoint averaging and the base-model replacement. Both campaign highs came from there.

The question that stays

Of the three competitions, this was the one I did best in. Yet I finished 41st. Most of the reason for 41st was research I did not do on the first day.

If I did it again, I would spend the first day this way. I would run no training at all and spend the entire day finding the strongest public checkpoints for this task type. It would feel wasteful because the score would not rise on the first day.

The anxiety of day one usually has to be repaid as interest over the remaining sixteen days. This is what I need to ask myself in the next competition: do I want to run training now because it is the best move, or because I want to see something running?