After three losses I didn't build a better model. I built a checklist.

A post-mortem of finished competitions. The final entry in a series that records three losses in order.

Putting the post-mortems from all three competitions side by side revealed one common point. Each decisive mistake happened on the first day.

• At SCPC, I built a script that counted rank from the listing on day one, then used that number to make decisions for ten days.
• At NYPC, I wrote on the first night, “The real objective function is participant rank,” then spent six days on a different metric.
• At SNU, I found the strongest public checkpoint for this task type on day 16. Had I found it on day one, the campaign would have taken a different shape.

None of the three came down to ability in the middle or at the end. I set the wrong coordinate system on day one, then worked diligently within it.

That is why I built a checklist rather than a better model. Its first-day section is the thickest.

Five things to settle on day one

1. Is the reward function a step function or a continuous one?

I settle this from the rules first. A structure where only the top N receive everything is a step function; rewards proportional to rank or score are continuous.

When it is a step function and I am below the cutoff, preserving rank is a strategy that guarantees a reward of 0. In that situation, variance is an asset rather than a cost. A redesign with a 15% chance of success is better than maintaining the status quo with a 100% chance of success.

Conversely, when I am defending a position above the cutoff, preserving rank is correct. The same rule changes sign with position. So I keep asking not “which is right?” but “which side of the cutoff am I on now?”

2. The strongest public artifact for this task type

This is the first question in my day-one research: a checkpoint trained with reinforcement learning, a domain-pretrained model, a specialized encoder, a specialized tokenizer.

When compute and time are constrained, I import capability instead of reproducing it. Five failed attempts at in-house reinforcement learning versus two days to transplant an already trained checkpoint. That ratio comes from direct measurement.

Doing this on day one is frustrating because the score does not rise. I recover that frustration as interest on the remaining time.

3. Deadline time and time zone, quota, and reset point

I do not record only the date. I pin down the time and time zone. I check how many submissions the quota allows, when it resets, and whether it expires or accumulates.

I also go all the way into the submission screen. Rules and FAQs miss provisions visible only there: the number of submission files, filename rules, or memo character limits.

4. The output definition of the measurement tool

This item came from SCPC. When I build a leaderboard crawler, score parser, or rank tracker, I verify what its output means.

For rank, I trust only my fixed row calculated by the server. Counting rows in a listing gets fooled by pagination.

Right after building a tool, I ask: How would I know if this number were wrong? Without an answer, I do not allocate a budget based on it.

5. The size of measurement noise

I run the same configuration twice and measure the variation. Re-running a 300-item validation set had a variation of 0.02, while matching noise in a game competition was skill rating ±100.

A delta smaller than the noise cannot be measured. Repeat a change whose expected gain sits inside the noise twenty-six times, and all twenty-six measurements are meaningless. Measuring this on day one prevents wasted motion in the middle.

What I run once a week in the middle

The layer-above review

When a stall lasts more than a week, I force myself to ask about the category one layer up from the variable I am touching.

Pushing a curve already pressed against its asymptote does not move the asymptote. I have to change the curve.

Check whether the verdict is valid

Before writing “this axis does not work,” I look at the amount of training first: samples actually seen, out-of-memory counter, and measured time per sample. Every verdict that nearly fooled me three times was undertraining.

I also record “this axis does not work” and “I cannot properly evaluate this axis with my resources” as different sentences. The latter needs another look when resources become available.

Midpoint checkpoint

I put the 50% mark of the competition period on the calendar. That day, I calculate two things: the current architecture's expected endpoint and its gap to the cutoff, then make an explicit decision about whether to add another layer or rebuild.

Without setting that point, inertia makes the decision for me.

What I do in deadline week

• Regenerate the final candidate in the latest environment. I have measured library-version drift moving even greedy decoding by 0.02.
• Use checkpoint averaging as standard practice. Average the weights of the top two or three checkpoints for about $1 and 0.5–2 percentage points, but this only works if I preserved every checkpoint.
• Recover expiring quota as measurement rounds. Ask the server about candidates validation could not decide. A version that failed the validation gate has set a server record before.
• Take the final shot with room for re-scoring, not immediately before the deadline.
• Destroy an instance only after confirming the artifact upload. I once lost the weights of my best-performing model.

How two dogmas coexist

Two rules that appear to conflict became my operating principle.

First: constrain spending through method. I judge paid inputs by expected prize value divided by the cost of input. I do not keep betting on an axis without measured evidence. Sunk cost stays out of the calculation.

Second: no plan contains a “hold steady until the deadline” branch. I attack with the available means until the deadline.

They operate on different layers. Refusing to surrender is a matter of posture; thrift is a matter of spending. Zero-cost attacks, including local analysis, free compute, research, and designing the next hypothesis, do not stop before the deadline. Paid runs go one at a time on measured evidence. With no signal, I stay quiet; with one, I do not hesitate.

Both failing to stay quiet and hesitating are violations. Begging is a violation too, as is failing to ask when the arithmetic supports it.

One more thing: after two times, move it into code

When the same accident happens twice, I fix the structure on the spot. I do not stop at documenting a rule.

I needed this rule because I made the same mistakes twice: a current GPU-architecture kernel issue twice, and incorrect VRAM notation twice. Even an accident learned once repeats unless I turn the filter into code.

Once is noise; twice is a pattern. Rules written in documents are advisory. Only rules blocked by a script are enforced.

The question that stays

After writing all three post-mortems, I found that most entries in the list of causes were about sequence rather than judgment.

I generally knew what to do. I knew research should come first, that I had to verify the reward structure, and that I had to validate measurement. In some cases, I had even written it in my notes. But the urgent beat the important.

That is why I need a checklist. A checklist does not give me new knowledge. It puts what I already know ahead of what feels urgent.

On the first day of the next competition, I need to ask: Is the task I want to do today the most irreversible decision in this competition, or merely the task that produces results fastest?