Crossing silver, stopped 0.0063 short
A public record of stacking alphas to raise my score on the quant trading platform WorldQuant BRAIN. In episode 6, I found and submitted my fourth alpha. This episode covers how that alpha landed on the scoreboard and pushed me into Silver, and how I nearly landed a fifth alpha before stopping one step short of the finish line.
0. One-line summary
The fourth alpha landed and pushed me past Bronze into Silver. The fifth candidate I found next cleared both gates of the pass criteria (Sharpe, Fitness), but fell short of the third gate, self-correlation, by exactly 0.0063, and despite more than twenty adjustments, I never managed to clear it.
1. A day spent waiting for it to land
The fourth alpha I built at the end of episode 6 wasn't something I could submit right away. Score has a daily cap, so it's advantageous to spread strong alphas across different days. So I waited a day before submitting.
There's also a lag between submitting and having it show up on the scoreboard. The score updates every day at dawn (US Eastern time), and if you don't submit the alpha in time to hit that update window exactly, you have to wait for the next day's update. That's what happened again this time. Because I submitted just past the update time, it took nearly 20 minutes from the update time for it to land. During those 20 minutes I checked my score again and again. When the number finally changed, the score jumped from 5,546 to 7,464, and my tier rose from Bronze to Silver. 2,536 points remain to the next threshold, Gold.
2. Revisiting a blocked path: sweeping in batches
Setting out to find a fifth alpha, I changed one thing about my approach this time. Up to now I'd designed and tested each candidate by hand, one at a time, but this repository had a tool called the "batch miner" built in from the start. It's a tool that automatically substitutes many data fields into several standardized templates, letting you test dozens at once. This was the first time I properly used it in this session.
The fundamental dataset has 886 financial statement line items. I ran the ones like assets, liabilities, and cost of goods sold through it one by one. But there was a hole in this tool that needed fixing. There are five templates to substitute into, and if you set the cap on how many to test at once equal to the number of fields, the first template alone uses up the entire cap on fields, and the other four templates never run even once. I only noticed this belatedly. So what I thought was a scan of the first 34 fields was actually just testing a single template, "simple ranking of fields."
3. A trap stumbled into by chance, and how to avoid it
While running the rest of these templates, one number caught my eye. It was an accounting item called "discontinued operations current assets," which only appears when a company is winding down or selling off part of its business. Run alone, it produced Sharpe 1.07. The best figure of this session.
But then a lesson from episode 3 came back to me: flashy numbers should be doubted first. This item is null or zero for most companies. Only companies winding down a business have a value for it. So I re-measured with a narrower set of stocks (only large caps with good liquidity). The 1.07 collapsed to 0.37. It had been a fluke concentrated in a handful of unusual stocks. Having confirmed the trap, I discarded it outright.
4. Ranking within groups
While continuing to test various financial line items across different templates, one pattern started to stand out clearly. When you changed an item from "rank across the whole market" to "rank within the same industry," the metric consistently rose for a particular class of items in particular. Items like liabilities, cost of goods sold, and liquidity behaved this way. My guess at the reason: the level of debt an industry can sustain differs by industry to begin with. Compared against the whole market, that difference becomes noise, but compared against peers in the same industry, the real signal shows through.
I re-measured twenty-seven items this way. Stock trading turnover and long-term deferred revenue came out highest at 1.03, followed by long-term debt issuance at 1.01, and long-term debt balance at 0.92. Valuation items like enterprise value or equity, on the other hand, didn't rise at all with this approach. Items where within-industry comparison works and items where it doesn't split cleanly apart.
5. Where liquidity meets competitor spillover
I multiplied "ranking by current ratio within the same industry" (ordering by how ample cash-like assets are relative to short-term liabilities, then flipping it so poorer liquidity ranks higher) by the competitor-return signal validated in episodes 4 and 6. The result: Sharpe 1.33 and Fitness 1.02. Both gates cleared.
I checked the third gate, self-correlation. 0.7189. The threshold is 0.7. Blocked by a gap of just 0.019.
6. Twenty rounds of fine-tuning and a final 0.0063
From here I spent more than twenty rounds tweaking this one alpha. I shortened the averaging window for the competitor signal one step at a time from 21 days. Cutting it to 19 days brought self-correlation down to 0.7141. At 18 days it was 0.7093, and at 17 days it fell as far as 0.7063. A clear trend of lower correlation the shorter the window got. But cutting it further to 16 days dropped Fitness to 0.95, blocking me at a different gate instead. 17 days was the only point where both requirements overlapped, and even there I was short by 0.0063.
I pulled every other lever too. Changing the degree of smoothing, adjusting the weight cap, splitting industries more finely or grouping them more broadly, even dropping the competitor signal entirely and swapping in different material, none of it worked. Dropping the competitor signal lowered self-correlation, but it weakened the alpha itself in exchange, so I got blocked at a different gate instead.
At one point I tried something that looked like a shortcut and ended up wasting the effort completely. I tried to rebalance the weight of the two ingredients by amplifying or suppressing the signal at an extreme, but applying that adjustment right inside the final ranking function had no effect at all. A ranking function only looks at order, not magnitude. Any adjustment that preserves order, no matter how it's done, produces an identical result. It took four wasted simulations before I realized this. To actually see the effect of the adjustment, I had to touch the stage before ranking, where values are actually multiplied or added, but doing that let extreme values swing the whole signal around, and the metric sank as low as 0.03.
In the end, this combination, "current ratio ranked within industry multiplied by competitor spillover," stopped at the self-correlation wall. Having put in more than twenty adjustments and more than fifty simulations, I consider this exact combination thoroughly checked.
7. Why self-correlation, of all things
Looking back, there's something that lines up. The competitor-return signal was already built into the third and fourth alphas, each in a different form. The first alpha didn't use this signal, but the third and fourth already did. The self-correlation criterion measures "how similar is this to every alpha I've submitted so far," so using the same material for a third time stacks its similarity to both alphas at once. When the third alpha used only this signal alone, similarity was low at 0.138; when the fourth mixed it with other material it rose to 0.43; and it appears that using it for a third time this round pushed it up to around 0.72. It means that even the same good material carries a different cost depending on how many times it's been pulled out of the drawer.
8. What I learned this episode
- There's a lag before a score lands. If you submit just past the update time, you have to wait for the next update instead of that day's.
- Even automated tools need to be verified before use. Four of the batch miner's five templates had never run even once. You have to check what a tool is actually doing before trusting it.
- Comparing within an industry reveals things. A signal that was dead when compared against the whole market came alive when compared within the same industry. Though this effect only worked for the debt/liquidity family, not for valuation.
- Even good material loses value with repeated use. Reusing a validated signal for a third time accumulated self-correlation into a wall. Repeated use of the same material carries a cost.
- Adjusting inside a ranking function doesn't change the order. Adjusting only magnitude right inside a ranking function has no effect at all. The adjustment has to happen at the stage before ranking, and even then it needs extreme-value management.
- You also have to know when to stop after trying enough. If twenty rounds of adjustment still don't work, that may not be a method you haven't found yet, but a limit of the combination itself. Still, that judgment should only be made after trying enough.
Next episode, I'll aim for a fifth alpha again using other promising material found this time (long-term deferred revenue, new debt issuance). Whether to use the competitor signal again, or go with completely different material, is the biggest question this hunt left me with.
Technical notes (appendix)
- Score status (07-02): SILVER, 7,464 points, rank 20,520, 4 alphas landed. 2,536 points to Gold (10,000).
- group_rank(field, subindustry) measured rankings (27 fields, USA TOP3000): highest fnd6_cshtr (stock trading turnover) / fnd6_drlt (deferred revenue LT) at 1.03, fnd6_dltis (long-term debt issuance) at 1.01, debt_lt (long-term debt) at 0.92, cogs (cost of goods sold) at 0.88, debt_st at 0.86, debt at 0.84, current_ratio (flipped) at 0.81. Confirmed a +0.07 to +0.13 uplift over val_rank (plain cross-sectional rank) for the leverage/liquidity/operating-scale family; no uplift for the valuation family (enterprise_value/equity/cashflow_op) (stuck at 0.24 to 0.30).
- 5th candidate (not submitted, ultimately failed on self-corr):
rank(-group_rank(current_ratio, subindustry)) * rank(winsorize(ts_mean(ts_backfill(rel_ret_comp,5),21),std=4)), USA/TOP3000/delay1/decay6/INDUSTRY/trunc0.02 → Sharpe 1.33, Fitness 1.02, sub-universe 0.83, self-corr 0.7189 (cap 0.7). Confirmed a linear decrease in self-corr from 0.7189 to 0.7141 to 0.7093 to 0.7063 as the competitor spillover window was narrowed from 21 to 17 days (at window 16, Fitness fell out of gate to 0.95). Confirmed decay/truncation/SUBINDUSTRY and SECTOR neutralization/comp-free substitute partners/signed_power reweighting were all ineffective, closed out after investing in 20+ batches and 55+ simulations. - Batch miner bug:
scripts/mine.py --template-set valuecycles through 5 templates (val_rank/val_rankneg/val_tsrank/val_grp/val_improve), but giving--maxequal to the field count means that under template-major traversal, only the first template uses up the whole cap and the other 4 never run. Worked around by separately specifying and running the remaining templates on strong fields. - Sub-universe trap reconfirmed:
fnd6_acdo(discontinued operations current assets) alone at S1.07/F0.90 collapsed to 0.37 in TOP1000 (properly verified by sign flip to -1.09). A textbook case of concentration overfitting on a sparse accounting item that's missing/zero for most stocks. - signed_power trap:
rank(signed_power(x,p))is a monotonic transform, so wrapping it right after rank() leaves the order unchanged, effectively a no-op. To actually change the weighting, the transform must be applied at the multiplication/summation stage before rank(), but multiplying raw values (zscore) is vulnerable to extreme values and can collapse the signal (measured at 0.03). - Self-correlation accumulation observation: a correlation between how many times the comp (competitor spillover) engine has been reused and self-corr. 3rd alpha (used alone) 0.138, 4th alpha (confidence-weighted combo) 0.43, 5th candidate (current ratio combo, 3rd use) rising to around 0.72. Reusing the same strong engine repeatedly raises the risk of accumulating correlation.