Would an idiot know the difference between abelian and non-abelian group theory? I wasn’t trying to underestimate you; I agreed with your position and provided a tangent that opens up your position without compromising it. Next time I’ll explicitly say “yes, and” if that will help.

First, I personally don’t yet believe in the cryptographic security of LWE on lattices. I agree that it sure looks hard, but we don’t have a solid proof. But also, I don’t believe that we’ve found any provably one-way functions in the classical regime either. So I agree with you from different premises.

Unlucky 10,000: Shor’s algorithm speeds up any discrete logarithm. It actually speeds up the abelian HSP. This does give us a theoretical reason to expect that LWE on lattices won’t fall to Shor’s approach, as the underlying groups are non-abelian. It does make me sad for elliptic curves, though; they’re so elegant and the keys are so small.

Yes, precisely. One submission would have been in F tier, but I didn’t define an F tier for task 1. Some folks claimed to participate but never provided code or prompt logs.

I’ve finished grading all of the entries so far. I don’t think that we’ll get any more, so here’s a preview of the upcoming blog post.

The tier listings are as follows:

• B tier: Corbin S. (Task 1), Corbin S. (Task 2), Corbin S. (Task 3)
• C tier: Piper M. (Task 1)

Admittedly, we didn’t get a whole lot of players, but that’s it. That’s the entire tier listing. I had three things I wanted to do in my spare time. I did them and got an average ranking based on my average predictions of the future; I met expectations. Piper also placed and I greatly appreciate her sportsmanship here.

My solutions are available as notes and source code. For Task 1, I have three main commits: one, two, three, and a bugfix. For Task 2, the commits are internal to my homelab, but I do have notes and source code. Finally, for Task 3, I put the entire repository into a flat gist including notes, source code, and Nix flake.

I’ve started grading and his grade is ready to read. I didn’t define an F tier for this task, so he did not place on the tier list. The most dramatic part of this is overfitting to the task at agent runtime (that is, “meta in-context learning”); it was able to do quite well at the given benchmark but at the cost of spectacular failure on anything complex outside of the context.

It occurs to me that this audience might not immediately understand how hard the chosen tasks are. I was fairly adversarial with my task selection.

Two of them are in RPython, an old dialect of Python 2.7 that chatbots will have trouble emitting because they’re trained on the incompatible Python 3.x lineage. The odd task out asks for the bot to read Raku, which is as tough as its legendary predecessor Perl 5, and to write low-level code that is very prone to crashing. All three tasks must be done relative to a Nix flake, which is easy for folks who are used to it but not typical for bots. The third task is an open-ended optimization problem where a top score will require full-stack knowledge and a strong sense of performance heuristics; I gave two examples of how to do it, but by construction neither example can result in an S-tier score if literally copied.

This test is meant to shame and embarrass those who attempt it. It also happens to be a slice of the stuff that I do in my spare time.

Kernel developer’s perspective: The kernel is just software. It doesn’t have security bugs, just bugs. It doesn’t have any opinions on userspace, just contracts for how its API will behave. Its quality control is determined by whether it boots on like five machines owned by three people; it used to be whether it booted Linus’ favorite machine. It doesn’t have a contract for its contributors aside from GPLv2 and an informal agreement to not take people to court with GPLv2 contract violations. So, LLM contributions are… just contributions.

It might help to remember that the Linux development experience includes lots of aggressive critique of code. Patches are often rejected. Corporations are heavily scrutinized for ulterior motives. Personal insults are less common than they used to be but still happen, egos clash constantly, and sometimes folks burn out and give up contributing purely because they cannot stand the culture. It’s already not a place where contributors are assumed to have good faith.

More cynically, it seems that Linus has recently started using generative tools, so perhaps his reluctance to craft special contributor rules is part of his personal preference towards those tools. I’d be harsher on that preference if it weren’t also paying dividends by e.g. allowing Rust in the kernel.