A tagged pointer setup for detecting buffer overflows with no branches or extra memory accesses.
Actually just the opposite of the title. A lovely story on why sometimes you actually need to write software to build a product, not just snap together some lego blocks.
Mmap is so great! Just do a single system call, and all the IO is hidden behind the scenes. Oh, wait. Turns out that sometimes that transparency isn't what you want at all. (Specifically sometimes you want non-blocking IO. Or as it happens, I found this post while looking for stuff on the MAP_POPULATE | MAP_NONBLOCK combination that Linux used to once support).
Robin Hood hash tables with linear probing are just sorted arrays. What a great way to think about it.
> This makes it surprisingly hard to answer a very simple question: what is the fastest join algorithm in 2015? In this paper we will try to develop an answer. We start with an end-to-end black box comparison of the most important methods. Afterwards, we inspect the internals of these algorithms in a white box comparison. We derive improved variants of stateof-the-art join algorithms by applying optimizations like softwarewrite combine buffers, various hash table implementations, as well as NUMA-awareness in terms of data placement and scheduling
> Zobrist hashing starts by randomly generating bitstrings for each possible element of a board game, i.e. for each combination of a piece and a position (in the game of chess, that's 12 pieces × 64 board positions, or 14 x 64 if a king that may still castle and a pawn that may capture en passant are treated separately). Now any board configuration can be broken up into independent piece/position components, which are mapped to the random bitstrings generated earlier. The final Zobrist hash is computed by combining those bitstrings using bitwise XOR.And obviously the punchline being that given a board state and its hash, recomputing the hash of an output board state is simply xoring the original hash with the bitstrings of the moved piece; once before the move, once after.
Using genetic programming to choose when to use which heuristic for directing a iterative deepening A* search. Though I have to say that the game trees this paper is talking of seem ludicrously small for 2009 (1.5M states).
Running multiple unrelated algorithms for Sokoban solving in parallel, on the assumption that the different algorithms have different blind spots. You get better average and worst cases by giving each of x algorithms 1/x% CPU than run just one with 100%. (Assuming the selection is diverse enough, of course).
Also had some ideas about having the different algorithms exchange information, but that seems very complicated and didn't seem to pan out.
> However, there isn't a Morrowind speedrun category where someone tries to become the head of all factions. For all its critical acclaim and its great story, most of quests in Morrowind are basically fetch-item or kill-this-person and there aren't many quests that require anything else. But planning such a speedrun route could still be extremely interesting for many reasons.
Some really neat stuff about treating speedrunning as a search/optimization problem. I was a little bit annoyed by the parts where the story strays from that, and the author instead uses human intuition to e.g. select which set of quests to do or which skills to train. Also, part 2.
Two things you don't see often in CS. Trying to replicate a systems design paper, and publishing a negative result. And also showing just how many crucial details can get left out in a systems description that makes it impossible to actually implement. And when you do implement it and don't get the hoped for performance, what then? Obviously more and more optimizations that the original system probably didn't have.
It's kind of interesting to read the original paper's HN comments after this.
> Assuming typical game theory for the jerks, here’s what the thinking would have been: I was a jerk too, and my real goal here was not to actually solve a problem, but was to leverage SIMD either to usurp the people who led parallel programming models in the compiler group or to advance some other nefarious agenda.
A personal retrospective on the development of ispc (a compiler for a shader-programming style C dialect for x86-64). What a great story of big-company intrigue and dysfunction. I'm reloading this site daily to check for new installments.
Proxying traffic through home Wifi routers that expose UPnP to the internet. (I'd heard of malicious proxying through home routers, but I'd thought they were compromised devices rather than just misconfigured ones).
The sophistication of modern reverse engineering tools is pretty amazing.
> When a typical Silicon Valley company decides to "sell off their assets" that generally means office chairs, white boards, and the occasional espresso machine. Not test equipment, test fixtures, extra parts, and tools.
Using the closure of what was apparently a famous electronics scrap store to reflect on how Silicon Valley changed in the last couple of decades.
Expose SACKs directly to the encoder. Always use the latest fully ACKed frame as the keyframe. (Plus other things, but that felt like the interesting insight to me.)
A worthy new entry in the popular "why Go sucks" genre.
> I definitely found the answer to my question about why so few graphical Kaypro programs exist. The Kaypro’s graphics are awful – it’s a text-mode machine with graphics bolted on as a box-checking exercise. That being said, the development experience was surprisingly nice and it was a lot of fun to go through the exercise of actually making a functional game for a machine slightly older than me.
On the performance and power efficiency of Xeons vs. Qualcomms server chips on SIMD workloads. My basic assumption on CF's tech blog posts is that they're 90% PR. But this does have hard numbers, and they're pretty surprising ones (specifically the power usage / unit of work numbers. though I wish they had raw power usage as well).
> This post will focus on types of tech debt I’ve seen during my time working at Riot, and a model for discussing it that we’re starting to use internally. If you only take away one lesson from this article, I hope you remember the “contagion” metric discussed below.
John Cowan linked to this in a comment on my post on tagged pointers. It's a very comprehensive look at datatype implementations in (mostly) Lisp implementations.
(Is this the by the same Stan Shebs who wrote XConq back in the day?)
There's a bunch of different reasons why packets might get corrupted in-flight. This research finds out signals for distinguishing between those cases (+ congestion-induced packet loss) and recommends specific maintenance tasks to fix the problems.
A design for a key-value store for update-heavy applications.
A critique of the GDPR as a piece of legislation from somebody who a) appears to be a privacy activist, b) works as a GDPR DPO.
Another story of debugging and mitigating a problem in a closed source program.
>This is a long-ish entry posted after multiple discussions were had on the nature of having or not having bounded mailbox in Erlang.
Multi-use software can be used in bad ways. If you sell such software to authoritarian governments (or government-controlled companies), it'd be good to have controls on exactly what they can do. Obviously that doesn't work if the system is arbitrarily scriptable, but few systems are.
But what really offends me about this article is just what garbage the Procera traffic rewriting implementation clearly was.
Just what it says in the title. Stories about genetic algorithms etc. generating unexpected results.
Reading through this, I kept thinking that I'd pretty recently read about someone else using TCP congestion control for RPC queue management. And indeed I had, it was this post by Evan Jones. First time this linkblog actually did what I intended it for! ;-)
Not actually the death of the sampling theorem. But an absolutely brutal takedown of some dodgy signal processing research. The punchline:
> As so often, one does have to ask: How did these dramatic claims get through peer review? Given the obvious conflict with the Sampling Theorem, weren’t some eyebrows raised in the process? Who reviewed these submissions anyway? Well, I did. For a different journal, where the manuscript ultimately got rejected.
A tool to transform JSON to a line-based format, where each line is prefixed with a path. And a tool to transform from that format back to JSON. Such a clever idea.
Procedural map generation using (cleverly designed) Wang tiles.
The DNS protocol design is becoming increasingly detached from the practice, leading to increasingly complex and bug-prone features.
The new version resolution algorithm for Dart's package manager, with special emphasis on error messages. The contrast between this and the recent work for Go package version is pretty interesting.
What a classic rant, can't believe I hadn't seen this before. (I hadn't seen the original Crockford DEC64 page either. But the rant is at least amusing, while the DEC64 proposal didn't really have many redeeming properties).
A lovely walk through optimizing a single Opus Magnum solution, step by step.
A tale of a surprisingly long-running positive-return lottery syndicate. Or actually, two of them. Who then end up eating into each others' profits, and start a dirty fight on both sides. (Ok, this last bit isn't a big part of the story. But it's what I chuckled at the most.)
This story did not go where I expected it to from the title.
> Dataflow and data dependencies can be viewed as the fundamental expression of the structure of a particular computation, whether it’s done on a small sequential machine, a larger superscalar out-of-order CPU, a GPU, or in hardware (be it a hand-soldered digital circuit, a FPGA, or an ASIC). Dataflow and keeping track of the shape of data dependencies is an organizing principle of both the machines themselves and the compilers that target them.
How to think about optimization.
>The other thing I’ll say is that even though I’ve been talking about adding cycle estimates for compute-bound loops here, this technique works and is useful at pretty much any scale. It’s applicable in any system where work is started and then processed asynchronously, with the results arriving some time later
Paul Khuong on one of the hardest bugs he has had to debug.
Debugging story. A Windows service causing an invisible 40GB kernel memory leak due to some kind of race condition where it occasionally fails to properly release process handles.
Some interesting thoughts on the value of writing blog posts vs. living documents that work as long-term resources.
Early versions of Windows did bit-blitting by JITting a routine specialized for the actual parameters. This is the evolution of those routines.
Thoughts from Martin Cracauer on what the GC implications would be on using LLVM as a SBCL code generator..
What goes into implementing enough of the OS/2 ABI to get an absolutely minimal graphical application working?
> There are lots of plausible ways to pack bits into bytes, and all have their strengths and weaknesses that I’ll go into later. For now, let’s just cover the differences.
Also, part 2
Writing an asynchronous (when allowed by the semantics) D3D to OpenGL shim.
Writing a SQL parser in Haskell isn't very interesting. The good part is everything else about this. All the way from the genesis of the tool (need to figure out what all the relations in the system really are, for a hellish schema transition), to where the system actually ended up at and what other use cases naturally appeared.
The typical HN second guessing comments feel even more depressing than usual. Why didn't they just read the documentation of the tables to figure out the details? Why not use a Python SQL parser instead of writing a new one? Why did they want this schema transition anyway? It's like there's zero empathy for other people's problems being more complicated than can be explained in the setup of a blog post.
A deep dive into reverse-engineering an ultra-obfuscated piece of malware, with multiple layers of custom virtual machines. Really awesome.
A network debugging war story, involved IPv6, fragementation and QUIC.
I'm probably going to disagree a bit on the moral of the story. The authors' takeaway here is that routers should not be reordering packets.
What I see here is yet another instance of full transport layer header encryption making it impossible to do the right thing. Why does the server need to MTU-probe with a massive packet? Because there's way for the path to give a signal about the packets size (MSS clamping in TCP). Why does the receiver end up blocking the queue on fragmentation? Because there's no way for it to know what the intended order was, the packet numbers are encrypted. So it has to assume the receiving order is the delivery order.
But look Ma! No ossification!
> I do not believe in objectivity in rankings. This is not to say I think being objective with regards to rankings is impossible, nor do I think "objective" tools serve no purpose (the tools I've written have already proven highly useful in generating baselines for seeding tournaments). No, more specifically I want to stress that "objective" ranking systems are much less objective than they actually seem, and the word "algorithmic" or "empirical" might be better.
Rating systems, once again. I don't think I agree with much of this article (e.g. the reasoning for Elo not working for double-elimination seem totally nonsensical). But the core idea of not having tournament seeding be purely algorithmic? Sure.
Blizzard going above and beyond on remastering an old game. There was apparently a large number of user-made StarCraft maps that relied on buffer overflows to read/modify game internals (all the way to basically rewriting some of the game logic). How do you not break these maps when the game is basically completely rewritten? By basically building an elaborate buffer overflow emulation layer.
Just a crazy level of dedication.
"How do you write a minesweeper puzzle generator that always generates a level that can be won without guessing" is a boring question. That kind of level generation sucks. For a moment it looks like it's where this article is going. It's not, though. The core idea here is a lot more clever.
"Dear ImGUI" in the browser with WebGL and webasm. I've been wanting to do something like this for a couple of small browser-based games.
(Something a bit odd going on with the keyboard handling though).
Using the Rust type system to make access to the global register state of embedded devices safe. (And some thoughts on API design).
A reasonably general-purpose system for fuzzing servers all the way from the main event loop, not just at some arbitrary "this is where we can feed the system a continous block of bytes" boundary.
(Also: Remind me to write about fuzzing the TCP stack itself, at some point).
Distributing video encoding with more granularity than by keyframe.
Absolutely lovely networking war stories (yay!) about early HTF (blech). All of the hacks are lovely. The one that resonated the most with me was figuring out a way of finding an application-level use case for out-of-order TCP transmitting (sending the header and footer of an order early, and once you've decided on a trade to make, sending out the price/count/stock id as a tiny packet that fills in the gap between the header and footer).
Sparklines of the values of individual memory locations. Works because early game consoles had so little RAM. Such a simple idea, such cool output.