If you haven’t used Lua before, it’s the go-to embedded scripting language for games like Factorio and World of Warcraft, and for applications like Neovim and OpenResty. LuaJIT, its just-in-time compiler, is widely praised for its speed, so it’s easy to assume your code is already running as fast as possible. In reality, you can unknowingly cripple performance by hitting one of LuaJIT’s NYIs.
NYI stands for “Not Yet Implemented,” meaning operations that LuaJIT cannot translate into optimized machine code. What happens next depends on the specific NYI, and untangling those differences is the subject of this post.
I ran into this while benchmarking grug-for-lua, my Lua implementation of the grug modding language. The same benchmark, with identical code and inputs, sometimes reported 6 billion iterations and other times only 300 million, a 20× difference. The culprit turned out to be a seemingly harmless operation in one part of the code that silently caused LuaJIT to blacklist a function that an unrelated hot loop later depended on.
This post follows that investigation. It aims to be approachable even if you’ve never touched LuaJIT or compilers before, so it links to background reading wherever something isn’t explained in full. We’ll look at two NYI-related pitfalls, and how to let your CI guard against them.
This article focuses on the foundations of solid benchmarking, so it helps you to decide which discussions to have with your team. It is not the full book. 
First the commercial message, so we can focus on the main subject. As benchmark-design is not always obvious, we help customers to set up a system that plugs into a continuous integration system and gives continuous insights. More about that in an upcoming blog. 
