Note: This testing described in this post was done over a year ago. It might be that things changed since then.

At work, we recently upgraded our PostgreSQL servers. This time, however, we encountered an unexpected roadblock when attempting to enable full disk encryption (FDE) with LUKS - our standard deployment. In past benchmarks, enabling LUKS full-disk encryption cost us ~10%. This time, it left us with only 10% of our throughput - a 90% drop.

So a bit of background, the new server is the most capable server I’ve benchmarked to date. 12 Intel P5520 NVMe disks and two 48-core Intel Xeon Sapphire Rapids CPU. Enough horsepower to handle our workloads for the foreseeable future.

With the 12 disks in a RAID10 configuration, we were happy with the raw performance figures: sequential write speeds of around 20 GiB/s and read speeds of 40 GiB/s (note gigabytes, not gigabits).

But our joy was short-lived. As soon as we added the LUKS encryption, performance plummeted. This was not at all what we expected, we had not even planned to do much benchmarking because previous synthetic benchmarking had only shown a 10% decrease, which in reality meant no real world impact.

But I had always had the understanding that modern cryptography is so fast that encryption won’t add any overhead. Most of it probably stems from the push for TLS and http://istlsfastyet.com/. So it might be wrong.

Fortunately, we weren’t the first to encounter such a noticeable impact. When researching, we found that Cloudflare had previously grappled with a similar problem and contributed a fix that was eventually merged into the Linux kernel. Enabling this fix reduced CPU usage to near-zero levels. However, surprisingly, it did little to improve the overall I/O performance.

As I had personally told many that encryption is so fast that there is no reason not to do it, even if the data wouldn’t be sensitive I felt I had to get to the bottom of this. So we set up a more comprehensive testing framework with various options to see if we could figure out how to improve the speeds. We also added a wildcard: the bcachefs filesystem which seemed promising and had some interesting features, like using an NVMe/SSD as read/write cache for spinning disks.

Our test setup consisted of a 10-disk RAID10 array of Intel P5520 NVMe disks, divided into eight equal partitions spread across all ten disks. The reason for not using all 12 was that we planned to do some single disk testing also, and with the unencrypted file system the raid10 scaled quite linearly. We decided the results of the 10-disk tests should apply to a 12-disk raid10 as well.

These were the options we decided to explore

• Plain ext4
• Plain LUKS
• LUKS with --sector-size 4096
• LUKS with Cloudflare patches
• LUKS with Cloudflare patches and --sector-size 4096
• Bcachefs
• Bcachefs with encryption
• LUKS with --sector-size 4096 and --perf-same_cpu_crypt
• LUKS with Cloudflare patches, --sector-size 4096, and --perf-same_cpu_crypt

The server was running the pre-release of Ubuntu 24.04 which was available in March 2024 (in order to get a new kernel 6.8 which supports bcachefs). To test these configurations, we ran four different FIO tests (gist with the .fio) on each of the targets. The tests are designed to test sequential read and write speeds, as well as random small reads and writes to see how many IOPS we can get out of them. While the test parameters might not be optimal (if you have ideas how to improve them, let me know) we stuck with them because it was the same settings we used many years ago, so it gives us an idea how things have improved.

The Results

You can view the full fio results and raw numbers in the google sheet here.

Looking at the graph, the key setting was the sector-size. This change brought us up to around 50% of the raw disk speeds – a substantial improvement.

All the other options we tried for LUKS seemed to have small to no measurable differences.

While we are not experts on the inner workings of LUKS, it does seem to make sense that a bigger block size would speed things up. Our interpretation is that the bottleneck is not the raw encryption but rather somewhere in the process of getting the blocks lined up for encryption. Therefore, reducing the number of blocks eightfold speeds things up significantly.

The younger bcachefs filesystem, did not manage to keep up with the encryption enabled. Our working theory is that since it uses ChaCha20/Poly1305 for encryption, which, while being a fast algorithm, does not benefit from specialized hardware instructions like AES does. It might be that Intel QAT could help but we did not have time to look into that.

Conclusion

In the end, we settled on using LUKS with --sector-size 4096, the Cloudflare patches enabled, and --perf-same_cpu_crypt. While not achieving the same great speeds as using RAW, this configuration provided us with good enough performance to meet our needs while allowing us to maintain the security benefits of FDE.

We also explored the possibility of leveraging the built-in encryption capabilities of the disks through the Trusted Computing Group’s OPAL specification. Unfortunately, our specific NVMe models lacked support for OPAL or any other form of disk-level encryption.

The key lesson: always benchmark, even if it’s just a quick one. Defaults can leave enormous performance on the table, and the fix may be as simple as a sector-size flag.

Special thanks to Björn D. and Niclas J. for their assistance with both the testing and giving feedback on this post.

I thought it was crazy to think about all the 8.2 billion people.

A even crazier thought is how many internet users there are, who in theory can talk to anyone else.

From an information point of view it’s just a amazing amount of informationand potential available.

Of course, not everyone has something to say to every other user. But the fact that people who are interested in a subject can “easily” talk to so many likeminded people should allow for humanity to develop at a never-before seen pace.

Are we? I don’t know. Feels like things are stagnating, partially because of bad incentives. Or just human nature not having caught up with things.

Anders Hansen, a Swedish psychiatrist talks a lot about this disconnect between what evolution has prioritised during the last 100k years vs what we would “need” right now to thrive in this information and abundance society.

If you have not read his book, “The Attention Fix” I strongly recommend that you do. It’s definitely on the short-list of most influential books I have read within the last few years. It has really allowed me to view things in a new light, which is I think the best a book can do.

There are a lot of people on this planet.

Our brains are not designed for numbers that big.

I put together a little art project about how many people there are.

nyman.re/everyone-…

Makes you feel small doesn’t it?

And still you can make a difference. Magical

Claude Code > Gemini Code > Codex

Based solely on my gut feeling after having played with each one of them on some toy projects.

One pet peeve I had with Codex was that it did not want to finish things, when asked to move stuff, if it thought it was too much it just left at the end of the file.

Gemini Pro worked fine the little I tried it but I dislike on principle because it’s not possible to deactivate the history when it’s part of the Google Suite/Workspace/Apps. I’m actually not sure if that affects Gemini Code.

Claude Code seems to be a good balance.

At some point I will try to compare by giving them the same instructions on the same project and see what actually happens.

Do you have an old Keybase account? Or do you know someone who has? If not, you can stop reading now. If you have, and you weren’t into the crypto stuff back in the days you probably have around 500 EUR / 600 USD lying around there in magic internet money.

With the world on fire, those money is not doing any good locked into some internet wallet. So I’d recommend getting them out of there and spend them on either yourself, someone you care about, or something good like charity.

It takes a bit of effort, as Keybase does not allow you to transfer them directly, but thanks to a great writeup from Yael you can do it.

After you have imported them into LOBSTR or some other wallet, you can donate them directly to charity with www.daffy.org (is one of the few I found that accepts XLM) , The British Red Cross accepts some other crypto.

Or you can transfer them to an exchange to turn them into real money and donate that.

Use a reputable exchange if so. Because crypto is mostly crime, you will have to do a lot of KYC (know your customer, sending your ID and such) so pick one you trust with your data. After a bit of research I used bitstamp, and hopefully that won’t turn out too badly. But do your own research and pick one suitable to your jurisdiction.

blaugust

Another blaugust post, not much editing. I have a much longer draft, discussing the various charities which accept crypto and how to make the most out of your XLM. I might publish in the future, will link to it then if so.

I have for a long time been using the + format to create “unique” emails for companies. Nowadays I’ve levelled that up with using <word>@rnd.mydomain which is a wildcard for the domain. I’ll write a blog about why it’s better and how to do it sometime.

Either way, this is a repost of an old twitter thread.

Back in 2020, I was been repeatedly called by a bitcoin/CFD company who refused to accept a “I’m not interested”. Let’s call them Company MagicMoney. I had my suspicions on where they got my number from and I got confirmation when I asked them to email me the inf, and they sent me a mail to gabriel+somehwat-legit-trading-platform@mydomain.

And as the original company is EU based I sent them a #GDPR request to see what they are doing with my data and maybe get them to stop giving it to shady trade platforms based in St. Vincent and the Grenadines.

I did get a response with a what looks like a JSON dump which seemed to contain most relevant interactions I had with the site. And it was completed within 30 days so at least their compliance department was working.

But as for how the data ended up in St. Vincent and the Grenadines? They denied ever sharing or selling it.

Which might be true or not, either way I just exercised my right to have my data deleted and called it a day.

Endnote

This happened in 2020, I don’t remember which companies were involved or why had signed up with Somewhat-Legit-Trading-Company. Interestingly, after I asked them to delete all my info, the calling stopped. Coincidence? Maybe.

Blaugust

Another blaugust post tagged as draft.

A few days ago I blogged about how great logcheck was. And towards the end I mentioned that writing rules was one of the more annoying parts.

I have for a while been considering how that can be done easier, and while I don’t have my dream solution yet, I have spent a few evenings vibe-coding something that I believe will be helpful.

Logcheck regex helper

You can try it out at nyman.re/logcheck-helper/ right now, or keep reading to learn more about the motivation and possible future development.

Why is writing rules for logcheck hard?

First, it uses egrep (which is an alias for grep -E nowadays), which uses POSIX Extended Regular Expressions. It’s in my experience not the most user friendly regex. Then when you have written one, testing it requires that you do something like logcheck-test -r /etc/logcheck/ignore.d.server/custom -l /var/log/syslog|grep named, but you need to pick the right log and the right rule file.

So what I usually end up doing is writing very coarse matches, which isn’t optimal.

How to fix this

The main problem with logcheck is that it’s noisy, and in combination with writing new ignores being hard, it becomes noise very quickly and nobody reads them. To allow the end user to cut down on the noise, it must really easy to do so. One idea I’ve had is that the alert emails would have a [ignore] button for each line (or all) which will create ignore similar log lines in the future. It would then optionally sync those ignores to all servers so we don’t need to repeat ourselves.

But I am not there yet.

What is Logcheck Regex Helper

It’s a quick proof of concept that tries to help you write the regex. It explores my ideas around providing the user with a working regex, while still allowing them to customise it. In the current version, you can click on the patterns in the pattern-configurator to switch between the regex targeting any text, or that literal text.

This should allow the user to quickly get a decent rule, that can be modified for their use case.

Why vibe coding?

Because this is what it’s good at. Putting together a “simple” proof of concept and allowing for quick experimentation.

And this is all client-side, so there is no risk that it would create a vulnerability.

And what it produces is not critical in any way. If it produces dumb logcheck rules, you will hopefully spot it and worst case you probably just fail to ignore what you wanted ignored.

But the PoC is now deployed?!? So it’s in production?!

Yes. But only because it’s all client side. So there is no risk that bad code could cause issues… (for me).

If this had a server side, I would either sandbox it really hard or not publish it. Probably both.

Will you open source it?

It is open-source… like all client side web applications. It’s not even minified. But if someone asks nicely and would like to contribute or fork it, I will publish the repo. The license is 2-BSD so feel free to copy it.

Blaugust

This is another blaugust post, no editing or spell-checking. Sorry.

Vibe coding is a slot machine. I agree.

You pull the lever and out comes code. Most of the time it’s close but not right, so you pull again.

Meanwhile you watch the funny little loading messages they added to give a bit of extra dopamine kick.

But it’s the most useful slot machine I’ve ever seen.

Why I’m not worried about my job part.

Evidence 17: Google forcibly rolling Gemini everywhere before anyone has had a chance to actually consider the security implications.

www.youtube.com/watch

From the risky biz newsletter

#blaugust

This is related to my previous post on LLM’s. Feel free to skip it if had too much LLM.

There have been several studies on how LLM’s seem to make us dumber. And there is probably truth in that. It’s quite logical if we look at the biology. The brain requires more energy when it’s working, and probably even more when forming new memories.

So if there is an easier way, it will prefer that way, as for most of humanity’s existence wasted energy was not good for survival. And biologically, like your muscles will become smaller unless you exercise them, the brain probably does the same.

So I think “brain exercise” will become more important as LLMs become more useful, just like intentional exercise like going to the gym is an important part of our lifestyle because of powered transport like cars.

I don’t know what will it look like. But I’m quite sure it won’t be like the “brain training” applications. They claim to make you smarter, but based on what I’ve understood they are a bit too limited. From what I’ve read, you become better at doing their brain games but it does not translate well effectively to becoming better other things.

So what kind of brain gym should you do? I guess whatever you want as long as it’s closely related to what you want to train and requires thinking, but intentionally leave the LLM out of it.

Although you might actually want the LLM before and after, because an important part of becoming better at something is that the difficulty is right, so it challenges you. Creating customised challenges, and evaluating them afterwards is probably something a LLM is good at.

blaugust

This is another blaugust post, very little editing so it’s marked as draft.