[RE: nyman]#_

don't let perfect be the enemy of bad

Recent Posts

ChatGPT does Advent of Code

published on

So, it’s that time of year again. Advent of Code is released, and I eagerly decided it’s a good time to learn a new programming language. This time the idea was to learn Clojure. But not being familiar with any other lisp style languages, I quickly got very stuck and was just about to give up.

But then I saw all these examples of ChatGPT going around, and I thought, “I wonder if ChatGPT could help me learn Clojure”. And it turns out, yes, it can!

Almost. Turns out, ChatGPT isn’t very good at Clojure. None of the solutions it provided worked without tweaking and debugging.

However, what I saw was so impressive, so instead of learning Clojure, I have decided to see how far ChatGPT can get with Advent of Code.

To begin with, as with most AI solutions out there, getting the prompt right is key. I did try to just copy the Advent of Code instructions into ChatGPT to see what happened, but that did not produce the right output. Instead, it took a few (sometimes a lot) of tweaks to the prompt to get a working result.

But eventually, we succeeded. ChatGPT (with a bit of help from me to combine outputs) produced a working solution for the first day of Advent of Code. So follow along with what will hopefully be 24 posts about how ChatGPT solves Advent of Code, using any means necessary. Get ready for some AI-powered fun!

ChatGPT tries to write some Clojure

I won’t repeat the original instructions here, instead you can find them here. Let’s try with a simpler version.

But that won’t do because Advent Of Code will provide me with a very long text file in the format, and this solution requires that I enter it in the above format. Also, we get the first indication that ChatGPT might not be that great at Clojure. When I tried running the code out of curiosity, I got an error.

; Execution error (ClassCastException) at java.lang.Class/cast (Class.java:3889).
; Cannot cast clojure.lang.PersistentVector to java.lang.Number

But no worries, I’ll just ask it to provide me a solution which reads and parses the input from a text file.

Sadly, it seems ChatGPT isn’t very good at Clojure. None of the examples it provided worked outright. There was always some issue. And even if I explained the error and asked ChatGPT to provide a solution, it rarely succeeded.

If not Clojure, then what?

After a while of trying to get it to produce correct Clojure, and helping me debug and explain it (which it is quite good at) I decided to just see what else it could do.

There is always a debate about which language is the “most efficient” or “concise” so let’s see what happens if we let ChatGPT choose.

awk of course. The second most popular choice for job-security (after perl). But this time it almost works. Awk throws a syntax error. For some reason, it does not like the ternary operating in the END{}. But if we just assume the last one isn’t the biggest one, we can change it to awk '{if($0!=""){s+=$0}else{m=s>m?s:m;s=0}}END{print m}' and. TADA! That works.

ChatGPT3 has (with a tiny bit of help) solved the first puzzle in the Advent of Code 2022.

Second part of the puzzle

Let’s see what the second step in the problem is. Ok, apparently we need to find the sum of the top three now. Sadly, I seem to have misplaced the prompt, but either I am getting better at writing prompts, ChatGPT is better at producing solutions, or we just got lucky but the solution it provided worked on the first try!

The code it created was

awk '
  BEGIN {
    sum=0
    num_groups=0
    top_groups[3]=0
  }
  {
    if ($0 != "") {
      sum += $0
    } else {
      if (sum > top_groups[1]) {
        top_groups[3] = top_groups[2]
        top_groups[2] = top_groups[1]
        top_groups[1] = sum
      } else if (sum > top_groups[2]) {
        top_groups[3] = top_groups[2]
        top_groups[2] = sum
      } else if (sum > top_groups[3]) {
        top_groups[3] = sum
      }
      sum = 0
      num_groups++
    }
  }
  END {
    if (num_groups < 3) {
      print "Not enough groups"
    } else {
      print top_groups[1] + top_groups[2] + top_groups[3]
    }
  }
' INPUT

The future?

Tomorrow we will try Day 2, or rather I have already tried and ChatGPT3 did succeed with the first part, I just haven’t had time to do the second yet. But already the first part took a lot of tweaking the prompt, and it highlighted one of the core issues with these Lange Language Models (LLM).

Tune in tomorrow to find out what the issue was and how it did.

published on

To extract Wireguard configurations from the official MacOS client, for example from an old Keychain file

security find-generic-password -l 'WireGuard Tunnel: <tunnel-title>' -w|xxd -r -p

#oneliner #documentation #wireguard #osx

Reduce python breakage

published on

Recently I ran into an issue with a python project I was working on. A dependency of a dependency decided to do a breaking change, which broke my project even if I had everything in requirements.txt pinned (same issue as here).

As part of fixing it I learned a few new things which I’ll share here.

First, you can constraint pip to prevent a newer version of the package from installing. pip a flag -c which you can read more about documentation here.

You can add

markupsafe==2.0.1

to a file called constraints.txt and then do pip install -c constraints.txt project. And voilà, you pip install hopefully works now.

Making things future proof

Next we will take things a bit further, we’ll download every dependency we need and include it in our repo. That way we won’t be sad if the maintainers suddenly decide you can’t have an old version anymore.

First, on a working install, do a pip freeze to get a list of things you need. Put these into for example freeze.txt (you might be able to use your requirements.txt directly, but not if you use -e .).

Then run

pip download -d deps/ -r freeze.txt -c constraints.txt

And boom you will have a bunch of whl files in your deps/ folder.

Next we need to tell pip to use them and not rely on some index. Check that things work with

pip install -c constraints.txt --find-links ./deps/ --no-index .

You probably need to do this inside a clean docker or something to be really sure the project will install from a “clean” install.

Then commit the deps/ to the repo and never worry about missing or broken dependencies again.

Note that the .whl files are platform specific, so you might want to look at the --platform option to pip --download if you want to be really sure you have the right files when you need them. Another option is to download the source, but compiling python packages can be tricky (even the –help for pip download says so).

Importing data from WikiData into Google Sheets with IMPORTXML

published on

Here comes another tip for leveraging one of the most important inventions in the 20th century, the spreadsheet1.

Say you have a list of the number of GDPR fines and the country where they were issued, and you would want to know what the fine/population ratio is.

The easiest and quickest way be to go to Google it, and copy/paste the first table you find into a new Sheet. Then using the =VLOOKUP function to grab the population for each.

But I got curious if there would not be some more automated way to do this. And turns out there was, in 10 years ago Google Spreadsheets had more or less this exact function, =GoogleLookup("entity" ; "attribute") but it was deprecated in 2011 (probably for being too useful… or more likely abused somehow).

Luckily, there are still a few ways to import data into Google Sheets programatically, using =IMPORTXML, =IMPORTDATA, =IMPORTHTML and some third-party solutions like =IMPORTWEB.

Now, when we have a way to import data, we need to find a good place to import the data from. Preferably a place which has all kinds of data, so we can reuse what we learn in this case to programatically fetch more complex data next time. Wikipedia seems like a good candidate, and it turns out there is a project called Wikidata, which aims to provide the knowledge stored on Wikipedia in a more structured format.

Getting data out of Wikidata is not that straightforward though, to represent the data, they use a graph format which you can query using a language called SPARQL. They do a much better job at teaching in on their site, and I’d recommend starting with this tutorial if you are interested.

After you have figured out your SPARQL query you can import it directly into Google Docs by copying the query URL and giving it to =IMPORTXML, and then pass it a XPATH to extract

The full =IMPORTXML command will look something like this

=IMPORTXML("https://query.wikidata.org/sparql?query=%23%20defaultView%3ABubbleChart%0ASELECT%20DISTINCT%20%3FcountryLabel%20%3Fpopulation%0A%7B%0A%20%20%3Fcountry%20wdt%3AP31%20wd%3AQ6256%20%3B%0A%20%20%20%20%20%20%20%20%20%20%20wdt%3AP1082%20%3Fpopulation%20.%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22%20%7D%0A%7D%0AGROUP%20BY%20%3Fpopulation%20%3FcountryLabel%0AORDER%20BY%20DESC(%3Fpopulation)","/*/*/*[name()='result']","utf8")`

Two important things to note about this. First, the SPARQL result is name-spaced. You can see it from the xmlns part in the beginning.

<?xml version='1.0' encoding='UTF-8'?>
<sparql xmlns='http://www.w3.org/2005/sparql-results#'>

This means you need to select that namespace before you can run queries like /*/*/result, but the =IMPORTXML command does not (as far as I know) allow you to do it. A workaround is to use Xpath functions which search all namespaces, like /*/*/*[name()='result']. 2

Another thing to consider is that the SPARQL response or specification gives no guarantees for which order the columns are. So, you might get back <binding name='countryLabel'> first or second within the result. This is annoying as for =VLOOKUP to work the key needs to be to the left of the value you are looking up.

A workaround I stumbled upon is to add a ORDER BY DESC population, that will cause the columns to be ordered as listed in the query.


  1. My personal opinion but I bet someone else also agrees. Spreadsheets excel(pun) at leveraging what computers are best at. [return]
  2. This caused quite some confusion for me, because because for example xpather.com does not take into account the namespace, so even if it worked there it did not work in the google sheet. Better to instead use https://extendsclass.com/xpath-tester.html which does require the correct namespace (or a function which searches all namespaces). [return]

Using WWWOFFLE to save a modern webpage for later

published on

Every so often when you want to archive a webpage, you notice it’s full of dynamic content and javascript which won’t easily be archived. I was recently looking to archive a matterport 3D image. This is a typical website that won’t easily save using normal web-archivers, as it relies on javascript to dynamically fetch images as you move through the 3D space.

One generic solution to capture something like this is to use a proxy in the web browser and save everything that passes through it. But most proxies only cache things for a limited time and respect headers like no-cache1. But if the proxy would ignore that and store all requests that flow through it indefinitely, you can maybe create a “snapshot” of a website by browsing it trough this archiving proxy.

Turns out I am not the first one to come up with this idea, there are at least two tools out there which do this. The first one I tried was Proxy Offline Browser, which is a Java GUI application which does this. It worked quite well, but the free version does not do TLS/HTTPS. The Pro version is only 30 euro, but I was curious to see if there was any open-source solution that could do this.

Turns out there is, it’s called WWWOFFLEand it has a lovely compatible webpage. After some trying, I got it working, and I’ll describe rough outlines on how to get it working here. Note though, if you value your time or don’t feel like fiddling around in the terminal, I do recommend just paying 30 euro for the Proxy Offline Browser and be done with it.

Steps for getting it working on OS X

First you need to download wwwoffle source code and ensure you have GNUTLS headers and libraries, so you can use it for HTTPS.
Then compile it with

./configure --prefix=/usr/local/Cellar/wwwoffle/2.9j/ --with-gnutls=/usr/local --with-spooldir=/usr/local/var/run/wwwoffle --with-confdir=/usr/local/etc/
make
make install

Then run it

wwwoffled -c /usr/local/etc/wwwoffle.conf -d

Now there is a few more steps before you can start archiving.

First reconfigure your browser2 to use wwwoffle as proxy. Then visit https://localhost:8080 in the browser to get to the wwwoffle page. Using this page, you can control wwwoffle and see what it has cached.

First, you will need to get the CA certificate, so you won’t get SSL warnings all the time. Go to http://localhost:8080/certificates/root, download and install it.

Then you need to put wwoffled into online mode, which you can do here http://localhost:8080/control/

Then configure wwwoffled itself, which you can do using the built-in web-based configuration tool.

The settings to change are

http://localhost:8080/configuration/SSLOptions/enable-caching to yes

and

http://localhost:8080/configuration/SSLOptions/allow-cacheto allow-cache = *:443

That should hopefully be enough. Now try browsing some website. Then go to the control page and put wwwoffled into offline mode. Hopefully, you should still be able to browse the same page, using the cache.

Additionally, I had to add

CensorHeader
{
 Access-Control-Allow-Origin = *
}

To http://localhost:8080/configuration/CensorHeader/no-nameto ensure AJAX3 requests worked in some cases.

If you run in to other issues, you can either start debugging or go back and cough up the money :-)


  1. which seems to be standard practice nowadays even for things that should definitely be cached [return]
  2. I recommend using another browser than your main one for this to keep things separated. On OS X I’d recommend Firefox as it keeps it’s trusted CA’s separate from the OS’s so you won’t need to have your whole computer trust the newly minted CA certificate. [return]
  3. yeah I’m old [return]

Farewell C1

published on

Yesterday in a datacenter somewhere in France there was suddenly an eery silence as the last remaining racks fell silent for the first time in a long time. As of yesterday, 1st of September 2021, Scaleway turned off their C1 ARM servers.

I know because I still had one trusty little C1 server until today, a server I have had since it was brought online 7 years ago. It was never the fastest, or the biggest server I’ve had, but it was my little dedicated server. It never complained, never crashed, never rebooted, just kept running, serving my homepage and some side-projects.

History of C1

If you are not familiar with the C1, and why it deserves its own little obituary, then let me give you a bit of backstory.

The C1 was introduced around 2015, first as a free 15-minute trial at labs.online.net 0 and then launched as a commercial product under the brand Scalewayarchive.is/scaleway. The C1’s were an interesting take on the virtualisation market, instead of cramming in as many virtualised hosts on a powerful machine, they crammed in as many tiny SOC’s they could into a rack. They built a tiny used custom SoC’s backed by a shared disk storage. A bit like a cloud-hosted Raspberry Pi, but with a network attached SSD disk. On a public and static IP with good connectivity.

What’s the big deal?

There is no big deal, for most people or project a virtual server will do just fine or even be a better choice than dedicated hardware. But there are a few reasons I like small dedicated servers. One is that with a dedicated machine, you can be sure that you are always getting the same performance(barring running multiple things on it). Virtual servers might be faster in bursts, but they are generally oversubscribed and if you are unlucky, you might have very varying performance depending on how busy your neighbours are. If it’s fast now, it will be as fast tomorrow. The C1 was never really fast, though, which I took as a fun challenge. I know if I could get X and Y working well on this limited machine, then if I ever need to scale it up it will be extremely fast on a top of the line server.

It’s more secure

Maybe, in theory at least. For a VM, when doing threat modelling, you should always consider the risk of someone else on the host escaping their VM and accessing your VM and files. Back in 2015 there had been a few VM escapes, but the future would bring many more and a whole new range of side channel attacks against shared processors or memory. My little dedicated ARM server never had to worry about Spectre, Meltdown, Rowhammer or any other of the processor bugs which has rattled the whole VPS ecosystem. 123. Being able to just go “oh, that’s interesting” when there is news of a new Spectre-like attack without having to even consider my little C1 loosing performance or needing to be rebooted was quite nice.

Another benefit was the 4 GiB of RAM, in 2015 that was unheard of for a €2.99 server (and it is still today I think). And that is “dedicated” ram. Which can’t as easily be accessed by the provider, which is important if you care about it. Although I bet, if Scaleway wanted, they could figure out some way to read it out using something like pcileech. [ Update-2021-09-06: I was informed by one of lead designers that it was designed with attacks like these in mind, so at least any physical attack would not have been straight forward ].

What next?

Life goes on, except for the C1. I do wonder what will happen to them. Maybe they will end up on a flea market somewhere. I am not ready to move my personal things to a VPS just yet, but there are not that many cheap dedicated alternatives out there. The only one (I’ve found) at that price point is Kimsufi, but they are mostly out of stock and lower specced.

In the end, I decided to stay with what I know and stay with another of Scaleway’s dedicated offers, an Atom C2350which has served me well for testing and I have now migrated everything to.

Persistent login to OpenWRT luci

published on

Sometimes, if you are logging in multiple times per day, the default 1 hour session time tied to a browser tab/window might be a bit annoying.

To increase the session time to for example 1 month 24 days1, you need to do

uci set luci.sauth.sessiontime=2147483
uci commit

But it’s still set as a session cookie, to fix that, you need to modify /usr/lib/lua/luci/dispatcher.lua and change the line which begins with http.header("Set-Cookie",. You need to insert Max-Age= to make it a persistent cookie. Like so

http.header("Set-Cookie", 'sysauth=%s; Max-Age=2629746; path=%s; SameSite=Strict; HttpOnly%s' %{

Then you need to clear the luci-modulecache or reboot

rm -rf /tmp/luci-modulecache/

There, if you re-login on to luci you should now have a persistent cookie which will persist for one month. To remove it, press the logout.


  1. Update 2021-06-12: After locking myself out I figured that on a 32 bit system you can’t set this to anything higher than a 32 bit signed integer, this seems to be a ubus limitation [return]

Backing up you VM with borg

published on

Recently, for no specific reason at all I did a review of my backup plans of my tiny personal VM:s I have.

Octave Klaba tweeting about the fire at OVH

As my disaster recover plan was mostly “I hope they don’t lose it all at once” I decided to upgrade it to “I have some backups, so I don’t lose it all at once”.

To keep things simple and as I love micro optimising to see for how cheap I can get my personal VM’s, I decided to use my home NAS for backups instead of just paying for third-party storage like B2.

So, here are a rough1 overview of how you can use a local Linux NAS as destination for backing up a cloud VM.

Turris Omnia

First we need2 to get borg working on the turris. Luckily the Turris has lxc, so we can just spin up an alpine instance and do apk add borgbackup and apk add openssh-server. Then update the network to none to share the host network and mount any disk you want.

# first comment out any other network
lxc.net.0.type = none
# bind-mount /mnt/sdb2/dir
lxc.mount.entry = /mnt/sdb2/mydir /mnt/sdb2/lxc/borg/rootfs/mnt/mydir rw,bind 0,0

I decided to use a separate ssh inside the lxc for a bit of additional sandboxing.

Add the following to authorized_keys to allow the server you want to back up to run borg, but nothing else.

command="borg serve --restrict-to-path /mnt/server-bakups",no-port-forwarding,no-agent-forwarding,no-pty,no-X11-forwarding ssh-rsa AAA...

C1 Server

Time to start backing up, first because C1 is an armv7 instance, download arm binaries from https://borg.bauerj.eu.

Then check that you can connect to your Turris and get some borg output back from the limited ssh-key. Similar to below.

a example of borg backup output

If that works you can initialise the repository and start backing up according to the borg instructions

Something like this

borg init -e=repokey ssh://root@100.127.112.32:40022/mnt/mydir/myserver

and if that works

borg create ssh://root@100.127.112.32:40022/mnt/server-bakups/personal::{hostname}-{user}-{now} /home /etc /var/log

And if that works, then either call it a day or address the obvious issues like running the receiving borg as root :-)


  1. This is a very rough guide, it will not work without modifications so don’t try to just blindly copy and paste the instructions. These instructions are specific for Turris Omnia with tailscale and a Scaleway C1. [return]
  2. you can probably ignore this or restrict ssh some other way, but I did this because I started out from the other direction, trying to get borg running on the NAS, and it would then reach out to the servers. [return]

Conditional access using only nginx

published on

Have you ever wanted to deploy a website to test that it works, without everyone else being able to see it?

If you are using a dynamic language or CMS for your webpage (PHP, Wordpress or Ruby on Rails) there are straightforward ways to accomplish this.

But what happens if you have a static webpage? Here I will present one solution using only a nginx config file to accomplish this.

# first we need to allow access to the soon.html
# and also a logo which is linked from the soon.html
# if your soon.html links more resources in this server
# you need to update the regex to match that also
location ~ /(soon\.html|images/logo_white.png) {
    try_files $uri =404;
}

# this is the secret way to get past the block
# it will set a magic cookie with a lifetime of 1 month
# and redirect back to the host  
location /iwanttobelieve {
  add_header Set-Cookie "iwantto=believe;Domain=$host;Path=/;Max-Age=2629746";
  return 302 $scheme://$host;
}

# this is the normal serve, but with a condition that everything
# everyone that does NOT have the magic cookie set will be served
# the content of soon.html
location / {
if ($http_cookie !~* "iwantto=believe") {rewrite ^ /soon.html last; }
	try_files $uri $uri/ =404;
}

That it! Copy and paste the above into a server {} block. Make sure to take not of the order though to ensure you don’t have anything else before this which would take precedence. Then change all occurrences of soon.html if you use something else. And remember that the first match needs to match everything that this soon.html tries to reference, otherwise they will just get back the content of /soon.html for all other requests.

Note that if is a bit finicky in nginx, check their documentation for more details.

Usability > Security

published on

Introduction

The other day I wanted to use my noscript.it with one of my old iPhone 4S running iOS 6, but I was met with “could not establish a secure connection to the server”.

Screenshot of safari showing error Turns out it was because I had, out of habit, configured the server with a “modern” list of TLS ciphers. And the poor old iOS 6 didn’t support any of them.

So, I went on a mission to ensure noscript.it works with as old devices as possible.

It turns out enabling TLS1 and TLS1.1 on Ubuntu 20.04 is a bit harder than I expected1. Luckily someone else solved it already.

So now, after using the old mozilla SSL config and appending @SECLEVEL=1, it works. Even on my vintage iPhone 3G. Hurray!

Screenshot of NoScript on iPhone 4S

Wait what?

But, I hear you say, isn’t this less secure? I mean now you only get a B on the Qualys SSL Report! Clearly this is bad!?

Screenshot of Qualys SSL resultsLet’s take a step back and think about what the score actually means. noscript.it automatically gets a B because it supports TLS1. But let’s go one step further and assume we’re looking a bank with a C2. A site gets a C if it supports SSLv3, meaning it is vulnerable to the SSLv3 POODLE3 attack. This is clearly bad for a bank!? Or is it? How likely is it that someone will successfully execute this attack, which requires the attacker to have the ability to intercept and modify the data transmitted. And compare this likelihood with how likely is it that someone will need to access the bank website from an old XP (pre-SP3) machine only supporting SSLv3? The second seems more likely to me.4

Okay, you say, but won’t keeping SSLv3 around make everyone vulnerable because of downgrade attacks? If that were the case, the risk calculation would be different. But luckily, we have TLS_FALLBACK_SCSV to avoid that. TLS_FALLBACK_SCSV ensures that modern client and browser won’t risk being fooled to downgrade its encryption.

Conclusion

So to wrap things up, don’t stare blindly at the rating or certification. A site with A++ is more secure than one with a C rating. But if you (or someone less fortunate) can’t access the site when they need it, it will be a pretty useless site. Personally, from now on, unless the site needs 5 absolute security, all my projects will optimise for compatibility rather than getting an A++. After all, it is much more likely someone will try using it with a Windows XP or old Smart-TV compared to someone MITM-ing that person at that moment.

Disclaimer

Please note though, don’t read this as an argument against doing things securely as default and following best practices. Rather it is just some thoughts on this specific issue of TLS and SSL configurations. If you break with best practice, make sure you understand the reason why it’s best practice to begin with and what risks or weaknesses you introduce by not following them.


  1. I 100% support secure defaults, make it hard to do the wrong thing. [return]
  2. A favourite hobby of some people in the security community is to publicly shame banks or websites for not getting a great grade in the Qualys SSL test. Here is an example from Troy Hunt [return]
  3. SSLv3, not to be confused with the POODLE attacks against some weak TLS implementations. [return]
  4. Also there are mitigations which you should of course implement unless it breaks more than it fixes. [return]
  5. in some cases, like highly sensitive data or targeted users the risk calculation will be different, it might be better for a user not to be able to use the site rather than risk some known (or unknown) attack. [return]

Categories

Adblocking (1)

Css (1)

Distractions (1)

Gpt (9)

Gptaoc2022 (9)

Linux (1)

Llm (9)

Microblog (2)

Php (1)

Security (4)

Servers (2)

Spreadsheets (1)

Sysadmin (2)

Tech (21)

Web (6)