Systems Engineer · HPC · AI · Ramblings on whatever
text/gemini format, rendered to HTML. For the best experience, view through a Gemini browser (like Lagrange). Atom feed available. I have been bothered with a problem on my laptop, an Framework 13 AMD - Steam and every electron applications (VSCode, Element, electron itself, etc..) won't run on Wayland. They run but hangs without creat. I can run Steam with X11. But it fails to run every game. No matter which version of Proton...
Read more →Recently I got a message from Tenstorrent's community manager for helping with improving the installation documents. To make it easier for everyone. While that is still in progress, I wanted to document how I got my Tenstorrent card running on Arch Linux (since Tenstorrent officially only supports ...
Read more →I was invited to give a talk at National Sun Yat-sen University (NSYSU) in Kaohsiung, Taiwan on April 26, 2024. The talk was about AI inference, invited by my Prof for master's degree. I totally forgot to upload the slides, so here they are. The sldies are made with Revel.js so you'll need a web s...
Read more →I've been wanting to replace nlohmann/json with something else in my codebase for a while now. Recently Glaze entered my radar and I decided to give it a try. Here are my thoughts after writing a PoC program comparing the two libraries. Glaze beats nlohmann/json. Just look at the numbers on glaze...
Read more →I saw Brodie Robertson's video about NetBSD banning AI generated code on YouTube and I wanted to share my thoughts on the topic. I think current AI generated code is fine. Here's why. For simple cases, like printing "Hello, World!" 10 times or centering a div, the code has shown up many, many tim...
Read more →I spent an hour or two trying to make the Lagrange browser more secure on OpenBSD by using the pledge system call. For those who don't know, pledge is a system call that allows a process to restrict the system calls it can make. This vastly reduces the attack surface of the process, and makes it mu...
Read more →I got my Tenstorrent card last week or so, and I set it up and gave it a test drive. My end goal is to develop it's software stack and applications such as it can be used as a replacement for Nvidia GPUs, for cheap and at a lower power consumption. But for now, it's time to get my hands wet and see...
Read more →Recently Purism assounted they are dropping the price of their Librem 5 from USD $999 to $699, slightly higher the original price the Librem 5 launched at. I preordered one 5 years ago when it was new and never received mine. So I wrote an email asking for my device. And received the device in abou...
Read more →I haven't blogged in quite a while. I've been busy on other stuff but they are just non blogable. At least for now. I hope one day I can publish them. Anyway, I was working on some web related code in Drogon and I need to read stdin so the user can control the application. Problem being, C++'s `std...
Read more →I've been trying different Vodka brands for a while now and I've come up with a teir list of my favorites. My perference for Vodka is that it should be smooth and not have a strong taste what so ever. I drink Vodka straight and use it as fuel for writing stupid code that is too hard for me usually....
Read more →I watched the GTC 2024 keynote and I was like. Gosh.. really.. this sounds so .. wrong! You can watch the short 16 minutes version by CENT here. The core accouncments are: The entire keynote gives me a vibe of Nvidia turning into what IBM was - mainframe and proprietary solution. But worse. Al...
Read more →This post is a draft for a lecture I'll be giving later this year at National Sun-Yat Sen University, Kaohsiung, Taiwan. Where I'm a guest lecturer, at a course by Prof. Chang. Whom is my advisor during my college years. Making a slide deck is hard without a script, hence this post where I blabber ...
Read more →I'm working on adding support for QOI images in Lagrange out of boredom. QOI is a very simple lossless image compression format that is comparable to PNG and very fast to compress. This page constains a few QOI images encoded by different encoders to test my integration into the Lagrange browser. I...
Read more →I've been a advocate to reduce consumption to the absolutly minimum ever since I started considering climate change a crysis that needs addressing immidiatelly. I didn't get why people can buy so much clothing that their closets are full of clothes they never wear. But a order I made half a year ag...
Read more →CORS, the problem that every web developer has faced at least once in their life. Drogon does not come with built-in middleware to handle CORS, but it is easy to implement it yourself. Usually, you'll want to expose your APIs to the world. This code will allow any request to the `/api` path to go ...
Read more →Around December, 2023. I had a quick talk with a chip maker about porting and what hardware features is needed for fast LLM inferencing - due to my work on porting llama.cpp to the RK3588 NPU. I started writing this post as the condensation of my view and recommendations. But.. I got busy and forgo...
Read more →It's probably not difficult to guess that I own several computers. It's a part of my job and there's no real way around it. Remember, the cloud is just someone else's computer. Today as I'm walking through a local mall, I saw an extension cord with a power meter on it. You know what, that's a good ...
Read more →I've looked into alternative AI accelerators to continue my saga of running GGML on lower power-consumption hardware. The most promising - and the only one that ever replied to my emails - was Tenstorrent. This post is me deeply thinking about if buying their hardware for development is a good inve...
Read more →Today is the last day of CNY and being honest, I have nothing to do. Out of nowhere, I decided to look deeper into RK3588's NPU performance characteristics. To figure out what it actually needs to be performant. Like how batch size and native/normal layout affects the performance. I haven't done an...
Read more →This post is my reply to some ideas in Roderic's post on altesq.net. It got me interred but I was abroad during CNY and I don't have much time to actually write down my thoughts. This post should have gone out a few days back. But that just gave me more time to think through my words, eh? By no me...
Read more →Bla bla bla.. I'll ignore the introduction to LLMs since everyone knows what they are. Even my mom asked me about them. The question I want to propose is: are we seeing diminishing returns on large language model scaling? The key observation that kick started GPT-2 evolving to GPT-3 was that with ...
Read more →2023 was horrible. It was HOT. This made me very worried about global warming. And I've gone through the 5 phases of grief. Denial, anger, bargaining, depression and acceptance. Well, not acceptance but the realization that I am not a part of the problem due to my (lack of) habits. And that I have ...
Read more →I'm invited to give a talk about porting Piper to RK3588 at local group. The talk is given in Chinese. But I've translated the slides into English. And both version are available at the end of this post. For reference, the slides are about a previous post of mine. There will not be any video reco...
Read more →On, December 16th, 2023, I got my shipment of Framework 13 that I ordered 3 days ago. Wow that's fast! But getting it working to my needs was another story. Things just.. go wrong at every turn. I don't blame Framework, I do have some specific needs that I recognized I will have to solve myself. I...
Read more →I was a fan of XKCD 936, correct horse battery staple. I agree that passphrases are much stronger then passwords. But there's a masive usiablity concern with passphrases. Namely, entering them is a pain in the butt. Recently I replaced one major password I use with a passphrase. Just 4 words long...
Read more →I wasn't planning on this being my 2024 resolution. I am not overly certain why I made the title. Must have felt like it last night. Nevertheless, this is the mood I'm in right now, at the end of 2023 going into 2024. Many things happened this year. A lot that I dislike. From climate change gettin...
Read more →Ho ho ho. Happy Hollidays! With Rockchip releasing rknn-toolkit2 1.6.0, the feature set becomes more and more complete. In this release, it's enough to be used to accelerate the Piper text to speech system. I want to document what I've done to make it work, what's my vision for it and what's more t...
Read more →Quick documentation for my self. Today, I dabbled into accelerating TTS using the RK3588's NPU. It works really well! I'm seeing a real time factor (RTF) of 0.15 during my initial tests, and I believe I can push it even further. One thing I had to do was to use dynamic input shapes. RKNN tradition...
Read more →Recently Rockchip released a new version of the RKNPU2 SDK. It enabled larger matrix multiplcation of up to K=10240 and int4 support. My last post describes briefly how I build the RKNPU2 backend for GGML. This time, I want to share what I am able to achieve with the new SDK release. Before I sta...
Read more →Here's the post where I justify buying myself a new laptop. I know exactly 0 human on the internet cares. But dedicating a post forces me to actually think about it and to not buying crap I don't NEED (that's the key word here). Even if no one reads it, publishing it on the internet feels like some...
Read more →