Systems Engineer · HPC · AI · Ramblings on whatever
text/gemini format, rendered to HTML. For the best experience, view through a Gemini browser (like Lagrange). Atom feed available. Around December, 2023. I had a quick talk with a chip maker about porting and what hardware features is needed for fast LLM inferencing - due to my work on porting llama.cpp to the RK3588 NPU. I started writing this post as the condensation of my view and recommendations. But.. I got busy and forgo...
Read more →It's probably not difficult to guess that I own several computers. It's a part of my job and there's no real way around it. Remember, the cloud is just someone else's computer. Today as I'm walking through a local mall, I saw an extension cord with a power meter on it. You know what, that's a good ...
Read more →I've looked into alternative AI accelerators to continue my saga of running GGML on lower power-consumption hardware. The most promising - and the only one that ever replied to my emails - was Tenstorrent. This post is me deeply thinking about if buying their hardware for development is a good inve...
Read more →Today is the last day of CNY and being honest, I have nothing to do. Out of nowhere, I decided to look deeper into RK3588's NPU performance characteristics. To figure out what it actually needs to be performant. Like how batch size and native/normal layout affects the performance. I haven't done an...
Read more →This post is my reply to some ideas in Roderic's post on altesq.net. It got me interred but I was abroad during CNY and I don't have much time to actually write down my thoughts. This post should have gone out a few days back. But that just gave me more time to think through my words, eh? By no me...
Read more →Bla bla bla.. I'll ignore the introduction to LLMs since everyone knows what they are. Even my mom asked me about them. The question I want to propose is: are we seeing diminishing returns on large language model scaling? The key observation that kick started GPT-2 evolving to GPT-3 was that with ...
Read more →2023 was horrible. It was HOT. This made me very worried about global warming. And I've gone through the 5 phases of grief. Denial, anger, bargaining, depression and acceptance. Well, not acceptance but the realization that I am not a part of the problem due to my (lack of) habits. And that I have ...
Read more →I'm invited to give a talk about porting Piper to RK3588 at local group. The talk is given in Chinese. But I've translated the slides into English. And both version are available at the end of this post. For reference, the slides are about a previous post of mine. There will not be any video reco...
Read more →On, December 16th, 2023, I got my shipment of Framework 13 that I ordered 3 days ago. Wow that's fast! But getting it working to my needs was another story. Things just.. go wrong at every turn. I don't blame Framework, I do have some specific needs that I recognized I will have to solve myself. I...
Read more →I was a fan of XKCD 936, correct horse battery staple. I agree that passphrases are much stronger then passwords. But there's a masive usiablity concern with passphrases. Namely, entering them is a pain in the butt. Recently I replaced one major password I use with a passphrase. Just 4 words long...
Read more →I wasn't planning on this being my 2024 resolution. I am not overly certain why I made the title. Must have felt like it last night. Nevertheless, this is the mood I'm in right now, at the end of 2023 going into 2024. Many things happened this year. A lot that I dislike. From climate change gettin...
Read more →Ho ho ho. Happy Hollidays! With Rockchip releasing rknn-toolkit2 1.6.0, the feature set becomes more and more complete. In this release, it's enough to be used to accelerate the Piper text to speech system. I want to document what I've done to make it work, what's my vision for it and what's more t...
Read more →Quick documentation for my self. Today, I dabbled into accelerating TTS using the RK3588's NPU. It works really well! I'm seeing a real time factor (RTF) of 0.15 during my initial tests, and I believe I can push it even further. One thing I had to do was to use dynamic input shapes. RKNN tradition...
Read more →Recently Rockchip released a new version of the RKNPU2 SDK. It enabled larger matrix multiplcation of up to K=10240 and int4 support. My last post describes briefly how I build the RKNPU2 backend for GGML. This time, I want to share what I am able to achieve with the new SDK release. Before I sta...
Read more →Here's the post where I justify buying myself a new laptop. I know exactly 0 human on the internet cares. But dedicating a post forces me to actually think about it and to not buying crap I don't NEED (that's the key word here). Even if no one reads it, publishing it on the internet feels like some...
Read more →So, I came across this Simon Michaux guy on YouTube yesterday. He is a geologist and claims that the energy transition is doomed to fail because of the lack of (rare earth) metals. Off by multiple orders of magnitude. His presentation was convinvcing to me at first. WHile I have some but very limit...
Read more →This post is not my usual rambling about tech. But I want to share my frustration with how I can't help with climate change. First of all. I'm not virtual signaling, but I think I'm likely one of the least consumerist people in the population - I host my blog on Gemini. Boring jokes aside, I truly...
Read more →I left my last job by the start on November, 2023 and have a week of free time before I join the new one. With time on hand, I decided to finish a feature request I made years ago in Drogon. A HTTP/2 client - And server, but that'll happen later. I never expected to get it almost finished in like 5...
Read more →I'm drunk on a Friday night again. This time I decide to investigate how to use OpenBSD's pledge(2) and unveil(2) to enhance the security of the Drogon web application framework. Drogon in written in the C++ programming language, not known to be safe. But in truth, as a maintainer of Drogon, non of...
Read more →This weekend, after a night of partying with my friend and somehow ending up hanging out at a near by McDonald. Back to home, I picked up my old work of running LLMs on the Rockchip RK3588's NPU. Last time I hacked around directly withing GGML and running the RWKV model. That was quite a failure, s...
Read more →Very quick and short one. I was trying to make something with LangChain. I already had a server running LLaMA 2 Chat using llama-cpp-python. It happens to provide a OpenAI like API. There must be a way to abuse the code to make it work with LangChain. Yet half and hour of Googling turned up nothing...
Read more →Want to quickly document how I got my PineTab 2 to play 1080p videos smooth(-ish) with hardware acceleration. With the defaul DanctNIX image, it was quite easy. Simply install `mpv` and ` ffmpeg-v4l2-request-git` from the AUR. Then, pass pass `--hwdec=drm` to `mpv` and you're good to go. This ...
Read more →I cam across two articles on Gemini, discussing what would happen if Chinese switched to using pinyin instead of characters. I want to share my thoughgts as a native speaker. First, I don't user pinyin to type Chinese. I'm from Taiwan and we use another system called Bopomofo(注音). I agree with...
Read more →Not long after my last benchmarking attempt. Rockchip releases a SDK update that fixes the crashing matrix multiplication API. Now I'm no longer restricted to using ONNX. Now I can directly do matrix multiplication from C! And now I can do an apple to apple comparison with OpenBLAS. That's benchmar...
Read more →I wasn't expecting this to be so hard. I ordered an RK3588 based Orange Pi 5+ a while back as a development platform. It's great. A good combination CPU and GPU power. As well as a very competitive price to performance ratio. I also bought a large heat sink for it. However it kept hitting thermal l...
Read more →My goal with my RK3588 dev board is to eventually run a large language model on it. For now, it seems the RWKV model is the easiest to get running. At least I don't need to somehow get MultiHeadAttention to work on the NPU. However, during experimenting with rwkv.cpp, which uses pure CPU for infere...
Read more →My recent job have been developing a very fast TTS system for our internal use. I can't share too many details about it because NDA. But I do want to share some findings while messing with our system. - My boss is ok with me sharing these findings. We are looking to open source it soon anyway. It's...
Read more →Just a few minutes ago, my fried sent me an encrypted ZIP file and says "This ZIP file contains some artwork, but the password is distributed in a past, live event. And it's the birthday of the artist. 8 digits. Can you crack it?" Apprantly he can't find the password. "Sure" I said. I've learned in...
Read more →Quick announcement. I'm running a publically accessible GNUnet hostlist at `http://clehaxze.tw:9876/` due to me failing to bootstrap 3 new nodes today. I was experimenting and this issue really annoyed me. I don't know what happened with the default one. `https://v15.gnunet.org/hostlist` is down. B...
Read more →Armed with experience converting scikit-learn to RKNN. I spent the past few days trying to get Waifu2x (Anime image upscaling) and endup defeated. Either RKNN got stuck submitting commands or I get the mysterious message `W RKNN: [07:11:36.064] Output(Deconvolution2DFunction_0): size_with_stride la...
Read more →