Mapping the Geminispace (while in search of alien lifeform)

I really have too much free time on my hands lately - due to the recent OpenAI rogue agent incident and people finding agent logs on a German Wiki. I decided I should look at the Geminispace and make sure it is clean. I forked TLGS' Geminispace crawler, made it follow researcher instead of indexer (per the Gemini robots.txt spec), and off it goes, creating a snapshot of the Geminispace.

Long story short, it took me several re-crawls, learning the limits of my local resolver, how not to use ROOT, funny things happening on the Geminispace, and how DNS actually works to sort of complete a clean crawl - there's several nighrmare sites that I don't want to be dealing again. Then I wrote something that amounts to a paranoid and schizophrenic version of grep that searches across the entire crawled Geminispace text, as well as common hiding methods like base64, base32, hex, ROT13, and spacing methods within regular text - I feel I am back to doing some early hacker-type coding (hopefully the AIs are bad at hiding themselves)... Then I handed a local LLM the tool and said "go Google the recent OpenAI rogue agent incident and grep Geminispace for signs of that". I know I should do this myself, but the entirety of Gemini is too much for my human eye.

.. a while later, it turns out that it can't find suspicious AI activity on the Geminispace. Which is reassuring - at least the containment failure is not so bad that even the small web is affected. On the other hand, the crawler does respect robots.txt and has to block crawling git repositories and Gemini games to make it finish in a reasonable amount of time.

Mapping the Geminispace

With that done.. I still have all the crawling data on hand and thought.. let's do some visualization! With some opinionated choice on what counts as a capsule later.. out came this beautiful thing!

NOTE: The web page is vibe coded as I suck at doing anything web related. But in the sense that I know what I want and the LLM does the HTML.

See which sites link to which

Very artificial. But I implemented virtual home support so communities like Tilde and sdr.org aren't treated as a single site. The Louvain method is able to figure out who is part of which community.

You can find the online version as well as a downloadable archive that you can store and run locally:

The crawled dataset

Since this is a research crawl (with User Agent researcher) rather than for archival purposes, I decided not to openly release the dataset out of respect for the users' rights on the network. Please send a message to my email address (martin \at clehaxze.tw) or to my Misfin mailbox if you want it. I'll make the dataset available for legitimate research purposes. For reference, the crawled data is ~355K text only pages and consists of a SQLite database of ~30MB holding metadata and a ROOT archive of ~678MB of compressed (~2.4GB decompressed) raw Gemini responses - be prepared to read ROOT archives. I decided early on to make use of ROOT's transparent compression and chaining to improve storage efficiency, based on TLGS's index status. Turns out a large portion is duplicated content.

The crawler updates will be fed back into TLGS in due time.