candycup.dev@gmail.com

Running a spying operation on a large minecraft server - 22/11/2024

'Stalking Others' on the Lifesteal Public server

Minecraft account StalkingOthers standing on a Minecraft server spawn at night

Some parts of this blog post are technical in nature. Beware.

Meet Lifesteal Public - one of the largest Factions servers of the era of Modern Minecraft. Officially referred to as the Lifesteal Network - which I will be referring to just ‘LSN’ from here on out.

The server averages several hundred people online at all times, with a last-season peak of well over a thousand people. This is to preface that there’s a lot of interest in the server. And from me as well - I was the server’s first admin up until early may of 2024! Spot me in this screenshot:

i spoke too soon pic.twitter.com/VXOP7Pelfh

— Reddoons (@reddoons) December 9, 2023

Either way - one thing LSN isn’t very well known for is making well-balanced economic changes to counter inflation and other nasty side products of running an economy of this size. It’s a hard job. A lot of these problems were caused by the fact that according to some off-hand statements made by the server’s owners and management - it almost seemed like LSN wasn’t actually properly tracking many aspects of their economy - such as what items people were selling and where all that cash was coming from. Seems like they were eyeballing their controversial balancing changes. Uh oh!

That does spark the question: could nerds do what lifesteal wouldn’t? Could an outside entity run an economic monitoring operation without the input of server developers or staff?

LSN has a fairly standard coin economy. Players get coins from selling items. There’s also teams/factions, that get power from placing down spawners. Fairly standard factions stuff. LSN has a public leaderboard for the top 225 richest people in coins. They also have a public leaderboard of teams - which uses the exact same system as the coins baltop - except instead of showing only the most 225 most powerful teams - it showed every single team? Just a completely random side note there. Both leaderboards were available as guis by doing /baltop and /teamtop respectively.

Lifesteal Network's baltop display is open in the image - showing the richest player, who has around 5 billion coins.

an economic forum

To aid me in this bizarre quest, on May 20th 2024, I founded the Lifesteal Economic Forum alongside with a fellow contributor Lonody.

We brainstormed potential options for monitoring the economy. Most of the ideas revolved around using an autonomous drone account sitting at the server spawn.

One issue - that would be like super against the rules. I wasn’t willing to burn all the existing good faith I had with the server’s staff and owners. We did have one other idea for observing the server’s economics, however.

What if we created a client-side Fabric mod for Minecraft that basically just quietly observed your gameplay on the sidelines while the player just played the game? If you opened the baltop using /baltop in-game, the mod would quietly observe the visible data and forward it to our internal tracking server via an API.

This would basically mean that we’d only have tracking coverage when someone with the tracking mod was online. We’d only know what the baltop looked like at a given time if someone with the mod happened to open the baltop at that time. There are a lot of limitations here.

Hey, good enough, right? Let’s get to work. Just one thing to get out of the way first:

a professional courtesy

So, we’re going to be extensively tracking the server’s economy. Technically speaking, we weren’t breaking any of the server rules. But we still wanted to check with the server owners.

So, we DM’d the one that was probably more likely to reply to me personally.

A screenshot of me messaging the server owner Reddoons: "hello big man, sorry to disturb your economy balancing ponder session, but thought i'd hit you up on a project idea that i kinda just need your permission on."

sidenote: we did not ask the server staff. the LSN staff was quite notorious with their ticket wait times at the time. sorry!

So we asked. We presented our case in a way that made it look like we were the warriors of good - protecting the economy from admin abuse (by tracking the balances of individuals and other public finances of individuals), alongside other things. We told them the truth of what we wanted to accomplish. Well, less of our intentions to commit economic espionage on a Minecraft Server, more like ‘funny graphs of money balances’. We did not mention cosmetic tracking, auction tracking and other feats that we were interested in.

Reddoons giving us the go-ahead

Sick!

PART II: The Tracking of Lifesteal Network

a world of silly gooses

This mod was to be called ‘silly goose’. This isn’t even a clever name. It’s not a pun on anything. We called the mod ‘silly goose’. That’s about the last time i’m going to attempt to somehow justify this naming scheme. No regrets!!

And so the development started. I was the only one in the group with experience in software development, so it was my honor do to the engineering. We cooked up a pretty barebones fabric client mod that intercepted packet data coming from the server to see if it was somewhat relevant to our needs.

This includes incoming chat messages that contained specific series of characters. These will be referred to as System Messages.

Here’s a couple System Messages: A system message indicating that a new user has joined the server

A system message showing a user redeeming a cosmetic from a lootbox

System messages will be the primary way we get interesting statistics about the server

You might be wondering - hey, how would you guys observe the primary issue of the economy, aka where is all that cash coming from? And you’d be right to ask that. Because we couldn’t. This is one aspect that we could never track. Sure, we could track what specific individuals were selling to the npc shops - individuals that were literally running our software on their computers. This wasn’t feasible. We could have the sell data of only a couple people from a server of tens and tens of thousands of people. Won’t bother with that, I fear. The ball is on LSN’s court with this one.

Here’s what we could track!

And a bunch more niche stuff.

This was pretty good. But how did we actually accomplish this?

If the post was too nerdy for you so far, it gets worse! And then it gets better at some point, and then worse again. What do you want from me?

tracking the economy on a technical level

To initiate the tracking system, we manually whitelisted specific user accounts using a token-based auth system. Basically, we’d manually create authentication tokens, like passwords, and grant them out on a person-basis to our contributors. This was to prevent the extremely unlikely edge case of someone obtaining api access and nuking us with garbage data. With the token system, no unauthenticated submissions were accepted, and if they were malfunctioning, we could wipe all submissions sent by a specific access token / individual.

After the contributor configured the mod with that token, we could begin.

When an user actually opens an interesting interface or sees an interesting message in chat, it creates a HTTP POST request to a remote express.js server behind a nginx reverse proxy.

This API then processed the incoming request, and stored the metrics data in our self-hosted Postgres instance.

big day

26th of July, 2024

And as the second season of the server was launching - we were preparing to get the show on the road. We got a couple contributors, one of them being server staff lol, this was a secret operation) to run the mod at launch.

one slight issue - i kind of might have potentially messed up the tracking code - for some reason, clients running the tracking software were sending a LOT more packets to our tracking server than they were supposed to.

A screenshot showing ridiculous log spam, tens of requests from a single client in under a second

Multiple duplicates, batches of tens of requests seemingly out of nowhere. It was fine, we could filter our garbage data by looking at submission times and finding duplicate submissions.

Everything seemed to be working fine! The mod was successfully able to calculate the total net worth of the richest people on the server, completely client-side using the baltop data.

A screenshot of the mod showing the total net worth of the richest people on the server

Here’s what our cloudflare panel looked like after like a week of tracking, if I remember correctly:

A cropped screenshot of our cloudflare panel showing 285k total requests

data!

OK - we have tracking. We’re receiving data. Some hiccups with the authentication system. Disregarding that, we could start visualizing our data! Here’s one of the first visualizations of the server’s economy we made.

An early graph render from Season 2 beta

An early graph render from Season 2 beta

We made a barebones visualizer in python and javascript - fetching the data Directly from the database with python and dumping that into a js file, and then rendering the graph with Highcharts.js. Pretty!

Yet another random graph

We sent one to servre owner Reddoons, who thought the system was dope.

We shared some of the graphs around! As a former admin and current discord bot developer, a lot of people thought that I was abusing my access in some way. In reality, I never had that kind of access at any point in my time working for them. I never got to touch the actual server or its data. We were relatively transparent with how we were getting this data, so those rumors weren’t a real issue.

[graphs] [reactions]

We even sent the server’s head developer some graphs.

Head developer 'BatmanFlatman' with a message saying 'BRUH'

I think we got the owners attention - they DM’d us asking for more stats the next day!

shortcomings of human labor

Unfortunately, our contributors were actually human beings with lives outside the server and did not actually act as drone accounts for our cause. Turns out, our coverage was frankly Awful. People were offline for days at a time, at the worst time, weeks. And it’s not like our selling pitch of ‘pls download our tracking mod Lol’ was really striking the hearts of the people.

Our project started fading. It was no longer providing clean enough data for us to showcase it. Gaps lasting days. Our coverage wasn’t cutting it anymore, and rapidly reaching towards having no coverage at all, and our contributors just not playing anymore.

PART III: The Industrialization of Minecraft Espionage

I just wanted to call the chapter that.

It had been a bit since we worked on the project. We currently had a 2 week gap in coverage.

A screenshot of a discord message of me poorly reacting to the fact that we had a 2 week gap in our system coverage

We’ve established that humans aren’t on the server 24/7. What’s the alternative? As earlier discussed, we really would have liked to utilize bot accounts - but as we mentioned, it was unlikely that the owners would give us a literal exception to the rules. So until we received a magical exemption from the rule of law - we were cooked.

Shortly after, one of our contributors happened to be in a discord voice chat with one of the server owners, Leowook. After some casual discussion about the topic, he dropped an ask:

Hey, so, would it be possible for me to have an alt running like /baltop every minute or something?

Leowook: Uh, Sure, Haha

Just that easy, I guess. The rule of ‘if its funny’ has once again proven to be the primary decisionmaking factor behind LSN. You know this to be true if you’re otherwise familiar with this server’s history.

Designing drone infrastructure

We have a new task at hand. We’re moving away from unreliable human contributor in exchange for a bot that automatically fetches the baltop, teamtop and submits the data back to us.

I sketched up for a while, and ultimately settled on a multi-drone approach, where even several accounts could connect to the drone manager as individual drones to speed up tasks by having multiple accounts to execute our tasks. Technically speaking, we were only permitted to use a single account - but this approach just seemed like the most reasonable and future-proof. This new system was named ‘Atlas’.

The central drone manager, aka the API, handles drone connections and task scheduling. The drone manager could for one, schedule a task for fetching the server’s baltop every 5 minutes. Every 5 minutes, the API checks the list of drones connected via Websocket, and forwards one of the available drones the task to fetch the baltop. Each task is paired with an unique identifier, that a drone must reference upon replying with the data.

The drone is tasked with ensuring the data is proper. The means of actually getting the data are also up to the drone. The API only cares that the data is returned is in the proper format. Upon completing the task, the drone sends a packet via websocket to the drone manager alongside the identifier, and that’s a task completed.

The actual bot is operating on Mineflayer (prismarinejs), a JavaScript-based framework for creating scriptable Minecraft bots. It was good enough for our needs!

https://github.com/PrismarineJS/mineflayer

Each component runs in their own Docker container, pushes to the GitHub page trigger an automatic build process to deploy the updated version on my german VPS.

The API takes the incoming data and stores most of the time-series graph data (such as a player’s balance at a given time) in our self-hosted InfluxDB instance. Influx is a time-series database built for storing metrics data like this - however I do slightly regret using it. The query language is cryptic. It’s not widely used from what I can tell.

Sidetrack:

After designing the drone infrastructure, I stumbled onto a postgres extension called TimescaleDB. Check it out. Seems pretty cool. In later benchmarks, it outperformed Influx on several metrics with the data from Atlas. Postgres optimized for time series data!

The API also stores some raw request data in our PostgreSQL instance.

A random picture of a random algorithmic graph that we intended to use for tracking who was buying auctions from who (this was not public information - lifesteal doesn’t entirely disclose who buys your auction etc)

AH logic graph

Fire away

For this to work, we needed a new Minecraft Account. After carefully considering our options, here were the finalists:

We ultimately decided in favor of one of Lonody’s suggestions, StalkingOthers. And on September 6th, the account was equipped with some random camera skin we found and was sent off to the races.

NameMC page of StalkingOthers

This time, we kept our methods a bit more down low than last time. With Silly Goose, we were relatively transparent with how we were getting all that data. This time, well, we were breaking the rules (kind of). Even although we did have an exemption from that specific one. Because of the community state at the time, we simply did not mention the fact that we had a bot account on the server at all times.

Surely no-one would pay attention to it, either.

A player noticing the account and saying 'StalkingOthers is a wild IGN' in chat

So there it sat. At the server spawn. Took about a day for us to start the tomfoolery. tomfoolery?

Pointless riddles

So - we made up a random riddle and sent it to one of the server mods, Hunter_cookie21, in morse. The details of this riddle are completely irrelevant on many levels. The details that are interesting enough to share are:

The strings of morse seemingly prompted the staff to seemingly launch an investigation into the account (whoops) and a couple days after, they traced the account back to my main, Candycup. Oops.

My friends over at the staff team DM’d me, asking if I knew something about ‘StalkingOthers’. I wanted to keep the bit going - so I effectively denied association. LSN’s staffing tools are what they are, so it looked like they bought the story. Story in question being ‘yeah Idk man not me’

We did eventually come clean to that specific staff member. We were friends, it’s all good.

Reaction image 1

Reaction image 2

Managing our drone

To manage the drone system, we hooked a discord bot up to our own API - that we could use to track incoming messages. Feeling bad about that whole Riddle-incident, we felt obligated to at least start tracking personal messages to the bot.

Here’s some funny encounters:

A discord message showing the bot DM

A discord message showing the bot DM

Guy that is catching onto us

Youtube?????

Wanne team?

So many graphs

Okay - if you’re reading to this point, you probably want to see some of the actual data. Well - here it comes. Just after this brief mini-chapter about how we actually graph our data.

Our old graphic system was scuffed. Like scuffed scuffed. Let’s talk about how that actually worked.

We had a Python script that fetched all our database records, and then created a JavaScript file, and then dumped all the data inside a javascript variable, and then dumped the file onto disk, and then opened up the associated HTML file. Here’s what it about looked like:

var data = [{data formatted by python}]

Anyways, down with the old, in with the new. I settled for Grafana.

Grafana is a popular monitoring tool that supports various data sources, including Influx, which we were using for data at the time.

We set up a self-hosted instance, sh ftw, and configured a bunch of panels to use our data. This was extremely satisfying to get working.

Combined power of the server’s teams:

Teamtop leaderboard

Here’s what said dashboard looks like today (video may take a sec to load):

After spending a bit too many man-hours on this project, we had to do a little bragging. We dm’d the video of the dashboard to the server’s head developer, who was impressed, but also hinted at the fact that I should probably get a job. In a good way.

[reaction]

We also DM’d the server owners with the data we had.

Sanitize your data, people!

In case you find yourself creating a tracking system to track a server’s economy, consider adding some limits on what kind of data you’ll accept in your system.

In mid-october, one of the server owners, Leowook, decided to give his alt a good 1 trillion coins and then bid on an auction. This created some nightmare graphs that I had to manually sort out.

Bugged graph peaking at 1 trillion

Graphs for everyone

We decided to start going public about all the data we had. We decided on doing weekly economic recaps for the community, where we decided to do weekly recaps with our data. Who was the richest, what teams were rising and falling in power, other developments in the economy, etc etc. The community didn’t really have this data before - we were creating and filling our own market.

Picture of a discord message showing the rough summary of the economy in the period of oct 13th -> oct 19th

Partial screenshot of a recap

That time we got Nuked

A screenshot showing a detailed crash log with the error message 'Your account has been suspended, Please contact customer service'

On the 17th of November 2024, StalkingOthers was suspended by Microsoft and lost its access to Minecraft’s authentication servers. The system was off the air for over 24 hours - after which we did actually get the account back after politely appealing.

PART IV: In Conclusion

Today, tracking is ingoing. Microsoft took us down once, they might do it again. We’re not having this battle forever. We’re thinking of expanding our data to make a public stats tracking website for the server, and already have basic designs for it. May or may not happen.

Today, both server owners are in the LEF’s public server and one is tuned into the weekly recaps. I’d like to think that we’ve provided a service for the server, and the occasional data inqueries we get from the management help prove this point.

In conclusion - this has been my tutorial on

I will include a QnA section in this blog post. Please DM me on discord @candycup to ask questions - and i’ll add them here anonymously.

SPECIAL THANKS:

Additional notes

Some aspects of this blog are out of chronological order for the sake of creating a somewhat clear story. Here are some facts that weren’t otherwise mentioned:

We ran internal tests with the bot account before we got authorized to actually operate a bot account. This was when we sent riddles to the staff. No ban plz.

In some Grafana screenshots in this blog, we use a custom font called ‘Lexend’ - same one this blog is using as the primary font. I did not immediately stumble on to an option to change the font used by Grafana so we made a custom browser rule using Stylus to just use css to push our way through.