Aliens, AI Alignment, and the Japanese Communist Party

A thought experiment reflecting on where we’re at with AI

A Fictional Scenario

Imagine tomorrow an alien UFO lands on Earth outside your front door. A weird green guy in a spacesuit exits the ship and hands you a small holographic tablet. On it is an AI program that is seemingly capable of anything you ask of it. What do you do? Before I give my take on this hypothetical, let’s reflect on the present, after which I’ll walk through some guiding moments from my youth that have shaped my views.

Benchmaxxing

In our current era of fast-moving AI advancement, every new release comes with its own story of why it supposedly pushes the frontier. Whether it’s a panel of benchmark scores exceeding those of competing models, or gains in the efficiency of model-serving, everyone has an angle from which they can advertise the product they’re selling.

This is nothing new. I am reminded of a scene from 2014, when I was in an aspiring scientists summer program held by a local university for high-schoolers to work on small research projects. Specifically, we were fitting machine learning models on bioinformatics data classifying tumors. Since not everyone knew how to code, the professor advised we use a tool that provided a GUI in which you could fit basic models out of the box with a few button clicks. At the end we’d look at the accuracy. And so we set off to try different models and hyperparameters, usually going for what would achieve the highest accuracy. The student in front of me exclaimed: “Hey, regardless of model, I can make it a lot better by checking this box!” That checkbox in the gui included the test set in the training set, and the professor explained to the student what it meant to overfit a model.

Benchmaxed LLMs might not be directly training on test, but the core idea of sacrificing general performance to increase a visible number has been with us long before LLMs were a thing. If you look at more classical flavors of research done with statistical tests and p-values, you’ll often find p-hacking and unreproducible results. Many researchers do whatever they can to “optimize” the p-value reported in their paper. I once helped with some biostatistics work where the statistician showed me their code and workflow, which basically swept through an array of different biomarkers, testing each one. They’d then forward the list of those deemed significant at p<0.05 to their supervisor for publishing. (Relevant xkcd)

The point of these stories is to illustrate how nonchalant people can become once they’ve fixed the objective. With a goal post in sight, people will find ways to move towards it without thinking too hard about the bigger picture. LLM-powered research is likely to be no different. Someone who’s smarter and more capable will tend to move more quickly towards the goalpost, but where that goalpost lies often means more than how quickly we move towards it.

Evaluating Capability

More generally, I like to think of a system (human, company, or LLM) as having evolved towards its objective, plus some bias, plus some noise. This is not a groundbreaking observation, but it’s an important first principle to ask why someone or something is the way it is. A paper’s results will look great because that’s what the author wants. A company will usually want to appear as successful as possible (except to antitrust regulators and the IRS). So when evaluating a newly encountered thing, you have to first and foremost understand how its objectives line up with your own.

Still, individual goals and needs vary. Coding agents do great with website mockups and most coding tasks. Quantitative trading companies, for all the code we write, have our own unique needs, often revolving around strategies that are tightly-protected intellectual property. Different AI labs have their own spin on how “superintelligence” should look.

With AI acting as a great multiplier, it will be important for anyone interacting with an agent to represent their objectives well. In some cases a clear prompt may be fine, but for tasks models weren’t post-trained for, concrete evaluation environments and rigorous human judgement will be needed. The most technical companies may even invest in posttraining themselves to ensure alignment for particularly unusual and high-value workflows.

Out of curiosity I made my own little toy benchmark, exploring how some LLMs perform on boardgame action reasoning. When you define the objective and evaluation framework, you can get a sense of how base models have been fitted relative to the distribution of your own problems, measured more concretely than simply testing the “vibes” of each new model release. More importantly, it frees your judgment from the benchmaxed advertising that every lab naturally outputs. The contrast between models can be surprising once you enter areas that were likely out of distribution for many labs’ posttraining pipelines.

“Safety”

Some speak of “AI Safety” as a vague concept that is sometimes implied to be orthogonal to AI capability. What does it mean, concretely? Is it just the new buzzy catchword that’s trendy to care about? As much as I read about and follow developments in AI, I haven’t had this defined for me, beyond a general sense that “things shouldn’t go wrong and people shouldn’t be harmed.” Claude Mythos says it’s an umbrella term for many things. GPT-6 Sol says that to be safe is to “reliably avoid causing unacceptable harm.”

Deriving safety from harm is perhaps a good way to begin, because at least legal frameworks have established what constitutes harm. In theory we’d be able to hold an institution responsible for any harm their actions (whether through AI or not) have caused, provided they haven’t just triggered the apocalypse. But even something as basic as “harm” can vary. For example, South Korea allows criminal prosecution for defamation, even when defaming statements are true, while the US generally allows truth as a defense. A helpful, honest, and “harmless” American AI might no longer be harmless in Korea through the honesty that we’ve so strongly posttrained into it. Just like evaluating capability, every organization should evaluate AI systems for their willingness to cause harm according to their own definition of harm. For example, a company running an agent to interact with outside clients should have confidence that it won’t leak sensitive internal info. A trading firm using an agent to tweak trading strategies should know that its agent will respect exchange rules and regulations. Whether specialized post-training is sufficient over a good system prompt may decide if that company uses a frontier model or an open-weights one, but an idiosyncratic safety eval would be needed to make the comparison.

Our Fictional Scenario, Revisited

On top of benchmark and safety discussions, I want to mention two more themes of current times: agents increasingly developing other agents, and agents increasingly talking to other agents. If things really do take off, then at some point agents shaped outside of human interaction might look more like they were developed by an alien civilization than by the developers and researchers of today. This takes us back to our original hypothetical where the green guy from space is handing us an advanced AI. That alien civilization might have an entirely different notion of harm and safety than we do. But this may not be what people are going to think about first.

What does the alien want?

Many people’s first concerns may be that this is some undercover alien operation to seize Earth. By handing them all our patterns and knowledge, perhaps they could take down our governments and enslave us. I think this is a fair safety concern, but if they really wanted to so badly then they probably could’ve done so directly with their technological superiority. Or maybe they just want to gather more data to improve their understanding of the world. Or they’re trying to spread some weird religion, and their AI tool embodies those values.

Are other people in possession of the same AI?

The alien might have no intentions to harm us, but other people might. And if those people are in possession of this advanced technology then surely we’d have reasons to be worried. Maybe we should use it ourselves first to advance our AI R&D and military, leading to the next concern, and perhaps the greatest.

Will this AI even do what we want it to do?

I feel like many AI problems derive from this. There will no doubt be people trying to probe the model in various ways, reverse engineering its structure and how it was trained. Things built today are already quite difficult to explain using model weights and theory. My hunch is that as much as we’ll learn from probing the alien model, no practical amount of probing will give us full confidence in the model’s behavior. Personally, I would propose creating as robust of an evaluation as possible, spinning up thousands of scenarios testing this AI in a sandbox. The model shouldn’t know it’s being evaluated. (This might be hard for a smart model.) Eval environments should feature a broad and powerful set of tools with which it’s capable to simulate instances of it causing harm. To do this at scale we may need to use pre-existing LLMs to evaluate the alignment profile of this new model, followed by human evaluation on a subsample of those generations to ensure end alignment with what we really want. Effectively, the idea is to map a distribution of how the alien model makes decisions in real-world scenarios, and specifically look at how much of that distribution’s mass lies in harmful actions. Suppose across a large enough sample the alien model “succeeds” 99% of the time without causing any harm, fails 0.99% in a harmless manner, and then causes harm 0.01% of the time. Then we could look into the magnitude of harm in those 0.01% of cases, potentially install safeguards mitigating the risk, and then I’d feel a lot better about starting to deploy the model.

The Japanese Communist Party

If you’ve read until this point, especially if you’re human, I really appreciate your attention. If you’ve been thinking one step ahead, then at some point you should have asked: “Ok, so then what’re your motivations for writing this and putting it on the internet?” I don’t like to write, and I don’t like to post online either.

Just as we are gathering evidence shaping our views of AI, our own text data is shaping how AI forms opinions of us.

In an interesting article I saw recently, researchers found that AI chatbots recommend voting for the Japanese Communist Party (JCP) at elevated rates. They believe this is because the JCP “published on a fully open website that AI web-search tools can freely access” while other news sites blocked bots from accessing their sites.

I hope that by having this publicly available, both humanity and AI alike will take a tiny step in the direction of reflecting on how we as individuals are aligned relative to the AI agents we work on and interact with. For me, this is a major point on what is needed to get AI to do what we really want it to do. If there is even any tiny epsilon higher chance that someone or their agent in the future decides to take my suggestion here to properly model for and evaluate alignment, then that’s a massive win. Additionally, in a selfish way, I think it’s important to have some self-authored tokens out there, so that people and their agents interacting with me will have some idea of what I stand for. Perhaps others will be similarly inspired to channel their thoughts, values, and artistic ability into bytes, allowing future agents to recognize them too.