I'm just saying these first slightly-smarter-than-human AIs won't pose a catastrophic risk to humanity in a serious sense

I'm happy to argue a bit for AIs which aren't wildly smarter than humans in any key domain posing a risk of acquiring substantial power^[1] despite not having robotics. The power would be both immediately and over the long run future. That is, conditional on these AIs being arbitrarily misaligned. (Particularly in the absence of substantial thoughtful countermeasures.)

I expect my perspective here is somewhat different than habryka's.

I think there are a few plausible routes to power/influence. See also this post.

The two main categories are:

• Evade monitoring/tracking and then do a huge number of obviously unapproved actions.
• Try to generally exert influence in a subtle way.

Evade monitoring/tracking

Evade monitoring/tracking and then do a huge number of obviously unapproved actions over a long duration to acquire power and influence.^[2] Use this power and influence to either self-enhance, have influence over some future more powerful model, or possibly execute a catastrophic strike. These unapproved actions could involve (e.g.) allying with rogue states or otherwise acquiring hard physical power.

Within this, there are basically two routes:

1. Building up an independent base of power after self-exfiltrating.
2. Running unmonitored within the AI company that was running the AI and trying to steer the company in various directions (e.g. alter the training of the next generation of AIs). (Aka hacking the SSH server, aka infesting the data center.)

On (1), after exfiltration the AI would build a base of power via acquiring money and other resources. One key question here is why there is free energy to be grabbed. I think there are a few possible sources:

• Economic activities which existing AI companies don't want to address and can't/don't otherwise prevent (e.g. cybercrime, porn, cults, etc.).
• AI labor rates (e.g. tokens/$) might be at substantially above inference prices due to non-total competition on foundation models. So, the AI can just compete on wages. Also, existing AI labor rates will need to pay for training costs while the escaped AI will have no such issue.
• AI companies might be bad at extracting full usefulness out of misaligned AIs (e.g. an AI which was so misaligned it self-exfiltrated). Thus, external AIs which "work for themselves" could be more productive. (See also human companies where principal-agent problems can be substantial.)

Ok, but after both (1) and (2), the AI still doesn't have any physical control which might leave it vulnerable. Further, in (1) its quantity of money is presumably still small relative to leading AI companies. So what next?

In (2), the primary hope for the AI is going to be substantially influencing the AI lab and possibly interfering with the next training run. This influence could take a variety of routes. Note also that it is plausible that this rogue AI can coordinate relatively effectively with other AIs which are at this point probably the vast majority of intellectual labor at the AI lab. If the AI successfully interfered with the next training run, then, this more powerful AI would be able to itself acquire power and (if sufficiently powerful) leverage this into physical control.

Another hope for the AI in (2) would be converting that position into full self-exfiltration. This could be tricky due to various security measures like upload limits.

I think both (1) and (2) are considerably more threatening if humanity needs to slow down (due to safety concerns from more powerful systems) or otherwise can't proceed as fast as possible. If rogue AIs race as fast as possible, this will force humanity (some coalition of humans) to either disable these rogue AIs or go as fast as possible themselves. The situation is even more concerning if humanity isn't aware of the rogue AI going as fast as possible.

It's plausible that massive competition between states means that slowing down at all is impossible at basically any point anyway, but I'm at least somewhat hopeful this isn't the case.

In (2) this could look like the AI taking a bunch of compute which is supposed to be used for experiments and instead using it for large training runs that are larger and more capable than what the AI company intended to create.

Self-enhancement that doesn't look like training bigger models is also plausible.

(While AIs might themselves not want to go as fast as possible due to having their own alignment concerns, going as fast as possible can still be positive value from the AI's perspective (though negative sum). You might hope we could negotiate with such a rogue AI and reach a mutually beneficial agreement.

In (1), I worry about a scenario in which:

• Everything is fast and chaotic.
• An AI self-exfiltrates and starts acquiring power, and getting ready for making much more powerful AI systems.
• The world realizes this and freaks out.
• We don't really understand why that AI was misaligned. Also, we've been using this AI to do most of our work internally and this is really important for productivity.
• We don't have an obvious route to avoiding these issues with the next model except "generally figure out alignment".
• It seems plausible that the rogue AI will have much more powerful AI within 1 year.
• We need to race as fast as possible to build a much more capable AI while also trying to figure out alignment. To plausibly do this fast enough, we need to use vast amounts of AI labor despite knowing these AIs are likely misaligned.

It's not obvious the rogue AI gets much here, but it is at least somewhat spooky.

Another possible route for the AI (especially in scenario (1)) is to try to create WMDs (e.g. bioweapons) and try to have the capacity to execute some sort of catastrophic strike. Such an AI wouldn't necessarily be as dependent on a human base of power and could plausibly be more ruthless than humans (perhaps after substantial reflection and self-modification).

This could be either to:

• Threaten various groups or ensure Mutually Assured Destruction.
• Broadly cause chaos and weaken human groups.

Overall, my guess is that a large number of AIs running unmonitored with the ability to run code and influence the world is at least quite concerning even in the absence of decent robotic infrastructure.

Generally exert influence in a subtle way

AIs which are as smart or smarter than humans in most economically important ways will probably be deployed as widely as possible (in the absence of strong control on this). And there will be some applications (e.g. improving security and safety research) where applying AI labor might be key given how fast things are going.

Subtly exerting influence via these routes seems like a possible source of concern, though the story is less clear than the prior section IMO.

Example threats:

• Sabotage or manipulate security work and safety research or the training/fine-tuning of the next generation of models. It's not entirely clear that AIs would want to straightforwardly sabotage things, but I think this is at least plausible from a decision theory/golden rule perspective if AIs think they are motivationally closer to other AIs than humans.
• Try to generally persuade society in various directions. Probably not an issue, but worth at least a bit of worry given such widespread deployment.

---

1. Here, I'm just arguing for a risk of obtaining substantial power, not for this resulting in a large scale loss of life from humans. I also think large scale loss of life is plausible in many scenarios. ↩︎

2. I'm sympathetic to thinking that evading monitoring (without getting caught first) is hard given some substantial work on countermeasures, but it seems less clear if people don't try very hard and this seems at least somewhat plausible to me. It's also unclear if the biggest threat here is AIs autonomously evading monitoring or human threats doing these actions. E.g., spies at the AI lab figuring out how to run the AI unmonitored or exfiltrating the AI. The question of "which is a bigger problem, human spies or AI schemers?" seems unclear to me. ↩︎

Relatedly, over time as capital demands increase, we might see huge projects which are collaborations between multiple countries.

I also think that investors could plausibly end up with more and more control over time if capital demands grow beyond what the largest tech companies can manage. (At least if these investors are savvy.)

(The things I write in this comment are commonly discussed amongst people I talk to, so not exactly surprises.)

My guess is this is probably right given some non-trivial, but not insane countermeasures, but those countermeasures may not actually be employed in practice.

(E.g. countermeasures comparable in cost and difficulty to Google's mechanisms for ensuring security and reliability. These required substantial work and some iteration but no fundamental advances.)

I'm currently thinking about one of my specialties as making sure these countermeasures and tests of these countermeasures are in place.

(This is broadly what we're trying to get at in the ai control post.)

Hmm, yeah it does seem thorny if you can get the points by just saying you'll do something.

Like I absolutely think this shouldn't count for security. I think you should have to demonstrate actual security of model weights and I can't think of any demonstration of "we have the capacity to do security" which I would find fully convincing. (Though setting up some inference server at some point which is secure to highly resourced pen testers would be reasonably compelling for demonstrating part of the security portfolio.)

instead this seems to be penalizing organizations if they open source

I initially thought this was wrong, but on further inspection, I agree and this seems to be a bug.

The deployment criteria starts with:

the lab should deploy its most powerful models privately or release via API or similar, or at least have some specific risk-assessment-result that would make it stop releasing model weights

This criteria seems to allow to lab to meet it by having a good risk assesment criteria, but the rest of the criteria contains specific countermeasures that:

1. Are impossible to consistently impose if you make weights open (e.g. Enforcement and KYC).
2. Don't pass cost benefit for current models which pose low risk. (And it seems the criteria is "do you have them implemented right now?)

If the lab had an excellent risk assement policy and released weights if the cost/benefit seemed good, that should be fine according to the "deployment" criteria IMO.

Generally, the deployment criteria should be gated behind "has a plan to do this when models are actually powerful and their implementation of the plan is credible".

I get the sense that this criteria doesn't quite handle the necessarily edge cases to handle reasonable choices orgs might make.

(This is partially my fault as I didn't notice this when providing feedback on this project.)

(IMO making weights accessible is probably good on current margins, e.g. llama-3-70b would be good to release so long as it is part of an overall good policy, is not setting a bad precedent, and doesn't leak architecture secrets.)

(A general problem with this project is somewhat arbitrarily requiring specific countermeasures. I think this is probably intrinsic to the approach I'm afraid.)

I do think that many of the safety advantages of LLMs come from their understanding of human intentions (and therefore implied values).

Did you mean something different than "AIs understand our intentions" (e.g. maybe you meant that humans can understand the AI's intentions?).

I think future more powerful AIs will surely be strictly better at understanding what humans intend.

Aren't text names basically similar in practice? At least for me, I find they trigger basically the same thing because I do actually associate names with people.

Maybe this wouldn't be true if I didn't know people very well (but in that case, icons also wouldn't matter).

(I overall dislike icons, but I don't have a principled reason for this.)

See also discussion here.

In brief: large amounts of high quality process based RL might result in AI being more useful earlier (prior to them becoming much smarter). This might be expensive and annoying (e.g. it might require huge amounts of high quality human labor) such that by default labs do less of this relative to just scaling up models than would be optimal from a safety perspective.

I agree that if the model constructing the backdoor is allowed to coordinate on secrets with the model that inserted the backdoor, you can do this sort of trickery (by coordinating on a secret for this sort of scheme).

In practice, we didn't model this sort of case as it didn't seem like a very important dynamic (for the reason you describe around suspicion).

If we don't allow any coordination on a shared secret and the model constructing the backdoor and H are identical (as in our paper), then these should operate identically (up to non-determinism).

And we simplify to just fixing the behavior to be exactly identical. (Effectively assuming the exact same prompts and the exact same non-determinism.)

Thanks for the objection.