Announcing the AI Alignment Prize

Stronger than human artificial intelligence would be dangerous to humanity. It is vital any such intelligence’s goals are aligned with humanity's goals. Maximizing the chance that this happens is a difficult, important and under-studied problem.

To encourage more and better work on this important problem, we (Zvi Mowshowitz and Vladimir Slepnev) are announcing a $5000 prize for publicly posted work advancing understanding of AI alignment, funded by Paul Christiano.

This prize will be awarded based on entries gathered over the next two months. If the prize is successful, we will award further prizes in the future.

The prize is not backed by or affiliated with any organization.


Your entry must be published online for the first time between November 3 and December 31, 2017, and contain novel ideas about AI alignment. Entries have no minimum or maximum size. Important ideas can be short!

Your entry must be written by you, and submitted before 9pm Pacific Time on December 31, 2017. Submit your entries either as links in the comments to this post, or by email to [email protected] We may provide feedback on early entries to allow improvement.

We will award $5000 to between one and five winners. The first place winner will get at least $2500. The second place winner will get at least $1000. Other winners will get at least $500.

Entries will be judged subjectively. Final judgment will be by Paul Christiano. Prizes will be awarded on or before January 15, 2018.

What kind of work are we looking for?

AI Alignment focuses on ways to ensure that future smarter than human intelligence will have goals aligned with the goals of humanity. Many approaches to AI Alignment deserve attention. This includes technical and philosophical topics, as well as strategic research about related social, economic or political issues. A non-exhaustive list of technical and other topics can be found here.

We are not interested in research dealing with the dangers of existing machine learning systems commonly called AI that do not have smarter than human intelligence. These concerns are also understudied, but are not the subject of this prize except in the context of future smarter than human intelligence. We are also not interested in general AI research. We care about AI alignment, which may or may not also advance the cause of general AI research.


sorted by
magical algorithm
Highlighting new comments since Today at 10:40 PM
Select new highlight date
All comments loaded

What should we be doing to help get more people to enter, whether by spreading the word or another way? We want this to work and result in good things, and it's iteration one so doubtless a lot we're not doing right.

Yeah, I had an initial gut sense of "oh man this seems important and but I'm worried it'd quietly fade out of consciousness by default." Much of my advice would be whpearson's. Some additional thoughts (I think mostly fleshing out why I think whpearson's suggestions are important)

i. Big Activation Costs

You are asking people to do a hard thing. You'd providing money to incentivize them, but people are lazy - they will forget, or start doing it but not get around to finish or not get around to finishing until too late.

Anything to reduce the activation cost is good.

1) Maybe have the first thing you ask is for people to apply if they might be interested, with as low a cost to doing so as possible (while gaining at least some information about people and weeding out dead-wood).

This gets people slightly committed, and gives you the opportunity to spam a much narrower subset of people to remind them. (see spam section)

2) It's ambiguous to me what kind of writing you're looking for, which in turn makes me unsure if it's be a good use of my time to work on this, which makes me hesitate. (I'm currently assuming that this is not the right use of my talents both for altruistic and selfish reasons, but I can imagine a slightly different version of me for whom it'd be ambiguous)

Whpearson's "list good existing articles, as diverse as possible" helps counteract part of this, but still doesn't answer questions like "should I be doing this if this is currently my day job? Presumably the point is to get more people workin on this." (and the correlary: if professional AI safety workers are submitting, what chance do I have of contributing something useful?)

(Relatedly - I'd originally thought you should spell out what sort of questions you were looking to resolve, then saw you had linked to Paul Christiano's doc. I think attempting to summarize the doc might accidentally focus on too narrow a domain, but the current linking of the doc is so small I missed it the first time)

ii. Spam vs Valuable-Self-Promoting-Machinery

By default, you need to spam things a lot. One way to get the word out is to post on all the relevant FB groups, discords, etc - multiple times, so that when they forget and fade to the backburner it doesn't disappear forever.

Being forced to spam everyone once a week is a bad equilibrium. If you can figure out how to spam exactly the people who matter (see i.1) that's also better.

If you can spam in a way that's providing value rather than sucking up attention, that's better. If you can make the thing spam itself in a way that provides value, better still.

One way of spamming-that-provides value might be having a couple followup posts that do things like "provide suggestions and reading lists for people who are considering working on this but don't quite know how to approach the problem." (targeting the sort of person who you think almost has the skills the contribute, and is just missing a few key elements that are easy to teach)

Another might be encouraging to post their drafts publicly to attract additional attention and comments that keep the thing in public consciousness. (This may work against the contest model though)

Some random initial thoughts.

Post on the SSC open thread ? Or the EA forum open thread (maybe the EA subreddit too). I've seen it posted to the control problem reddit.

I'll post it on the ai danmark safety facebook page, although I've never managed to go to one of their reading groups (it is now pending).

Ask nicely the people running lesserwrong to see if you can see the referrer for where traffic comes in to this thread, this will give you an idea where most of the traffic comes from.

To get more people to enter, imagine you were running the competition previously, pick N articles out there on the internet and link them as things that would be short listed. This would give people an idea of what you are looking for. Try and pick a diverse range else you might get articles in a cluster.

Perhaps think about trying to get some publicity to sweeten the deal, e.g. the winner also gets featured in X prestigious place (if the submitter wants it to be). Although maybe only after the quality has been shown to be high enough, after the first couple of iterations.

Happy to give you any analytics data for the page.

Can you send them to Zvi and me? (vladimir.slepnev at gmail, thezvi at gmail)

I think this is the most important thing, and I would be happy to help with that.

That would be great! Do you need any input from us to do it?

Should I submit? Working on this is my job, so it's maybe better to encourage others to come on board?

We certainly don't want to exclude experts! Please do submit.

I saw a talk earlier this year that mentioned this 2015 Corrigibility paper as a good starting point for someone new to alignment research. If that's still true, I started writing up some thoughts on a possible generalization of the method in that paper.

Anyway, submitting this draft early to hopefully get some feedback whether I'm on the right track:

GeneralizedUtilityIndifference_Draft_Latest.pdf (edited)

The new version does better on sub-agent shutdown and eliminates the "managing the news" problem.

(Let me know if someone already thought of this approach!)

EDIT 2017-11-09: filled in the section on the -action model.

Zvi/Vladimir, what's your role in this - are you the judges?

Organizing and judging. Might bring in other judges if things go well.

Are you looking for entries with actionable information, or would you be interested in a paper showing, for example, that AI alignment might not be as big a problem as we thought but not for a reason that will help us solve the AI alignment problem?

OK, I went on a rant and revived my blog after 4 years of inactivity because entries aren't supposed to be entered as comments but are supposed to be linked to instead.

Thanks! Can you give your email address so we can send feedback?

Just comment on the blog or here or both, if you want to send private feedback try JoeShipman a-with-a-circle-around-it aol end-of-sentence-punctuation com

Are there any limitations on number of submissions per person (where each submission is a distinct idea)? On number of wins per person?

One win per person, and it's okay to have many ideas. Might be more convenient if you submit them as one package though.

Should've saved my decsion alignment loop post a few days. Maybe an expansion of it? Hmm.

Yes, an expansion of that post would qualify.

How much should I try to make it self-contained?

I'd prefer a self-contained thing. In the extreme case (which might not apply to you), an entry with many links to the author's previous writings might be hard to judge unless these writings are already well known.

You should think about the incentives of posting early in the 2 month window rather than late. Later entries will be influenced by earlier entries so you have a misalignment between wanting to win the prize and wanting to advance the conversation sooner. Christiano ought to announce that if one entry builds in a valuable way on an earlier entry by someone else, the earlier submitter will also gain subjective judgy-points in a way that he, Paul, affirms is calibrated neither to penalize early entry nor to discourage work that builds on earlier entries.

I have unpublished text on the topic and will put a draft online in the next couple of weeks, and will apply it to the competition. I will add URL here when it will be ready.

Good. If you submit it early we'll give feedback so you can improve it before the deadline if needed.

Is it possible to enter the contest as a group? Meaning, can the article written for the contest have several coauthors?

my idea: