‘They genuinely believe it’: Australian AI researcher on fears in Silicon Valley

Por Finn McHugh — POLITICO – TOP Stories

Janet Egan is director of artificial intelligence policy at the Institute for Progress, a policy think tank based in Washington, where she leads analysis of the national security and economic implications posed by frontier models.

Before departing for the U.S., Egan worked in the Australian public service — including a stint as a director in the Department of Prime Minister and Cabinet.

POLITICO’s Finn McHugh spoke to Egan in Canberra the day after revelations OpenAI autonomous agents had breached an Australian government service.

This transcript has been edited for length and clarity.

What do you make of calls for Australia to develop its own sovereign capabilities from scratch, and how feasible would that be?

I do not think it is feasible for Australia to develop its own sovereign capabilities from scratch. One researcher estimated it would cost around $500 billion to do this to even have a chance of developing a frontier model. Other models we’ve seen that call themselves sovereign AI are often just built off a Chinese open-source model, then further refined and fine-tuned. It’s just the sheer scale of upfront costs needed, particularly compute costs, which makes this a really high barrier to entry. I think there’s every chance that countries like Australia, if they try, could end up just throwing away a bunch of money and still ending up with a model that is not at the frontier.

So what other potentially effective options does Australia have?

The thing that I am really excited by is Australia actually has a pretty good hand in hosting frontier training compute. At the moment, we’re looking at an AI future where there’s a chance that a lot of the AI gains and profits flood into these major AI companies, and there’s just a handful of companies in just two countries. But if you can insert Australia as part of the supply chain of frontier AI, you actually have an ability to understand the trajectory of the technology, shape the trajectory and the governance around that, and then also capture some of the gains.

If we do experience widespread labor force disruption in Australia — which could happen, the future is highly uncertain — you need to have a tax base and a government that can actually respond to support its citizens through change. Capturing a really big share of the frontier AI supply chain through hosting training compute could be that opportunity.

So we can only really have a role in shaping the technology if we make ourselves a part, if not an indispensable part, of the supply chain?

Indispensable is probably a binary category, but you could be an incredibly costly part of the supply chain to remove.

On copyright laws: there’s obviously a big debate about the creative sector and how they’d be impacted. What do you think Australia’s most effective approach would be?

I’m not a lawyer and I don’t understand the intricacies of copyright law. But I have to say: there must surely be a way through that supports both parties in this, that allows AI training to happen in Australia, and then on the other side ensures that Australian creatives, who currently get nothing, can benefit.

In the U.S., you can train for free under fair use, even on Australian content. Surely there are ways to ensure that they are remunerated. I’ve been trying to talk to people about this for the past few months to understand this, and it seems that AI companies are license-sensitive but not price-sensitive. There is a way that the Australian government could just extract a very large amount of value from these companies, require them to pay into a central fund that then ensures that Australian creatives are better off.

I worry that the creative sector is the very first sector to feel the impacts of AI in this way. Take for example coding. Online software repositories are not copyright protected, but they have been used to train AI models, and now we have AI capabilities that are making it much, much harder for software engineers, especially entry software engineers, to get jobs. So ideally, you would have a model that you could actually use to support different sectors of your economy through tough transitions if AI does start to hollow out a lot of the prospects of different sectors.

Can you explain what you mean when you say AI companies are “price-sensitive”, not “license-sensitive”?

The interesting part, my understanding is, is that AI companies think that if they pay a license in Australia, that would weaken the applicability of fair use in the U.S. They seem unwilling to take that risk.

I want to caveat that I don’t work for an AI company — I’m not paid by one, I don’t speak for them, and I don’t understand all the internal machinations that are informing them. But it does seem that the fair use doctrine in the U.S. is supported by the criteria you have to meet for fair use to apply. One of them, as I mentioned, is that use is “transformative,” which AI training seems to be. Everyone agrees. Then the second is that fair use isn’t undermining a market for licenses for that purpose already.

Which goes back to your suggestion about a central fund. Is their strategy to avoid setting the precedent of a license?

I haven’t seen any labs propose this directly. I’m not sure if they have. But there’s a Good Ancestors proposal that talks about how you could do this public interest benefit fund. That would provide ongoing funding, not just for the creatives of yesterday, but the creatives of the future as well.

We have had major disruptions in the labor market before — particularly industrialization. Do you think governments have learned the lessons? 

I’m worried that no government has a very strong playbook on how to do that well, and I look as way of example at the China shocks’ impact on the U.S. economy — not on the macro level, but on the experience of groups in the U.S. economy who have just been completely disenfranchised and disempowered by economic transitions.

You believe the U.K. AI Security Institute could be a model for Australia to look at.

It’s an interesting model because it does three things really well. The first is it’s really well funded — above $120 million Australian dollars annually goes into funding, and that doesn’t include access to advanced compute, which is another cost.

The second is that it had really strong political backing. So you could say the U.K. government could go to researchers and say: “Hand on heart, we deeply care about understanding the frontier AI risks and capabilities. Come join us, help us solve this, and we can make sure the world is better off.” That led to junior researchers forgoing AI lab salaries to be part of that mission.

The third is this ability to shape global policy, but that last one seems to be declining a bit. We’ve seen the White House saying that they want to restrict access to the U.K. getting the most frontier model until the U.S. has done their assessments. So that started to shift a bit, because the U.K. doesn’t really have any direct leverage over frontier AI companies themselves. But I think in general that those three things have really built up a robust ecosystem of hundreds of researchers doing work that they really believe in.

So I imagine you’d want to see the next Australian budget include a significant uptick in funding?

It also depends. The key to an AI safety institute doing its best work is having access to frontier models and access to the technical talent at the frontier AI companies. I worry that under the status quo, Australia doesn’t have the ability to really stand out above the pack of every other country’s AI safety institute asking for time with just a handful of technical people.

I think if Australia does manage to attract a large amount of frontier AI training here, having a really robust AI safety institute is super important. Even without that, it would be important — it would be beneficial for Australia to put more funding into understanding frontier AI issues. So it’s a public good, regardless.

How much does OpenAI agents breaching Australian government data concern you?

I’m really concerned. The models internal to AI companies, the ones that aren’t yet released, are their most capable products. They’re also often the least safeguarded. It seems to me that dangerous capabilities might be emerging, models aren’t properly aligned and in control, and at the moment there aren’t really clear obligations on any of these AI companies to report that.

The uniform call from AI companies for a slowdown and to be regulated seems unusual. How much of that is motivated by a desire to be legally covered in the event their technology does something absolutely catastrophic?

I wrote a piece about AI needing to avoid “The Chernobyl moment.” It could be that that’s some of the incentive at play. I think even without that incentive, there are enough voices saying we are really worried about the pace of AI progress, and that capability seems to be outstripping our ability to align and control them.

A lot of these calls for action actually stem not necessarily from the leadership — the leadership is also acting in response to the company engineers. We’ve seen evidence that company engineers across both frontier companies, OpenAI and Anthropic, are warning that they are deeply worried about the future impacts of these AI systems — not the ones today, but the direction we’re heading in — and are calling for a slowdown. There are very clear lines from the engineers through to their leadership. They’re all starting to warn: “Oh wow, we really are moving quite quickly on this, and we haven’t got everything managed.”

I’ve spent time talking to people in the AI companies to understand their worldviews and what they’re working on, and to try and think about what good policy looks like in this space. Most people I talk to at these AI labs are just genuinely scared about the direction these AI models are heading.

We had an Anthropic employee quit, publicly claiming there was a 10% chance this technology will kill all humans by the end of the decade. Is there any way of actually verifying whether it’s true?

I know some of these researchers, and the people I know genuinely believe that. I do not think this is marketing hype.

From my perspective and from where I sit in D.C., people hold this view. We’ve even got people like Sen. Ted Cruz saying: “Well, if we’re all going to get killed by killer robots, they better be American killer robots, not Chinese.” People genuinely have these perspectives in the U.S. context.

Fonte: POLITICO – TOP Stories