Will AI Empower or Disempower Humanity?

Technology is traditionally thought of as a tool that empowers its users. But AI is increasingly capable of acting autonomously, and superintelligent AI agents could arrive any year now. Many experts are concerned such agents might "go rogue", disempowering -- and likely destroying -- humanity in pursuit of misaligned goals. But even if AI never goes rogue, it might still cause human disempowerment and extinction. Why? Competitive pressures will increasingly push people and institutions to hand power over to AI systems. And even if AI alignment is "solved", we may end up with AIs optimizing for objectives like national security or quarterly profits, even if these objectives become increasingly disentangled from human interests. How can we avoid disempowering ourselves? The future is not written, and I will also discuss arguments against disempowerment as the default outcome, and ways of addressing the risk. But there is no plan. And ultimately, given the rate of progress in AI, the safest course of action may an international agreement to halt further progress towards superintelligence.