Thank you for this intelligent informed post. In reply....
I would ask that we think bigger than just accepting the old paradigm of "we will race ahead and try to fix any problems that arise". That's outdated thinking, in my view. That era ended in 1945 over Hiroshima.
As the scale of powers available to us grows, the room for error shrinks. And as the number of advanced powers grows, the room for error shrinks further.
The knowledge explosion feeds back upon itself, resulting in an accelerating pace of further knowledge development. That is, both the scale and number of powers available to us is likely to grow at an ever faster pace, reducing the room for error at an ever faster pace. On the path we're on, sooner or later we arrive at the "one bad day, game over" scenario by some method.
Trying to make each power of vast scale safe as it rolls off the end of the knowledge explosion assembly line is a loser's game. What we need to do instead is learn how to take control of the knowledge explosion assembly line. Yes, this is a very challenging project, but it's not negotiable, because failure equals the end of the modern world as we know it.
The above may all sound very speculative and alarmist, until you realize that we're already in this situation. One bad day with nuclear weapons, game over. One bad day. Room for error shrunk to close to nothing.
Someone will now write that we've successfully managed nuclear weapons for 80 years. First, 80 years is nothing, a single life time. Second, on multiple occasions both the US and Russia have come within minutes of launching their arsenals BY MISTAKE. We aren't still here because we're smart, we're still here because so far we've been very very lucky.
If we keep creating powers of vast scale faster and faster while planning to fix them later, sooner or later that luck will run out.
We're like the man playing Russian Roulette who feels confident because he's pulled the trigger a number of times, and nothing bad has happened yet.
Well written, Olivia! Excited to see more of these ideas take root in government.
Thanks, Ben!
Thank you for this intelligent informed post. In reply....
I would ask that we think bigger than just accepting the old paradigm of "we will race ahead and try to fix any problems that arise". That's outdated thinking, in my view. That era ended in 1945 over Hiroshima.
As the scale of powers available to us grows, the room for error shrinks. And as the number of advanced powers grows, the room for error shrinks further.
The knowledge explosion feeds back upon itself, resulting in an accelerating pace of further knowledge development. That is, both the scale and number of powers available to us is likely to grow at an ever faster pace, reducing the room for error at an ever faster pace. On the path we're on, sooner or later we arrive at the "one bad day, game over" scenario by some method.
Trying to make each power of vast scale safe as it rolls off the end of the knowledge explosion assembly line is a loser's game. What we need to do instead is learn how to take control of the knowledge explosion assembly line. Yes, this is a very challenging project, but it's not negotiable, because failure equals the end of the modern world as we know it.
The above may all sound very speculative and alarmist, until you realize that we're already in this situation. One bad day with nuclear weapons, game over. One bad day. Room for error shrunk to close to nothing.
Someone will now write that we've successfully managed nuclear weapons for 80 years. First, 80 years is nothing, a single life time. Second, on multiple occasions both the US and Russia have come within minutes of launching their arsenals BY MISTAKE. We aren't still here because we're smart, we're still here because so far we've been very very lucky.
If we keep creating powers of vast scale faster and faster while planning to fix them later, sooner or later that luck will run out.
We're like the man playing Russian Roulette who feels confident because he's pulled the trigger a number of times, and nothing bad has happened yet.