Speculating on such risks when the mechanisms are wholly unclear say much more about the speculator than the number. It reflects a personal position more than any objective analysis. We need to focus more on known risks, right in front of us now. One I've been thinking about is cyber security. There's pretty strong evidence that the AI labs have built, in the name of defense, the best cyber hacking tools on the planet. We also can't ignore how AI is affecting education, human interaction and cognition in general, though I think that's going to be more of an adaptation process. We clearly have the power to change the p(doom) through action. We need to start now.
1. I have often found that offering a specific numerical value (in this case, probability) conferred greater credibility. When I worked for a large bank and was arguing for technology budget, I found that if I said we needed, say, "$23.4 million" people assumed our budget forecasts were highly accurate (when they could have been precise, but still inaccurate), while $23 million was taken as more of an estimate.
2. Is there really "no reference class" to support the Inductive method? While AI systems have not yet caused a catastrophic or extinction event, there have been numerous instances of AI systems taking significant actions with negative consequences (e.g., attempting to change other systems and databases). Can this be analogized to "observing thousands of small [asteroid] impacts" being useful in estimating the likelihood of a large (extinction-level) asteroid impact?
"Claiming to have a high p(doom) (one’s estimate of the probability of AI doom) seems to have become a way to signal one’s identity and commitment to the cause."
Reminds me of what I also wrote 2 years ago "Publicly providing a value of p(doom) has become somewhat of a social phenomenon among AI researchers and other interested parties."
We argue many of the same points. I have suggested something more meaningful might be "risk = unpredictability * capability". At least you can somewhat benchmark unpredictability and capability, but of course that still doesn't allow you to calculate any specific risk.
Ultimately p(doom) is useless for assisting policy and likely harmful. But if we cared about something more imminent and relevant, we would probably be talking about p(dystopia) and AI being used to spy on everyone, warp public opinion, use in military weapons etc.
Speculating on such risks when the mechanisms are wholly unclear say much more about the speculator than the number. It reflects a personal position more than any objective analysis. We need to focus more on known risks, right in front of us now. One I've been thinking about is cyber security. There's pretty strong evidence that the AI labs have built, in the name of defense, the best cyber hacking tools on the planet. We also can't ignore how AI is affecting education, human interaction and cognition in general, though I think that's going to be more of an adaptation process. We clearly have the power to change the p(doom) through action. We need to start now.
A comment and a question:
1. I have often found that offering a specific numerical value (in this case, probability) conferred greater credibility. When I worked for a large bank and was arguing for technology budget, I found that if I said we needed, say, "$23.4 million" people assumed our budget forecasts were highly accurate (when they could have been precise, but still inaccurate), while $23 million was taken as more of an estimate.
2. Is there really "no reference class" to support the Inductive method? While AI systems have not yet caused a catastrophic or extinction event, there have been numerous instances of AI systems taking significant actions with negative consequences (e.g., attempting to change other systems and databases). Can this be analogized to "observing thousands of small [asteroid] impacts" being useful in estimating the likelihood of a large (extinction-level) asteroid impact?
"Claiming to have a high p(doom) (one’s estimate of the probability of AI doom) seems to have become a way to signal one’s identity and commitment to the cause."
Reminds me of what I also wrote 2 years ago "Publicly providing a value of p(doom) has become somewhat of a social phenomenon among AI researchers and other interested parties."
We argue many of the same points. I have suggested something more meaningful might be "risk = unpredictability * capability". At least you can somewhat benchmark unpredictability and capability, but of course that still doesn't allow you to calculate any specific risk.
Ultimately p(doom) is useless for assisting policy and likely harmful. But if we cared about something more imminent and relevant, we would probably be talking about p(dystopia) and AI being used to spy on everyone, warp public opinion, use in military weapons etc.
My similar and supporting argument. FYI.
https://www.mindprison.cc/p/pdoom-the-useless-ai-predictor
https://substack.com/@wisdomfirst/note/p-213789273?r=1nn7vp&utm_medium=ios&utm_source=notes-share-action