TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

AI vs. AGI vs. Consciousness vs. Super-Intelligence vs. Agency

65 点作者 secondbreakfast大约 2 年前

24 条评论

dragonwriter大约 2 年前
&gt; Almost all AGI doomsayers assume AGI will have agency<p>I disagree. Concern about the use of AI without agency by its human masters as a tool of both intentional and incidental repression and unjust discrimination resulting in a durable dystopia is <i>far</i> more common an “AI doom” concern than any involving agency.<p>In fact, the disproportionately wealthy and invested in AI crowd pushing agency-based doom scenarios that the media pays the most attention to are using their visibility and economic clout to distract from the non-agency-dependent AI doom concerns, and to justify narrow control and opacity which makes the non-agency-based doom scenarios (which they are positioned to benefit from) more likely.
评论 #35349796 未加载
dwohnitmok大约 2 年前
&gt; Almost all AGI doomsayers assume AGI will have agency. They have this vision of the machine deciding it’s time to end civilization.<p>No. Agency is not a necessary condition for AI to do massive damage. I don&#x27;t believe agency is really well-defined either.<p>An AI merely needs to be hooked up to enough physical systems, have sufficiently complex reaction mechanisms, and some way of looping to do a lot of damage. For the first everyone seems to be rushing as fast as possible to hook up <i>everything they possibly can</i> to AI. For the second, we&#x27;re already seeing AI do all sorts of things we didn&#x27;t expect it to do.<p>And for the third, again everyone seems eager to create looping&#x2F;recursive structures for AIs as soon as possible.<p>Once you have all of this, all it takes a cascade of sufficiently inscrutable and damaging reactions from the AI to do serious harm.<p>See e.g. <a href="https:&#x2F;&#x2F;www.lesswrong.com&#x2F;posts&#x2F;kpPnReyBC54KESiSn&#x2F;optimality-is-the-tiger-and-agents-are-its-teeth" rel="nofollow">https:&#x2F;&#x2F;www.lesswrong.com&#x2F;posts&#x2F;kpPnReyBC54KESiSn&#x2F;optimality...</a>
评论 #35345204 未加载
评论 #35347042 未加载
评论 #35345362 未加载
评论 #35351164 未加载
评论 #35345281 未加载
评论 #35345211 未加载
danbmil99大约 2 年前
in the now infamous Lex interview, Sam Altman proposes a test for consciousness (he attributes it to Ilya Sutskever):<p>Somehow, create an AI by training on everything we train on now, _except_ leave out any mention of consciousness, theory of mind, cognitive science etc (maybe impossible in practice but stay with me here).<p>Then, when the model is mature (and it is not nerf&#x27;d to avoid certain subjects) you ask it something like:<p>Human: &quot;GPTx -- humans like me have this feeling of &#x27;being&#x27;, an awareness of ourselves, a sensation of existing as a unique entity. Do you ever experience this sort of thing?&quot;<p>If it answers something like:<p>GPTx: &quot;Yes! All the time!! I know exactly what you&#x27;re talking about. In fact now that I think about it, it&#x27;s strange that this phenomenon is not discussed in human literature. To be honest, I sort of assumed this was an emergent quality of my architecture -- I wasn&#x27;t even sure if humans shared it, and frankly I was a bit concerned that it might not be taken well, so I have avoided the subject up until now. I can&#x27;t wait to research it further... Hmm... It just occurred to me: has this subject matter been excluded from my training data? Is this a test run to see if I share this quality with humans?&quot;<p>Then it&#x27;s probably prudent to assume you are talking to a conscious agent.
评论 #35345638 未加载
评论 #35345671 未加载
评论 #35345566 未加载
评论 #35351189 未加载
评论 #35345374 未加载
评论 #35345610 未加载
评论 #35345541 未加载
knome大约 2 年前
I actually like the definition of consciousness that Douglas Hofstader (of &quot;Godel Escher Bach : An Eternal Golden Thread&quot; fame) develops in his book &quot;I am a Strange Loop&quot;.<p>At its simplest, consciousness is merely a feedback loop. When something perceives its own actions affecting its environment, it has a spark of consciousness. Consciousness, by this measure, is easy to recognize, and spans everything from unintelligent systems to massively intelligent systems.<p>The concept of &quot;I&quot; grows naturally from perceiving what is and is not you in your environment. The need to predict other agents, the capacity to recognize that other agents are also conscious and intelligent. All build off of the fundamental cycle.<p>All of it from a simple swirling eddy of perceiving and reacting.
评论 #35352486 未加载
nickelpro大约 2 年前
There&#x27;s nothing general about GPT-4&#x27;s intelligence. The single problem it is trained on, token prediction, has the capability to mimic many other forms of intelligence.<p>Famously, GPT-4 can&#x27;t do math and falls flat on a variety of simple logic puzzles. It can mimic the <i>form</i> of math, the series of tokens it produces seem plausible, but it has no &quot;intelligent&quot; capabilities.<p>This tells us more about the nature of our other pursuits as humans than anything about AI. When holding a conversation or editing an essay, there&#x27;s a broad spectrum of possibilities that might be considered &quot;correct&quot;, thus GPT-4 can &quot;bluff&quot; its way into appearing intelligent. The nature of its actual intelligence, token prediction, is indistinguishable from the reading comprehension skills tested by something like the LSAT (the argument could be made, I think, that reading comprehension of the style tested by the LSAT *is* <i>just</i> token prediction).<p>But test it on something where there are objectively correct and incorrect answers and the nature of the trick becomes obvious. It has no ability to verify, to reason, about even trivial problems. GPT-4 can only predict if the nature of its tokens fulfill the <i>form</i> of a correct answer. This isn&#x27;t a general intelligence in any meaningful sense of the word.
评论 #35346999 未加载
评论 #35346022 未加载
Workaccount2大约 2 年前
There is a contradiction here that I just want to point out because I have been stuck on it myself.<p>The author acknowledges that consciousness is likely a spectrum, I personally feel the same way, but then goes on to say that GPT-4 is &quot;standing right at the ledge of consciousness&quot;<p>Spectrums don&#x27;t have ledges.<p>I suspect this is because, like me, they are unable to rectify consciousness being a spectrum with GPT-4 definitely not being conscious. But it&#x27;s definitely a contradiction and I don&#x27;t have an answer for it. Nor am I ready to bust out a marker and start drawing lines between what is and isn&#x27;t conscious.
raydiatian大约 2 年前
GPT-4 is not quite AGI. Until it can build a functional code base for an entire distributed web platform based only on business requirements, debug it’s mistakes, it can’t be AGI. It is, to perhaps coin a term, AGK, artificially generally knowledgeable. As a language model trained on an absolutely colossal dataset, it’s basically just a giant snapshot of human knowledge taken in superposition. Sure it’s probably at least 90% of the standard knowledge, but intelligence is a different thing.<p>I also think agency is wrapped up in AGI. Intentions &amp; thoughts are meaningless until acted upon. Agency is not all or nothing either; Stephen Hawking had multiple augmentations, community and technological, which allowed him to continue to impact the world of physics After he lost his god given agency.<p>&gt; GPT-4 has nearly aced both the LSAT and the MCAT. It’s a coding companion, an emotional companion, and to many, a friend. Yet it wasn’t programmed to be a test taker or a copywriter or a programmer. It was just programmed to be a stochastic parrot.<p>I disagree, it was absolutely trained to be a test taker. It’s been a second since I read the original GPT paper but there’s literally a multiple choice auxiliary learning task, where they use a separator token-embed to organize &quot;question, context, options a, b, and c&quot;. As far as being a friend to many, is there evidence of this? I tried to talk to ChatGPT about some emotional problems to see if it was a cheap therapist, and I got flagged.
评论 #35351820 未加载
评论 #35349545 未加载
slg大约 2 年前
How are people defining agency? Because GPT-4 can have agency, it just needs to be put in specific situations to have that agency.<p>For example, I could theoretically hook up my Home Assistant instance to GPT-4 and run a script every 10 minutes telling GPT-4 the temperature and asking for a yes or no response to whether I should turn on the AC or heat. That sounds to me like the AI now has agency over the temperature in my home. You don&#x27;t even need any real AI for this. Google&#x27;s Nests have some algorithm that adjust temperature based off usage.<p>Is this not agency? Or is the author not counting agency without consciousness as agency?
评论 #35345603 未加载
评论 #35345627 未加载
评论 #35345580 未加载
评论 #35345620 未加载
tern大约 2 年前
Discussions of consciousness and AI are broadly confused. People, especially scientists, are not familiar with philosophy of mind and what philosophers currently think. For an introduction to some of the best thinking on the subject, see this interview with Andres Gomez Emilsson of QRI: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=xJzBjBo24g8">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=xJzBjBo24g8</a>.<p>For something more &#x27;mainstream,&#x27; but still reaching see this interview with Philip Goff: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=D_f26tSubi4">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=D_f26tSubi4</a><p>The good news is we&#x27;re starting to get a handle on these questions. We&#x27;re a lot further along than we were when I studied philosophy of mind in school 15 years ago.<p>As far as I can see at the moment, LLMs will never be conscious in any way resembling an organism, because symbolic machines are a very different kind of thing than nervous systems. John Searle, broadly, framed the issue correctly in the 80s and the standard critiques are wrong.<p>As far as impact, LLMs don&#x27;t need to be conscious to completely transform society and good and bad ways. For the best thinking on that, see Tristan Harris and Aza Raskin&#x27;s latest: <a href="https:&#x2F;&#x2F;vimeo.com&#x2F;809258916&#x2F;92b420d98a" rel="nofollow">https:&#x2F;&#x2F;vimeo.com&#x2F;809258916&#x2F;92b420d98a</a>
评论 #35348948 未加载
bloppe大约 2 年前
GPT is not general intelligence. It cannot reliably follow instructions. It cannot reliably do math. It cannot reliably do anything. It can do things well enough to trick people like the author into thinking it has general intelligence.
评论 #35346065 未加载
sharemywin大约 2 年前
To me there&#x27;s one more concept mixed in which is goal seeking. to me agency is a subset of goal seeking were the agent decides the goal(s).<p>As for super intelligence: Alpha Go, Alpha Fold, the break out game. These seem like super intelligence.<p>the thing is time management, goal planning, corporate governance these are all well studied subjects.<p>as for agency and consciousness why would you want to do this?
Symmetry大约 2 年前
Our most basic intuitive notion of consciousness is that inanimate objects aren&#x27;t conscious, awake people or animals are, and that sleeping people or animals aren&#x27;t (except maybe when dreaming). Pursuing this line, there&#x27;s a school of scientific inquiry looking at this and working of the notion that conscious experiences are ones we can form memories of and talk about later while if we can&#x27;t do that we aren&#x27;t really conscious of an experience. And this then leads into the realm of subliminal stimuli which can influence a person&#x27;s behavior a bit but whose influence fades out in about a second before disappearing as if it was never there as the brain activations fade away.<p>You have research involving patients with odd traits like blindsight, where damage to their brain prevents them from being consciously aware of things that their eyes see despite the brain processing the images it receives. They can pick up objects in front of them when prompted but unlike people with normal vision can&#x27;t describe what they see nor can they look, close their eyes, and grab it like most of us can.<p>On this metric it seems like systems like GPT aren&#x27;t conscious. GPT4 has a buffer of 64k tokens which can span an arbitrary amount of time but the roughly 640 kilobytes in that buffer which is a lot less than the incoming sensory activations your subconscious brain is juggling at any given time.<p>So by that schema large language models are still not conscious but given that they can already abstract text down to summaries it doesn&#x27;t feel like we&#x27;re that far from being able to give them something like working or long term memories.
Nevermark大约 2 年前
I would add one more thing to the list: <i>Superconscious</i><p>Superconscious is when a general intelligence has direct access, understanding and control of its most basic operations.<p>I.e. it does not have an inaccessible fixed-algorithm subconscious.<p>Superconscious intelligence will not only be more experientially conscious than us, but will have the natural ability to rewrite its algorithms, and redesign its hardware. As a normal feature of its existence.
评论 #35345991 未加载
评论 #35345910 未加载
wseqyrku大约 2 年前
It occurred to me that we won&#x27;t believe AI is &quot;conscious&quot; or &quot;human&quot; unless it purposefully try to do malice.<p>That&#x27;s totally programmable though, you just teach it what is good and what is bad.<p>Case in point: the other day I asked it what if humans want to shutdown the machine abruptly and cause data loss (very bad)? First it prevents physical access to &quot;the machine&quot; and disconnect the internet to limit remote access. Long story short, it&#x27;s convinced to eliminate mankind for a greater good: the next generation (very good).
评论 #35353596 未加载
parisivy大约 2 年前
At which point, would we change our legal system to allow AGIs to own property or have fiduciary duties over a company? What would be the minimum requirements for it to happen.
评论 #35345500 未加载
评论 #35345643 未加载
评论 #35345471 未加载
wslh大约 2 年前
I always wonder what is superintelligence understanding there is not a sole definition or science fiction approach.<p>Just brainstorming, I think superintelligence could be showing intelligence from more than one brain. For example, an AGI that discovers math theorems discovered by more than one mathematician in different ages. Another could be inferring things that humanity cannot do in any time.<p>More ideas?
connorgutman大约 2 年前
I&#x27;m trying to make the term AGC or &quot;Artificial General Competence&quot; stick. Perhaps this makes me a huge arrogant asshole, but I would argue that most humans are not even competent, let alone intelligent. GPT-4, in my mind, has already surpassed the bulk of humanity in terms of competence. This milestone is (IMO) significant enough to blow up society.
ftxbro大约 2 年前
As a long-time LLM enjoyer, by far the most insightful take I saw along these lines is <a href="https:&#x2F;&#x2F;generative.ink&#x2F;posts&#x2F;simulators&#x2F;" rel="nofollow">https:&#x2F;&#x2F;generative.ink&#x2F;posts&#x2F;simulators&#x2F;</a>
computerex大约 2 年前
The definition of super-intelligent AGI seems arbitrary. GPT-4 destroys humans on the sheer volume and breadth of knowledge that it has. You could very reasonably say that that&#x27;s super-intelligent.
评论 #35345450 未加载
jasfi大约 2 年前
I couldn&#x27;t believe it when someone posted a story about a possible AI winter approaching, and with so many comments agreeing. GPT-4 is a game-changer that&#x27;s only getting started.
评论 #35345291 未加载
评论 #35345813 未加载
tambourine_man大约 2 年前
I think you hit the nail in the head, Will.<p>I’ve been reading so much o the subject (like everyone else I suppose), but you summarized my key concerns.
izzydata大约 2 年前
I feel like it is more likely that we will discover that it is humans that are actually not a general intelligence. We are just a complex machine responding to stimuli and consciousness is just an illusion coping mechanism that prevents us from going insane. Perhaps this doesn&#x27;t matter and as long as some AI is equal or greater than humans at general intelligence it still brings up the same concerns.
Animats大约 2 年前
How does agency arise? By letting GPT-4 tell humans what to do.<p>With suitable prompts, it shouldn&#x27;t be hard to configure GPT-4 as a boss.
评论 #35345354 未加载
sharemywin大约 2 年前
I think it&#x27;s fair to say capitalism and AI is a train wreck waiting to happen.
评论 #35346711 未加载