TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Ask HN: AI/ML papers to catch up with current state of AI?

222 点作者 hahnchen超过 1 年前
I used to be into ML back in the R-CNN, GAN, ResNet era and would read papers&#x2F;blogs.<p>Seems like ML is taking off recently and I want to get back into it! So far on my list I have attention is all you need, qlora, llama’s and q learning. Suggestions?

28 条评论

hapanin超过 1 年前
Since nobody is actually recommending papers, here&#x27;s an incomplete reading list that I sent out to some masters students I work with so they can understand the current research (academic) my little team is doing:<p>Paper reference &#x2F; main takeaways &#x2F; link<p>instructGPT &#x2F; main concepts of instruction tuning &#x2F; <a href="https:&#x2F;&#x2F;proceedings.neurips.cc&#x2F;paper_files&#x2F;paper&#x2F;2022&#x2F;hash&#x2F;b1efde53be364a73914f58805a001731-Abstract-Conference.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;proceedings.neurips.cc&#x2F;paper_files&#x2F;paper&#x2F;2022&#x2F;hash&#x2F;b...</a><p>self-instruct &#x2F; bootstrap off models own generations &#x2F; <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2212.10560.pdf" rel="nofollow noreferrer">https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2212.10560.pdf</a><p>Alpaca &#x2F; how alpaca was trained &#x2F; <a href="https:&#x2F;&#x2F;crfm.stanford.edu&#x2F;2023&#x2F;03&#x2F;13&#x2F;alpaca.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;crfm.stanford.edu&#x2F;2023&#x2F;03&#x2F;13&#x2F;alpaca.html</a><p>Llama 2 &#x2F; probably the best chat model we can train on, focus on training method. &#x2F; <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2307.09288" rel="nofollow noreferrer">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2307.09288</a><p>LongAlpaca &#x2F; One of many ways to extend context, and a useful dataset &#x2F; <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2309.12307" rel="nofollow noreferrer">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2309.12307</a><p>PPO &#x2F; important training method &#x2F; idk just watch a youtube video<p>Obviously these are specific to my work and are out of date by ~3-4 months but I think they do capture the spirit of &quot;how do we train LLMs on a single GPU and no annotation team&quot; and are frequently referenced simply by what I put in the &quot;paper reference&quot; column.
评论 #38663062 未加载
评论 #38662687 未加载
评论 #38659012 未加载
kozikow超过 1 年前
My view is to focus on doing stuff. That&#x27;s what I did. Pick up some task you want the model to do, try finetuning llama, playing with APIs from OpenAI, etc. Googling and asking GPT along the way.<p>Foundational model training got so expensive that unless you can get hired by &quot;owns nuclear power plant of GPUs&quot; you are not going to get any &quot;research&quot; done. And as the area got white-hot those companies have more available talent than hardware nowadays. So just getting into the practitioner area is the best way to get productive with those models. And you improve as a practitioner by practicing, not by reading papers.<p>If you&#x27;re at the computer, your time is best spent writing code and interacting with those models in my opinion. If you cannot (e.g. commute) I listen to some stuff (e.g. <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=zjkBMFhNj_g" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=zjkBMFhNj_g</a> - Anything from Karpathy on youtube, or <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;@YannicKilcher" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.youtube.com&#x2F;@YannicKilcher</a> channel).
carlossouza超过 1 年前
<a href="https:&#x2F;&#x2F;trendingpapers.com&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;trendingpapers.com&#x2F;</a><p>This tool can help you find what&#x27;s new &amp; relevant to read. It&#x27;s updated every day (based on ArXiv).<p>You can filter by category (Computer Vision, Machine Learning, NLP, etc), by release date, but most importantly, you can rank by PageRank (proxy of influence&#x2F;readership), PageRank growth (to see the fastest growing papers in terms of influence), total # of citations, etc...
评论 #38654526 未加载
kasperni超过 1 年前
Maybe this tweet by John Carmack can help you:<p>This is a great little book to take you from “vaguely understand neural networks” to the modern broad state of practice. I saw very little to quibble with. <a href="https:&#x2F;&#x2F;fleuret.org&#x2F;francois&#x2F;lbdl.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;fleuret.org&#x2F;francois&#x2F;lbdl.html</a>
评论 #38653205 未加载
评论 #38676809 未加载
d_burfoot超过 1 年前
Bear in mind that ML skillset is now bifurcating into two components. On the one side are the people who work at places like OpenAI&#x2F;DeepMind&#x2F;Mistral&#x2F;etc, who have billion dollar compute budgets. They are the ones who will create the foundational models. At this point a lot of this work is very technically narrow, dealing with CUDA, GPU issues, numerical stability, etc. On the other side are people who are using the models through the APIs in various ways. This is much more open-ended and potentially creative, but you don&#x27;t need to know how QLearning works to do this.<p>It&#x27;s a bit analogous to the situation with microprocessors. There is a ton of deep technical knowledge about how chips work, but most of this knowledge isn&#x27;t critical for mainstream programming.
magoghm超过 1 年前
The book that just came out, &quot;Understanding Deep Learning&quot;, is an excellent overview of the current state of AI: <a href="https:&#x2F;&#x2F;udlbook.github.io&#x2F;udlbook&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;udlbook.github.io&#x2F;udlbook&#x2F;</a><p>Read that first, then to keep up to date you can follow up with any papers that seem interesting to you. A good way to be aware of the interesting papers that come out is to follow @_akhaliq on X: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;_akhaliq" rel="nofollow noreferrer">https:&#x2F;&#x2F;twitter.com&#x2F;_akhaliq</a>
评论 #38654737 未加载
评论 #38657423 未加载
评论 #38655529 未加载
评论 #38657505 未加载
评论 #38678574 未加载
jpdus超过 1 年前
Hey, imho best overall technical intro to LLMs (I guess that´s your main interest as you mentioned qlora + llama) is by Simon Willis [1]. Additionally or if you prefer videos, the recent 1h &quot;busy persons intro&quot; by Andrei Karpathy is great + dense as well [2].<p>[1] <a href="https:&#x2F;&#x2F;simonwillison.net&#x2F;2023&#x2F;Aug&#x2F;3&#x2F;weird-world-of-llms&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;simonwillison.net&#x2F;2023&#x2F;Aug&#x2F;3&#x2F;weird-world-of-llms&#x2F;</a> [2] <a href="https:&#x2F;&#x2F;youtu.be&#x2F;zjkBMFhNj_g?si=M6pRX66NrRyPM8x-" rel="nofollow noreferrer">https:&#x2F;&#x2F;youtu.be&#x2F;zjkBMFhNj_g?si=M6pRX66NrRyPM8x-</a><p>EDIT: Maybe I misunderstood as you asked about papers, not general intros. I don´t think that reading papers is the best way to &quot;catch up&quot; as the pace is rapid and knowledge very decentralized. I can confirm what Andrej recently wrote on X [3]:<p>&quot;Unknown to many people, a growing amount of alpha is now outside of Arxiv, sources include but are not limited to:<p>- <a href="https:&#x2F;&#x2F;github.com&#x2F;trending">https:&#x2F;&#x2F;github.com&#x2F;trending</a><p>- HN<p>- that niche Discord server<p>- anime profile picture anons on X<p>- reddit&quot;<p>[3] <a href="https:&#x2F;&#x2F;twitter.com&#x2F;karpathy&#x2F;status&#x2F;1733968385472704548" rel="nofollow noreferrer">https:&#x2F;&#x2F;twitter.com&#x2F;karpathy&#x2F;status&#x2F;1733968385472704548</a>
评论 #38655224 未加载
评论 #38654346 未加载
antirez超过 1 年前
This one is very good, and will provide certain key insights on the way you should think at NNs. -&gt; <a href="https:&#x2F;&#x2F;www.amazon.it&#x2F;Deep-Learning-Python-Francois-Chollet&#x2F;dp&#x2F;1617294438&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.amazon.it&#x2F;Deep-Learning-Python-Francois-Chollet&#x2F;...</a><p>This is a good explanation of the Transformer details -&gt; <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=bCz4OMemCcA&amp;ab_channel=UmarJamil" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=bCz4OMemCcA&amp;ab_channel=UmarJ...</a><p>This is old but covers a lot of background that you needs to know to understand very well the rest. What I like of this book is that it often explains in a very intuitive way the motivations behind certain choices. -&gt; <a href="https:&#x2F;&#x2F;www.amazon.it&#x2F;Natural-Language-Processing-Pytorch-Applications&#x2F;dp&#x2F;1491978236&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.amazon.it&#x2F;Natural-Language-Processing-Pytorch-Ap...</a>
knbknb超过 1 年前
Once a week (at least!) some research group publishes another review paper to the cs.AI section on ArXiv. Look for new [papers with &quot;survey&quot; in the title](<a href="https:&#x2F;&#x2F;arxiv-sanity-lite.com&#x2F;?q=survey&amp;rank=time&amp;tags=cs.AI&amp;pid=&amp;time_filter=300&amp;svm_c=0.01&amp;skip_have=no" rel="nofollow noreferrer">https:&#x2F;&#x2F;arxiv-sanity-lite.com&#x2F;?q=survey&amp;rank=time&amp;tags=cs.AI...</a>). You&#x27;ll get surveys on every conceivable subtopic of ML&#x2F;AI.
lukeinator42超过 1 年前
I&#x27;d also add &quot;Deep reinforcement learning from human preferences&quot; <a href="https:&#x2F;&#x2F;proceedings.neurips.cc&#x2F;paper_files&#x2F;paper&#x2F;2017&#x2F;file&#x2F;d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf" rel="nofollow noreferrer">https:&#x2F;&#x2F;proceedings.neurips.cc&#x2F;paper_files&#x2F;paper&#x2F;2017&#x2F;file&#x2F;d...</a> and &quot;Training language models to follow instructions with human feedback&quot; <a href="https:&#x2F;&#x2F;proceedings.neurips.cc&#x2F;paper_files&#x2F;paper&#x2F;2022&#x2F;file&#x2F;b1efde53be364a73914f58805a001731-Paper-Conference.pdf" rel="nofollow noreferrer">https:&#x2F;&#x2F;proceedings.neurips.cc&#x2F;paper_files&#x2F;paper&#x2F;2022&#x2F;file&#x2F;b...</a>.<p>These papers outline the approach of reinforcement learning from human feedback which is being used to train lots of these LLMs such as ChatGPT.
andyjohnson0超过 1 年前
I kind of despair of keeping up to date with ML, at least to the extent that I might ever get current enough to be paid to work with it. I did Andrew Ng&#x27;s Coursera specialisation a few years back - and I&#x27;ve worked through some of the developer-oriented courses, implemented some stuff. read more than a few books, read papers (the ones I might have a hope of understanding), and tried to get a former employer to take it seriously. But its seeming like unless you have a PhD or big-co experience then its very difficult to keep up to date by working in the field.<p>Notwithstanding the above, I&#x27;d agree with others here who suggest learning by doing&#x2F;implementing, not reading papers.
gschoeni超过 1 年前
I put together a reading list for Andrej Karpathy&#x27;s intro to LLMs that would be helpful for all of the latest LLM and multi-modal architectures:<p><a href="https:&#x2F;&#x2F;blog.oxen.ai&#x2F;reading-list-for-andrej-karpathys-intro-to-large-language-models-video&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;blog.oxen.ai&#x2F;reading-list-for-andrej-karpathys-intro...</a>
cs702超过 1 年前
<i>Build something</i> of personal interest to you. Start by looking for similar open-source projects online. Look at the online posts of the authors. <i>Then</i> look for the papers that <i>you</i> think will be useful for <i>your</i> project. Before you know it, you&#x27;ll become an expert in your area of interest.<p>Above all, be wary of programmatic lists that claim to track the most important recent papers. There&#x27;s a ridiculous amount of <i>hype&#x2F;propaganda</i> and <i>citation hacking</i> surrounding new AI research, making it hard to discern what will truly stand the test of time. Tomas Mikolov just posted about this.[a]<p>---<p>[a] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=38654038">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=38654038</a>
maxlamb超过 1 年前
Part 2 of the fast.ai course might be a good start: <a href="https:&#x2F;&#x2F;course.fast.ai&#x2F;Lessons&#x2F;part2.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;course.fast.ai&#x2F;Lessons&#x2F;part2.html</a>
auntienomen超过 1 年前
I found Cosma Shalizi&#x27;s notes on the subject pretty insightful.<p><a href="http:&#x2F;&#x2F;bactra.org&#x2F;notebooks&#x2F;nn-attention-and-transformers.html" rel="nofollow noreferrer">http:&#x2F;&#x2F;bactra.org&#x2F;notebooks&#x2F;nn-attention-and-transformers.ht...</a><p>Definitely read through to the last section.
评论 #38655013 未加载
eurekin超过 1 年前
<a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;@algorithmicsimplicity" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.youtube.com&#x2F;@algorithmicsimplicity</a> - that series cleared up the fundamental question about transformers I couldn&#x27;t find an answer for in many recommended materials.<p>Here&#x27;s also nice tour de building blocks, which could also double as transformers&#x2F;tensorflow API reference documentation: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=eMXuk97NeSI&amp;t=207s" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=eMXuk97NeSI&amp;t=207s</a><p>The #1 visualization of architecture and size progression: <a href="https:&#x2F;&#x2F;bbycroft.net&#x2F;llm" rel="nofollow noreferrer">https:&#x2F;&#x2F;bbycroft.net&#x2F;llm</a>
gurovich超过 1 年前
This resource has been invaluable to me: <a href="https:&#x2F;&#x2F;paperswithcode.com&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;paperswithcode.com&#x2F;</a><p>From the past examples you give it sounds like you were into computer vision. There’s been a ton of developments since then, and I think you’d really enjoy the applications of some of those classic convolutional and variational encoder techniques in combination with transformers. A state of the art multimodal non-autoregressive neural net model such as Google’s Muse is a nice paper to work up to, since it exposes a breadth of approaches.
sgt101超过 1 年前
No emergence<p>[2304.15004] Are Emergent Abilities of Large Language Models a Mirage? - arXiv <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2304.15004" rel="nofollow noreferrer">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2304.15004</a><p>Can&#x27;t plan<p><a href="https:&#x2F;&#x2F;openreview.net&#x2F;forum?id=X6dEqXIsEW" rel="nofollow noreferrer">https:&#x2F;&#x2F;openreview.net&#x2F;forum?id=X6dEqXIsEW</a><p>No compositionality <a href="https:&#x2F;&#x2F;openreview.net&#x2F;forum?id=Fkckkr3ya8" rel="nofollow noreferrer">https:&#x2F;&#x2F;openreview.net&#x2F;forum?id=Fkckkr3ya8</a><p>Apart from that it&#x27;s great
sthoward超过 1 年前
Would suggest our weekly paper club called Arxiv Dive - <a href="https:&#x2F;&#x2F;lu.ma&#x2F;oxenbookclub" rel="nofollow noreferrer">https:&#x2F;&#x2F;lu.ma&#x2F;oxenbookclub</a>. You can see past ones on our blog (<a href="https:&#x2F;&#x2F;blog.oxen.ai&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;blog.oxen.ai&#x2F;</a>) - have covered papers like Mamba, CLIP, Attention is all you need, and more. We also do a &quot;hands on&quot; session with live code, models, and real world data on Fridays!
aakashg99超过 1 年前
I recently started reading research papers related to GPTs and LLMs. I have listed them here, along with my short synopsis and links to their code and datasets<p><a href="https:&#x2F;&#x2F;www.thinkevolveconsulting.com&#x2F;large-language-models-30-papers-that-matter" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.thinkevolveconsulting.com&#x2F;large-language-models-...</a>
youngprogrammer超过 1 年前
Little late to this thread but from my list:<p>LLM (foundational papers)<p>* Attention is all you need - transformers + self attention<p>* BERT - first masked LM using transformers + self attention<p>* GPT3 - big LLM decoder (Basis of gpt4 and most LLM)<p>* Instruct GPT or TKInstruct (instruction tuning enables improved zero shot learning)<p>* Chain of Thought (improve performance via prompting)<p>some other papers which are become trendy depending on your interest<p>* RLHF - RL using human feedback<p>* Lora - make models smaller<p>* MoE - kind of ensembling<p>* self instruct - self label data<p>* constitutional ai - self alignment<p>* tree of thought - like CoT but a tree<p>* FastAttention,Longformer - optimized attention mechanisms<p>* React - agents
lysecret超过 1 年前
The good (and some might say bad thing) is that when it comes to fundamental technologies there are only 2 that are relevant:<p>1. Transformers 2. Diffusion<p>The benefit is that, focus on understanding them both reeaaalllyy well and you are at the forefront of research;)<p>Also, what is the reason you want to do this? If it is about building some kind of AI enabled app, you don&#x27;t have to read anything. Get an API key and let&#x27;s go the barrier has never been lower.
评论 #38655567 未加载
评论 #38655281 未加载
pomatic超过 1 年前
Posted in another thread, but sadly I got no replies...<p>Related question: how can I learn how to read the mathematical notation used in AI&#x2F;ML papers? Is there a definitive work that describes the basics? I am a post-grad Engineer, so I know the fundamentals, but I&#x27;m really struggling with a lot of the Arxiv papers. Any pointers hugely appreciated.
评论 #38654475 未加载
评论 #38654573 未加载
neduma超过 1 年前
From ChatGPT:<p>&gt;&gt; To catch up with the current state of Artificial Intelligence and Machine Learning, it&#x27;s essential to look at the latest and most influential research papers. Here are some categories and specific papers you might consider:<p>1. *Foundational Models and Large Language Models*: - Papers on GPT (Generative Pre-trained Transformer) series, particularly the latest like GPT-4, which detail the advancements in language models. - Research on BERT (Bidirectional Encoder Representations from Transformers) and its variants, which are pivotal in understanding natural language processing.<p>2. *Computer Vision*: - Look into papers on Convolutional Neural Networks (CNNs) and their advancements. - Research on object detection, image classification, and generative models like Generative Adversarial Networks (GANs).<p>3. *Reinforcement Learning*: - Papers from DeepMind, like those on AlphaGo and AlphaZero, showcasing advances in reinforcement learning. - Research on advanced model-free algorithms like Proximal Policy Optimization (PPO).<p>4. *Ethics and Fairness in AI*: - Papers discussing the ethical implications and biases in AI, including work on fairness, accountability, and transparency in machine learning.<p>5. *Quantum Machine Learning*: - Research on the integration of quantum computing with machine learning, exploring how quantum algorithms can enhance ML models.<p>6. *Healthcare and Bioinformatics Applications*: - Papers on AI applications in healthcare, including drug discovery, medical imaging, and personalized medicine.<p>7. *Robotics and Autonomous Systems*: - Research on the intersection of AI and robotics, including autonomous vehicles and drone technology.<p>8. *AI in Climate Change*: - Papers discussing the use of AI in modeling, predicting, and combating climate change.<p>9. *Interpretable and Explainable AI*: - Research focusing on making AI models more interpretable and explainable to users.<p>10. *Emerging Areas*: - Papers on new and emerging areas in AI, such as AI in creative arts, AI for social good, and the integration of AI with other emerging technologies like the Internet of Things (IoT).<p>To find these papers, you can check academic journals like &quot;Journal of Machine Learning Research,&quot; &quot;Neural Information Processing Systems (NeurIPS),&quot; and &quot;International Conference on Machine Learning (ICML),&quot; or platforms like arXiv, Google Scholar, and ResearchGate. Additionally, following key AI research labs like OpenAI, DeepMind, Facebook AI Research, and university research groups can provide insights into the latest developments.
ricklamers超过 1 年前
If you want good up to date resources on the applied side I’d recommend checking out <a href="https:&#x2F;&#x2F;hamel.dev&#x2F;notes&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;hamel.dev&#x2F;notes&#x2F;</a>
hoerzu超过 1 年前
At the Twitter section at the bottom there is usually good papers <a href="https:&#x2F;&#x2F;news.mioses.com" rel="nofollow noreferrer">https:&#x2F;&#x2F;news.mioses.com</a>
yieldcrv超过 1 年前
you don’t need papers, Arxiv are self aggrandizement from some meme in East Asia<p>just join communities on discord or locallama on reddit
voidz7超过 1 年前
can I get some insights on ai and robotics some papers to implement and get my hands dirty