For the researchers out there, how do you do research?<p>Background:<p>I've been looking at how to create a recurrent seq-to-seq model, that's not transformers. The ideas I implement do not work. It seems like off the well trodden path, there are traps everywhere - how should I tune parameters, add biases, normalize, is this dataset impossible, gradient explosion and vanishing, etc.<p>From a "research = gradient descent" point of view, I'm stuck at a point with no gradient - I have no idea what I'm doing wrong, or what to will get a better result. Am I missing a workflow. intuition, or tools, or other things?
Here's some AI research links I've collected over the past while, I put them in a public gist: <a href="https://gist.github.com/TikkunCreation/5de1df7b24800cc05b4823da4116f1ad" rel="nofollow">https://gist.github.com/TikkunCreation/5de1df7b24800cc05b482...</a><p>Karpathy's post about the research process in particular may be helpful for you