World Emulation via Neural Network

Its a time capsule, among other things. I want to take many, many videos of my grandpa's farm, and be able to walk around in it in VR using something like this in the future.

&gt; So, if traditional game worlds are paintings, neural worlds are photographs. Information flows from sensor to screen without passing through human hands.<p>I don&#x27;t get this analogy at all. Instead of a human information flows through a neural network which alters the information.<p>&gt; Every lifelike detail in the final world is only there because my phone recorded it.<p>I might be wrong here but I don&#x27;t think this is true. It might also be there because the network inferred that it is there based on previous data.<p>Imo this just takes the human out of a artistic process - creating video game worlds and I&#x27;m not sure if this is worth archiving.

This might be a vague question, but what kind of intuition or knowledge do you need to work with these kind of things, say if you want to make your own model? Is it just having experience with image generation and trying to incorporate relevant inputs that you would expect in a 3D world, like the control information you added for instance?

Very nice work. Seems very similar to the Oasis Minecraft simulator.<p><a href="https:&#x2F;&#x2F;oasis.decart.ai&#x2F;" rel="nofollow">https:&#x2F;&#x2F;oasis.decart.ai&#x2F;</a>

I think this is very interesting because you seem to have reinvented NeRF, if I’m understanding it correctly. I only did one pass through but it looks at first glance like a different approach entirely.<p>More interesting is that you made an easy to use environment authoring tool that (I haven’t tried it yet) seems really slick.<p>Both of those are impressive alone but together that’s very exciting.

Really cool. How much compute did you require to successfully train these models? Is it in the ballpark of something you could do with a single gaming GPU? Or did you spin up something fancier?<p>edit: I see now that you mention a pricepoint of 100 GPU-hours&#x2F;roughly 100$. My mistake.

What used to be cutting edge research not so long ago is now a fun hobby project. I love it.

It&#x27;s a really interesting project, reminds me of the 360° videos I used to watch on my phone, back in 2015.<p>But there&#x27;s one thing that I&#x27;m a little bit worried about: I was getting like 8 stable FPS on my 3 years old flagship phone. My concern is that these models are not optimized to run on this type of hardware, which may or may not lead to hardware obsolescence quicker than planned. And it&#x27;s not like these aren&#x27;t powerful, they really are.

This is great but I think I&#x27;ll stick to mushrooms.

Next we should try &quot;Excel emulation via Neural Network&quot;. We get rid of a lot of intermediate steps, calculations, user interface etc!<p>What could go wrong?<p>Jokes aside, this is insanely cool!

I love this! Your results seem comparable to the counter strike or minecraft models from a bit ago with massively less compute and data. It&#x27;s particularly cool that it uses real world data. I&#x27;ve been wanting to do something like this for a while, like capturing a large dataset while backpacking in the cascades :)<p>I didn&#x27;t see it in an obvious place on your github, do you have any plans to open source the training code?

Appreciate this article that shows some failures on the way to a great result. Too many times, people only show how the polished end-result: look, I trained this AI and it produces these great results. The world dissolving was very interesting to see, even if I&#x27;m not sure I understand how it got fixed.

This is very impressive for a hobby project. I was wondering if you were planning to release the source code. Being able to create client-hosted, low-requirement neural networks for world generation could be really useful for game dev or artistic projects.

I want to see a spiritual successor to LSD: Dream Emulator based on this.<p><a href="https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;LSD:_Dream_Emulator" rel="nofollow">https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;LSD:_Dream_Emulator</a>

Gotta say, Ive always wanted to try building something like this myself. That kind of grind pays off way more than shiny announcements imo.

Awesome work &#x2F; demo &#x2F; blog<p>Link to the demo in case people miss it [1]<p>&gt; using a customized camera app which also recorded my phone’s motion<p>Using phone&#x27;s gyro as a proxy for &quot;controls&quot; is very clever<p>[1] <a href="https:&#x2F;&#x2F;madebyoll.in&#x2F;posts&#x2F;world_emulation_via_dnn&#x2F;demo&#x2F;" rel="nofollow">https:&#x2F;&#x2F;madebyoll.in&#x2F;posts&#x2F;world_emulation_via_dnn&#x2F;demo&#x2F;</a>

This seems incredibly powerful.<p>Imagine a similar technique but with productivity software.<p>And a pre-trained network that adapts quickly.

Is this a solo&#x2F;personal project? If it is is indeed very cool.<p>Is OP the blog’s author? Because in the post the author said that the purpose of the project is to show why NN are truly special and I wanted a more articulate view of why he&#x2F;she thinks that? Good work anyway!

It would be quite interesting to try to mess with the neural representations do add or remove the images of some objects there. I&#x27;m also curious if the topology of the actual place is similar to the topology of the embedding space.

Amazing work. Could you elaborate on the model architecture and the process that lead you to using this architecture?