Turning web design mockups into code with Deep Learning

A lot of people in this thread seem to think that this is a neural network that takes an image and produces HTML, when that&#x27;s not the case here at all.<p>This is a neural network that takes an image and predicts very simple blocks (like BODY, TEXT, BTN-GREEN in the bootstrap example) and then uses a map to convert them to well-formed HTML. While I think it&#x27;s a great learning example I think it&#x27;s important to note that this does generalize at all to any other websites -- you are NOT going to replace an actual person writing HTML with anything like this.<p>You can see the mapping here: <a href="https:&#x2F;&#x2F;github.com&#x2F;emilwallner&#x2F;Screenshot-to-code-in-Keras&#x2F;blob&#x2F;master&#x2F;floydhub&#x2F;Bootstrap&#x2F;compiler&#x2F;assets&#x2F;web-dsl-mapping.json" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;emilwallner&#x2F;Screenshot-to-code-in-Keras&#x2F;b...</a>

Well, this seems like it could be useful for the times you just want a quick prototype mocked up and can&#x27;t be bothered to code it, or when you&#x27;re dealing with sites that don&#x27;t need much in the way of dynamic functionality.<p>However, it also makes me think that:<p>A: Maybe developers (and software engineers in general) should stop thinking their own jobs are necessarily safe from automation, since something like this could be the first step to the field going the same way as lorry or taxi driving will in future.<p>B: That agencies might start seeing their basic web development work start drying up in the not too distant future.

This is really great.<p>Many years ago, I was neck deep in frontend coding. I got so good that when given a mockup, I could code the skeleton of the page &quot;blind&quot; without viewing it in the browser. I would load the page at the end and grade myself on how accurate I was. I&#x27;ve always wanted to do a contest with other frontend coders to see who could get closest to a complex layout—like the NYTimes—in one go.<p>One day, this type of skill will become a curiosity—a relic of the past similar to horseback riding. I would be happy to see the automation of pure translation of boxes into code.

I&#x27;m still waiting for the generation of tools that are native to the medium—fluid, responsive, interactive, and aware of the final context for the design. So we don&#x27;t start with static mockups that have to be converted, but rather build web-native from the beginning.<p>There are some promising moves that direction (such as responsive resizing in Sketch and projects to automatically convert sketch files to React components), but we&#x27;re not there yet—most designers I know are still producing largely static designs that have to be put through development cycles.<p>And I know you <i>can</i> live design with HTML and CSS, skipping the static mockup phase. That&#x27;s my preferred method at the moment, but I&#x27;m here to tell you it&#x27;s still pretty slow and tedious and makes me long for better web design tooling.

While cool I am not sure if this is genuinely useful. I can now take my design and convert it to HTML- now what?<p>I still have to integrate my api, handle responsiveness, add JavaScript and other animations&#x2F;actions etc. So I’ll probably end up rewriting most of it anyway.<p>What I actually get out of this is not much better than emebedding the original image in an img tag, in fact it might be worse, because it creates technical debt that I now have to maintain.

Wow, really cool article!<p>I am not a huge fan of front-end dev, so I am really excited about the great opportunities behind this work&#x2F;research. It will be another major step in the democratization of UI and less work at maintaining the web views.<p>My biggest concern is the model capacity. My question is: in your opinion, will it be able to handle very long HTML page? not to mention when it will be added a lot of CSS or JS. Even with more layers, will the model be able to generate corrected syntax(for really long pages)? will it need some additional modules(not only layers) to handle this? maybe a different problem formulation?

Another step closer to my dream: a cross-compiler that takes as input a Powerpoint presentation, and outputs the enclosed-described mobile app, cloud-hosted website&#x2F;backend, and Delaware C Corporation.

I&#x27;m not sure about the usefulness of coding UI prototypes.<p>When I owned a web design firm, if the UI wasn&#x27;t polished, clients had a hard time wrapping their head around the design or what the final product might look like. In every case I remember we had to make it pixel perfect in the design phase and then again in the development phase.<p>It was a huge pain and an area ripe for innovation but I think the problem like other posters have mentioned is a lack of quality responsive design tools that are better than simple HTML&#x2F;CSS design in the browser.

This is a cool idea. I think this technology could potentially assist front-end developers and the designers who work with them even though there&#x27;s still a fair amount of craft involved. Browser compatibility has gotten better on the desktop in recent years but the mobile appearance of websites is as diverse as ever. It might be interesting to explore &quot;mobile-first&quot; or &quot;progressive enhancement&quot; applications of this technology.

In the short-term, this approach will struggle to compete against WYSIWYG editors. But as soon as they can match them in output, they’ll improve exponentially faster. WYSIWYG editors has a ton of code to maintain, while a model is simple to improve.

It&#x27;s awesome to see how people have picked up and built on top of pix2code (original author here). Very exciting time for front-end development in general!

If the goal is to recreate the <i>layout</i> of a site then the context text isn&#x27;t really important. You could represent each letter with a filler character e.g. . Then, you only need as many tokens as you have words with distinct numbers of characters. This approach (similar to how we use lorem ipsum for prototyping) would dramatically reduce the complexity of the model.

When an AI does create a web page successfully in terms of replacing what a human can do completely, it will probably make the code in such a way that it is unmaintainable because why make it maintainable? We would not have to maintain it anyway. It would be like spaghetti code yet would look amazing on the presentational side. No one would give a damn anymore about coding styles or elegant frameworks like react, you could just bark at it and it would morph around. Sites&#x2F;apps would just be thoughts you could construct on a whim that typically changed frequently or autonomously to produce the maximum desired result. Meanwhile we&#x27;re sucking down nutrient packs floating in stasis in our travel pods spending the majority of our time in virtual consciousness.

<i>&gt;Currently, the largest barrier to automating front-end development is computing power.</i><p>Actually, it&#x27;s poorly though-out web technologies that don&#x27;t allow for reasonable WYSIWYG editing.

Stack Overflow answer iterator + genetic build algorithm = 90% of software development

I have worked mostly on frontend development and have to say that the quality of pixel perfect code being generated will make many a designers happy!<p>Super awesome!

I know alot of you smart ones think this isn&#x27;t that great, but I think this is incredible. I&#x27;d love to see someone host this someplace so I can just upload an image and see what it spits out.<p>I&#x27;m not smart enough to take the code and get it running...

Cool work! I&#x27;m very interested in this topic. Just wondering, how good does it generalize your training data other than just remembering strict input-output mapping?

This is, approximately speaking, useless. Which is not to say it shouldn&#x27;t have been done; you have to make a lot of useless prototypes with any new technology before you can actually make something useful. So long as one considers it in this light, it&#x27;s cool. Just don&#x27;t get too excited about not having to do this work yourself (if you need it done) for at least the next several years (perhaps decades).

This is great example of deep learning that is applied to real world problem. I am just curious if that can be done more easier and more robust by using simple image processing algorithms? Box detection and OCR can work well and may produce better results with different types of mockups. Sometimes I feel like we are making problems even more complicated trying to solve them by using popular concepts.

&quot;Nothing made sense until I understood the input and output data. The input, X, is one screenshot and the previous markup tags. The output, Y, is the next markup tag. When I got this, it became easier to understand everything between them. It also became easier to experiment with different architectures.&quot;<p>Thanks a lot for a great write-up. Truly a good browse for anybody learning Neural Nets.

You didn&#x27;t turn anything into &quot;code&quot;, you converted an image to markup; you&#x27;ve just &quot;described&quot; the image. This doesn&#x27;t work for any sort of web app. A flat image from a designer does not convey enough information to build a proper interface. What of responsiveness? Or hooking it up to an API? Or accessibility? Or animations?

Really cool. I wonder what a differentiable HTML renderer (akin to <a href="https:&#x2F;&#x2F;www.researchgate.net&#x2F;publication&#x2F;270158331_OpenDR_An_Approximate_Differentiable_Renderer" rel="nofollow">https:&#x2F;&#x2F;www.researchgate.net&#x2F;publication&#x2F;270158331_OpenDR_An...</a>) would look like, since it could be used in a similar manner.

I&#x27;m curious about how it handles design changes. Will it just rewrite from scratch? or adjust the code it has already written?

This is great to see!<p>It&#x27;s popular to apply AI to healthcare, transportation and legal work. However, these fields are heavily regulated, require domain expertise and data is hard to get. The nature of front-end dev and other digital skills make them more ripe for automation. I&#x27;m surprised there is so little progress in this area.

I worked on a similar project but was generating graphics code with both java and ruby: <a href="http:&#x2F;&#x2F;www.jtoy.net&#x2F;projects&#x2F;sketchnet&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.jtoy.net&#x2F;projects&#x2F;sketchnet&#x2F;</a>

It&#x27;s just the loud title. Generated templates are consist from ready components in examples. It&#x27;s quite simple to do with minimum knowledge about code and usually useless, because people need more from interface

So... it recognizes text (OCR), detects style (bold, italic, etc.), maybe font face, and outputs the result? No pictures, nothing else, of course.<p>Props for the marketing though.

From reading the comments, it looks like a lot of people are bored of mundane web design. I&#x27;m right there with you. Give me something interesting to do!

If this can get one step further and generate a layout from a hand-drawn sketch, it would really be a gamechanger!

Can you make the page &quot;pop&quot; more?

Would be nice if this was in some adobe product. Then you can go from illustrating straight to web site.

This is phenomenal.

didn&#x27;t Dropbox also recently blog about this?

I love this! Hope no one turns this into some SaaS startup though :(

this would go well with a template language that maps objects to markup rather than having to put template code within the markup.

HTML is not code, its markup.

But, in today&#x27;s workflow, it is trivial for designer to generate the code or even animation with their mockup tools, right? This is useful in a sense if you only have an image of the design, like to quickly copy a competitor&#x27;s work, but not really that groundbreaking for company&#x27;s internal design workflow, the hard part of which is to figure out what is good design, not given a design layout then translate into markup