Mike Innes is one of the real rockstars in the Julia community. He has a knack for making surprisingly small and elegant packages which compose so naturally with the base language that they feel built-in.<p>After a `using Flux`, julia is suddenly a machine learning language, instead of a language with a machine learning library. I'd argue it shouldn't be surprising that he found his way to Julia because Julia is one of few languages that allows one to make such packages.<p>His other packages such as MacroTools.jl, Lazy.jl and Zygote.jl are also well worth checking out.
As an old Lisp user, I am impressed by Flux (which I started using this weekend, after someone on HN recommended Flux to me) as transforming Julia, like building Lisp up to a new language for whatever problem you are working. I also appreciate how incredibly easy it was to get started with Flux which ‘just worked’ with CUDA 10 and the GPU in my laptop and the model zoo was great to get started. Really quality stuff!
Since the blog post does not have many code samples, this non-trivial AD example with Zygote.jl is worth sharing (it's from their readme):<p><pre><code> julia> using Zygote
julia> fs = Dict("sin" => sin, "cos" => cos);
julia> derivative(x -> fs[readline()](x), 1.0)
cos
-0.8414709848078965
julia> -sin(1.0) # 'true' derivative in 1.0
-0.8414709848078965
</code></pre>
So Zygote can apply AD to an anonymous function that looks up a function in a hash table from user input.<p><a href="https://github.com/FluxML/Zygote.jl" rel="nofollow">https://github.com/FluxML/Zygote.jl</a>
As someone who works in merging differential equations and machine learning, I have found this kind of work essential for what I do. Pervasive AD that allows merging neural networks and diffeq solvers is allowing us to explore all of kinds of new models and new problems. Sure it doesn't impact vanilla machine learning all that much (though Zygote.jl does allow for a lot of optimizations that wouldn't be possible with tracing-based AD), but it definitely opens up a new wave of AI possibilities.
As an XLA:GPU person I'm curious how the performance of Julia natively compiling to CUDA compares to using XLA:GPU.<p>In particular, is this a promising approach, or do you see it as a dead end compared to generating GPU code natively? If it's promising, are there things we need to do in XLA:GPU to make it less awful for you?<p>(Reasons you might want to use XLA:GPU include, you don't have to reinvent all our performance and correctness hacks for cudnn, and maybe our kernels run faster since we're targeting such a limited domain?)
I'm tasked with running several ML algorithms on a new hardware accelerator. Currently there is an LLVM toolchain for that new hardware, but no Python support is expected for a while which means implementing a bunch of ML code in C or maybe C++ (not a very pleasant prospect). I'm wondering, since Julia has an LLVM backend would it be possible to emit LLVM IR from Julia which could then be fed into our LLVM toolchain?<p>One thing that comes to mind here: does Julia use some kind of primitives for various things like matrix multiplication that might be difficult to export at the LLVM-IR level?
I'd be interested in a direct comparison with similar efforts undertaken by existing frameworks; for example Torch Script[1], which aims to produce a language which shares a syntactic frontend with Python while getting all the goodies that ahead-of-time compilation gives you (symbolic diff, operator fusion, etc).<p>Seems to me that the primary challenge for any "next-generation" framework or language is getting people to actually use the thing. Sharing a front-end with Python and a backend with PyTorch seems like a good way to bootstrap that.<p>[1] <a href="https://pytorch.org/docs/master/jit.html?highlight=torchscript" rel="nofollow">https://pytorch.org/docs/master/jit.html?highlight=torchscri...</a>
I wonder if this feature is in any way different from LINQ for expressions?<p>In C# you can say<p><pre><code> Expr<Func<float,float>> sin = Math.Sin;
</code></pre>
And then write a function<p><pre><code> Expr Derivative(Expr expr) => ...
</code></pre>
Which will take the above sin, and compute its derivative as another Expr, which can be later compiled using Expr.Compile()<p>In C# this has been introduced to make SQL bindings.<p>So far, the only difference I see is that in C# there's a distinction between expression trees and functions themselves, but in Julia there's not.
> Meanwhile, the idea of ML models fundamentally being differentiable algorithms – often called differentiable programming – has caught on.<p>> We need a language to write differentiable algorithms, and Flux takes Julia to be this language.<p>Recently on HN there was some discussion of this paper by Conal Elliott on automatic differentiation in a pure functional language (Haskell): <a href="https://arxiv.org/abs/1804.00746" rel="nofollow">https://arxiv.org/abs/1804.00746</a><p>This is a rather large and vague question but I'm curious whether people have comments on the relative merits of Julia vs a pure functional language for supporting "differentiable programming" for ML?
Also posted here: <a href="https://news.ycombinator.com/item?id=18593453" rel="nofollow">https://news.ycombinator.com/item?id=18593453</a>. Maybe a mod could combine the two?
This looks awesome, and makes me wonder if anyone done any real-time graphics experiments with Julia? With this great AD and gpu support, I would love to try using this on some graphics applications!
This is gonna be very interesting when this can be combined with a distributed network of specialized CNNs for highly specialized tasks (if there isn't already)