I've been going through Cem Yuksel's "Introduction to Computer Graphics" course and thought that writing a volume renderer would be a good way to test my knowledge. It is a common technique used to render 3D medical data. Works by ray marching a specific step size, reading a 3D texture (e.g. MRI data), and calculating opacity values.<p>Code should be easy to get started with for anyone familiar with the JS ecosystem.<p>Questions for the HN community: I spent 20-25% of the entire time just setting up the project and fighting issues with the JavaScript ecosystem. This experience has made me consider learning Cpp, Metal, and XCode. Has anyone made the transition from WebGL/TS to Cpp or done it the other way around? What was your experience with it? And what about debugging? That's a big issue with WebGL.<p>As for now, I'm thinking about picking up WebGPU next because it has an up-and-coming debugger made by Brendan Duncan[0] that looks promising.<p>Edit: Anyone know why MRI data is sent via CDs and not the web? I started working on this project specifically because some people close to me got MRI scans and received CDs of their results. I know that some MRI data can be huge so downloading over the web doesn't make sense, but anything under 64MB should be sent over the web, no? For reference, I believe most MRI data can be under 250MB[1].<p>[0] <a href="https://github.com/brendan-duncan/webgpu_inspector">https://github.com/brendan-duncan/webgpu_inspector</a><p>[1] <a href="https://www.researchgate.net/figure/Typical-image-dimensions-and-uncompressed-file-sizes-for-common-medical-imaging_tbl1_320511493" rel="nofollow">https://www.researchgate.net/figure/Typical-image-dimensions...</a>
There are many more desktop tools for debugging GPU code, largely because of game development. Xcode as an IDE is pretty out of date. Visual studio has much better debugging tools for GPUs. See nvidia's tools. As a long time GPU programmer, I'm particularly interested in testing out rust.<p>The medical industry moves slowly. Medical data is often covered by HIPAA, which is why it is often not available easily on the web. Using CDs is just an artifact of their slow movement through technology.
A cool property of volume rendering is that it requires much simpler calculations than other 3D rendering problems. It's simple enough that you can use Homomorphic Encryption to calculate the rendering of _encrypted_ medical data!<p>Here is my summary of it[1] and the original paper[2] I found this in<p>[1] <a href="https://azeemba.com/posts/homomorphic-encryption-with-images.html#image-analysis" rel="nofollow">https://azeemba.com/posts/homomorphic-encryption-with-images...</a><p>[2] <a href="https://arxiv.org/abs/2009.02122" rel="nofollow">https://arxiv.org/abs/2009.02122</a>
You don't get the "run everywhere" ability that threejs has. It appears you "fighting the js ecosystem" comes from lack of experience and familiarity<p>I recently tried Metal a bit. Some features are only available for visionOS (the headset) and not iOS or macOS and vice versa.
Does it support DICOM as input files?<p>I was provided MRI DICOM files a month ago via the radiology office's used SaaS platform, Sectra PatientPortal.<p>After the MRI session I was given a login code. Back home, I could use that code on the Sectra PatientPortal.<p>It provides a very basic DICOM image web viewer (contrast/dynamic range modification and slice scrolling, four windows), albeit 2D only, and download of the DICOM files as well as PNG exports.<p>I am looking for a web-based or Linux-based FLOSS DICOM viewer, but couldn't find any professional program so far.
Brings back fun memories, wrote something similar in college with qt and opengl: <a href="https://github.com/fargiolas/qvrc">https://github.com/fargiolas/qvrc</a>. It had a super buggy but somewhat working live transfer function editor, blinn phong and toon shading, arcball camera. Really nice project for GLSL learning.
Oh man, this is awesome. I have been periodically checking the internet for something like this that I can hand off to an intern and get some visualization of data at work. I am definitely gonna be using this.<p>Also I love your videos. The internet is a smaller place than it seems.
Three.js and React, yet not R3F - curious why?<p>I really like how you organized your project. As someone who's developed with Three.js for years both imperatively and in a highly-declarative way using R3F I'm always interested to see different approaches to architecture/organization.
This is cool - for Metal, the debugging tools are pretty useful in Xcode - you can access buffers and read values directly.<p>Related - a while I made a process to convert DICOM / MRI files to voxels for printing at the native resolution of 3D printers [0]. It means you can preserve all the fine detail of the scans, rather than converting to a mesh (segmentation).<p>On the CD question - it's probably because there is little incentive to build a secure / cross platform solution for patients to access their scans. The CD model is very outdated, but does work, and there is no need for HIPAA compliance even though a CD isn't very secure.<p>[0] <a href="https://www.lukehale.com/voxelprinting" rel="nofollow">https://www.lukehale.com/voxelprinting</a>
Most volume renderers lack a good transfer function editor.
When analyzing volumes, especially explorative analysis, the most effective tool is to dial in colors and opacities for certain value ranges, in order to find structures.<p>The volume rendering engine I have been working on uses a histogram for the value distribution, and on top of it, one can draw lines that indicate the opacity. Additionally, one can set colors to the control points, which are then linearly interpolated for the given ranges.
I’ve received MRI data by both CD and online. I think it just comes down to what systems the hospital/consultant supports and how recently their systems were set up of modernised.
This looks great - will try it out tonight.<p>At first I thought you‘ve been using ImGUI as your interface looks similar.<p>Maybe that‘s a good starting point for your CPP career.
I would recommend you to upload everything using Netlify since it's free and you just have to click 2 buttons.
In this way, you may have a demo online, which is the overwhelming top advantage of the web compared to anything else around programming.
Dan Kaminsky once showed me a volume renderer he was noodling with, but instead of MRI data, he had loaded frames from video footage of an explosion. I remember it having a 4D feel to it.
Check out vtk.js (and VTK proper) as a fairly mature implementation of this. I haven’t kept up with the development but it was pretty far ahead when it came out years ago.
Author of the WebGL volume rendering tutorial [0] you mentioned in the readme here, great work!<p>Working in WebGL/JS is nice since you can deploy it everywhere, but it can be really hard for graphics programming as you've found because there are very few tools for doing real GPU/graphics debugging for WebGL. The only one I know of is [1], and I've had limited success with it.<p>WebGPU is a great next step, it provides a modern GPU API (so if you want to learn Metal, DX12, Vulkan, they're more familiar), and modern GPU functionality like storage buffers and compute shaders, not to mention lower overhead and better performance. The WebGPU inspector [2] also looks to provide a GPU profiler/debugger for web that aims to be on par with native options. I just tried it out on a small project I have and it looks really useful. Another benefit of WebGPU is that it maps more clearly to Metal/DX12/Vulkan, so you can use native tools to profile it through Chrome [3].<p>I think it would be worth learning C++ and a native graphics API, you'll get access to the much more powerful graphics debugging & profiling features provided by native tools (PIX, RenderDoc, Nvidia Nsight, Xcode, etc.) and functionality beyond what even WebGPU exposes.<p>Personally, I have come "full circle": I started with C++ and OpenGL, then DX12/Vulkan/Metal, then started doing more WebGL/WebGPU and JS/TS to "run everywhere", and now I'm back writing C++ but using WebGL/WebGPU and compiling to WebAssembly to still run everywhere (and native for tools).<p>With WebGPU, you could program in C++ (or Rust) and compile to both native (for access to debuggers and tools), and Wasm (for wide deployment on the web). This is one of the aspects of WebGPU that is most exciting to me. There's a great tutorial on developing WebGPU w/ C++ [4], and a one on using it from JS/TS [5].<p>[0] <a href="https://www.willusher.io/webgl/2019/01/13/volume-rendering-with-webgl" rel="nofollow">https://www.willusher.io/webgl/2019/01/13/volume-rendering-w...</a><p>[1] <a href="https://spector.babylonjs.com/" rel="nofollow">https://spector.babylonjs.com/</a><p>[2] <a href="https://github.com/brendan-duncan/webgpu_inspector">https://github.com/brendan-duncan/webgpu_inspector</a><p>[3] <a href="https://toji.dev/webgpu-profiling/pix" rel="nofollow">https://toji.dev/webgpu-profiling/pix</a><p>[4] <a href="https://eliemichel.github.io/LearnWebGPU/" rel="nofollow">https://eliemichel.github.io/LearnWebGPU/</a><p>[5] <a href="https://webgpufundamentals.org/" rel="nofollow">https://webgpufundamentals.org/</a>