TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Is multimedia programming inherently hard?

9 pointsby Brazilian-Sigmaalmost 5 years ago
I spent 100+ hours developing a system that performs fairly simple functionality: capture screen&#x2F;connected webcams on motion detection, add overlays, and save it as .mp4. The system grew up to be a mess, so I rewrote it from scratch again spending 60+ hours, this time with functionality to capture sources in parallel. It uses Windows Media Foundation API, written in 7000 lines of C++ code.<p>The thing is, it doesn&#x27;t work and has nasty bugs on slower systems, it crashes once in a while, sometimes has audio&#x2F;video synchronization issues, etc. - the bugs are mostly very hard to reproduce. Prior to this, I considered myself to be a good programmer but it has almost made me reevaluate my decision whether I should pursue programming anymore (I&#x27;m a CS student), not to mention the toll it took on my self-worth and mental health.<p>The question is, what&#x27;s the reason after spending so much time, the end result is mostly unusable: quirkiness of Windows API, C++, or just my incompetence as a programmer? Or is multimedia programming by nature tricky?<p>P.S. Please ignore my username and new account as I&#x27;m too embarrassed to ask from my real account that gives away my identity.

10 comments

photawealmost 5 years ago
If I understand what you&#x27;re saying correctly, you would need to pretty much be medium-to-intermediate at multi-threaded programming + C++.<p>Capturing screen&#x2F;webcams and adding extra layers can get really complex. If you can&#x27;t simulate and easily test it, it&#x27;s a recipe for disaster.<p>Using MF API is quite old (relies on DirectX 9 as far as I remember).<p>C++&#x2F;COM&#x2F;multi-media is not such a good match - I always felt like puking when having to deal with COM interfaces.<p>I would strongly suggest you switch to C# (note: I&#x27;ve programmed 13 years in C++, and the last 9 in C#). There&#x27;s pitfalls in this too - namely, not sure if&#x2F;how easy you can easily do multimedia without UWP (Universal Windows Platform). That&#x27;s a big can of worms, really hard to grasp, big learning curve, but well worth it in time (if you want to see what you can achieve, you can take a quick look at www.phot-awe.com - that&#x27;s UWP).<p>Having said all the above - I highly suggest C#&#x2F;UWP, but DO NOT expect an easy ride. It will take quite a while to be productive, but that will be time well spent.
thecupisbluealmost 5 years ago
60+ hours is not that much tho.<p>&gt; The thing is, it doesn&#x27;t work<p>What doesn&#x27;t work? Why doesn&#x27;t it work? Is the source of the problems in code or in the libraries?<p>Have you covered all the edge cases and handled all error states?<p>&gt;and has nasty bugs on slower systems<p>Due to concurrency issues or?<p>&gt;it crashes once in a while<p>That happens! Find&#x2F;Write a reporting tool and analyse the traces :)<p>&gt;Sometimes has audio&#x2F;video synchronization issues<p>Smells like concurrency issues to me.<p>Hey,it&#x27;s all good. You wrote something cool that connects a dozen different technologies, did it all in C++ in a short time frame - and you&#x27;re just a student! That&#x27;s great!<p>You&#x27;re not a bad programmer - maybe you&#x27;re just bad at system design&#x2F;software architecture. And that&#x27;s not bad - that&#x27;s great. A lot of folks go into the field and work for a few years until they start learning about architecture and experience what problems it can cause. You just had that experience, now go read up and figure out how to rewrite it into V3. Take some extra time to think about the models, the abstractions, the components, go through edge cases in your head, figuring out if and how your system acccounts for them. Try not to overengineer it - you probably will - but don&#x27;t think that means you will suck forever, you will learn with time and experience.<p>Imagine if Picasso quit after his first sucky drawing. Maybe even HN wouldn&#x27;t be here.
评论 #23458743 未加载
psyklicalmost 5 years ago
You show admirable qualities that good programmers have. For example -- boldness to take on a challenging project, tenacity to pursue a rewrite, and drive to pursue improvements such as parallelism. After all, if you only pursue easy projects then you are unlikely to improve your skills!<p>There are definitely challenges involved with multimedia apps. They demand good performance, and they require you to rely on third-party libraries and drivers.<p>I assume you are decent at debugging to already have gotten as far as you have. So, I would continue to take guesses at what the problem might be then test whether you are correct. Based on your description, perhaps the CPU cannot always keep up with the framerate. You could test this (for example) by seeing if rendering a lower-resolution output would alleviate some of the bugs.<p>If none of your guesses seem to work, pare your code down to make it as simple as possible (e.g. a basic webcam app without overlays). Once you get it reliably working, then slowly re-add your features. Worst case, you can always post-process the video after streaming it to disk, or have a tool such as FFmpeg add the overlay.<p>Good luck!
评论 #23458568 未加载
lucozadealmost 5 years ago
Multimedia is tricky but, from the way you describe it, you&#x27;ve come across the more general problem that writing concurrent software in general, and parallel software in particular, can be a real sod.<p>What you&#x27;re likely to be experiencing are bugs due to race conditions. These are notoriously hard to debug.<p>It&#x27;s one of the reasons why there&#x27;s so much fuss around languages like Rust and Go. They have builtin features that try to make these types of issue less prevalent. I appreciate this doesn&#x27;t help you directly but should make you feel a bit better; there are whole engineering teams working to avoid the types of issues that you&#x27;re experiencing.<p>In terms of how to help you move forward, that&#x27;s quite hard in this format. Sorry.
评论 #23458772 未加载
saltcuredalmost 5 years ago
It sounds like you are talking about real-time data capture, analysis, visualization, encoding, and archiving. If so, what you describe is not just multimedia. It is a real-world hybrid of multiple specialities, and definitely becomes more difficult because you have to address all these aspects at once to meet your original goal.<p>If you had started with pre-recorded videos, you could imagine having pursued the challenges incrementally. Process a video and output some frame-by-frame analysis metadata. Process the analysis results and visualize them in the context of the source video imagery. Re-encode the results to a good storage format. Explore options for quality, correctness, performance, and resource requirements.<p>But, if you need to do this on live streams and run for indefinite periods of time, you suddenly need to be much more aware of computational delays, working memory sizes, buffering of input and output data, etc. You need it to work consistently and predictably while handling the full workload and any potential variability in other elements of the system. How much can the incoming video stream vary in bandwidth? How much can decoding, analysis, visualization, and encoding speeds vary with changes in the video content? How reliably can the input and output IO paths support the bandwidth?<p>And, as you mentioned, it becomes challenging to then squeeze this into a slower system than it was originally designed and tested on. Ultimately, you have to design for failure: think about how you want it to perform if it cannot satisfy the full objective in the time and resources available. Drop frames? Degrade stream quality? Drop entire low-priority streams...?
kleer001almost 5 years ago
Yes, it&#x27;s hard. Plugging anything into the real world at real time is hard. That is, constraining data to certain bandwidths to play or compress. The real world is noisy at every level and that&#x27;s difficult to program around.
cyberdrunkalmost 5 years ago
Your intuition is correct, programs which have to do complex processing on lots of data (e.g. a video feed) in real-time are just inherently very hard.
s1t5almost 5 years ago
Not an answer to your question but I&#x27;m struggling with getting anything done at all and I&#x27;m impressed by the amount of work that you&#x27;ve put into this.
brudgersalmost 5 years ago
You are not your code.
lodialmost 5 years ago
Don&#x27;t beat yourself up about it. My impression is that you&#x27;re probably an above-average coder.<p>First of all, doubting your own competence is a <i>good</i> sign. Especially combined with a willingness to ask questions and act on feedback. I&#x27;d be extremely skeptical hiring someone who had no self-doubt.<p>It sounds like you just tackled too many novel things at the same time:<p>- 160h, 7000 line project is a very large project for a student. Managing software complexity is its own skill.<p>- Multithreaded programming is hard.<p>- Performant, low-latency programming is hard.<p>- &quot;Systems&quot; programming is hard.<p>- Video capture, codecs, etc. are a narrow niche. Microsoft COM API&#x27;s are not exactly the simplest things...<p>- C++ is a particularly unforgiving language. It doesn&#x27;t help that the language is so versatile that 90&#x27;s C++ libraries&#x2F;tutorials&#x2F;etc look nothing like &quot;modern&quot; C++ from the 00&#x27;s, which was itself obsoleted by more &quot;modern&quot; C++ from the 10&#x27;s...<p>In terms of advice, I get the impression that you&#x27;re not looking forward to writing a third version of this project, so here are a few generic tips for whatever you choose to do next:<p>- It sounds like you wrote the code that does whatever the application is supposed to do, but didn&#x27;t write the supporting tests&#x2F;tooling&#x2F;etc. For example, elsewhere you mention that you think the application might be dropping frames due to this or that, but you&#x27;re not sure. The next step--and honestly you need to do this preemptively even before there are any problems--is to add very detailed logging to provide you with hard data. Furthermore, in an audiovisual application like this, you&#x27;ll probably have to write specialized tools to parse and analyze those logs graphically since they&#x27;ll be dumping a tremendous amount of data to disk. Don&#x27;t be afraid to write more test and tool code than main application code.<p>- Sometimes it&#x27;s worth it to take some extra time to perform &quot;studies&quot;--code that you write to experiment with something, then throw away. For example, elsewhere in this thread you asked if it would be better to buffer 10 frames at a time or to process frames synchronously. Well, this is a great opportunity to do some &quot;science&quot;: think about what you expect will happen, fork the code to a new branch in source control, implement the change in quick-and-dirty way, and check the results. If it confirmed your hypothesis and helped, switch back to your original branch and re-implement it in a clean way. If it confirmed your prediction and didn&#x27;t help, switch back to your original branch and just make a note of this experiment in the documentation. Negative results like this are important since they can reduce the surface area of things you need to troubleshoot later. If the result was surprising in some sense, investigate further and if post to IRC or StackOverflow if you get stuck.<p>- Regarding &quot;solving hard problems&quot;. Forget about how hard or easy something is in an absolute sense. I wouldn&#x27;t even worry too much about comparing yourself with peers. Just worry about self improvement. And the best way to improve yourself is to introduce a little bit of novelty at a time. For example, if you&#x27;re learning a new language, re-implement some toy application you&#x27;ve already written before in C++. Don&#x27;t try to learn a new language while also introducing known-hard things like networking, concurrency, etc.