Wednesday, October 25, 2017

CineForm Goes Open Source

16 years is a long time for a piece of software to remain useful. The CineForm codec, initially developed in 2001, was designed to enable real-time consumer video editing. CineForm, Inc. was a team of engineers that knew image and video processing, but very little about codec design (which likely helped.) Back then, DV (remember mini-DV tape?) was popular but too slow on the consumer's Pentium III and Pentium 4 desktops for software based video editing. We had worked out that the Intel processors had plenty of speed for editing without dedicated hardware, but the camera's compression, DV or MPEG based, were too difficult to decode. So in 2001, the CineForm codec became the first "visually lossless" intermediate codec (not a proxy) that replaced the source with a new file with much more speed. We didn't know this was a somewhat new idea. While there were other performance optimized codecs, like HuffYUV, CineForm was the first to offer significant compression, balancing quality, speed and performance like no other before it. Since then, Avid DNxHD and Apple ProRES have followed similar strategies. As CineForm Inc (the start up), the codec became a core differenciator, open source was never felt to be a viable option (we were probably wrong.)

Consumers weren't the only one's with speed and compression issues. So in 2003, the company pivoted from the consumer market to the professional when we created the CineForm HD codec.  This version shares the same DNA as today's compression, with 10 and 12-bit support and the resolution increase needed for film and television production. Video producers aren't interested in codecs, workflow is what was sold. CineForm compression was bundled with products like Aspect HD, Prospect HD, NEO 4K and Neo 3D, selling a workflow that depended on a codec. We knew there was little value in compression modules alone. Typically codecs sold for only a few dollars a unit, so the workflow obfuscated the codecs value. While the decoder was free, the encoder was not sold separately. The little start-up was not brave enough to make the encoder free, let alone its source.

One reason for keeping it closed, you might be surprised to hear, is that the codec's core tech was very simple. The codec idea was sketched out on a single piece of paper and the performance was determined first by counting the number of Intel instructions needed using an Excel spreadsheet -- even that fit on a single page. The codec was primarily written and maintained by two engineers, Brian Schunck and myself, with some great early help from Yan Ye. The simplicity meant that the magic of the compression was best shrouded in secrecy.

In 2011, CineForm was aquired by GoPro, which it used it to drive its HD and 3D editing utility, GoPro Studio and resulted in producing millions of consumer-created edits. Our initial play for real-time consumer video editing finally happened…10 years later. Now the CineForm codec could be released, and it was licensed to many, including Adobe, for free distribution within Creative Cloud. But it was still not open source.

The CineForm codec has shined most brightly when there is a market change, changes like SD to HD, HD to 2K+ compressed RAW, 2K to 4K, 2D to 3D and HD to 4K. At these times of transition, CPU and GPU vendors didn't yet have hardware optimized solutions -- try software HEVC decoding of 4K60 so see this issue. For the last few years, our computers and mobile devices have had hardware accelerated video decoding for most types of media we are producing. Even GoPro switched from the CineForm powered GoPro Studio to the hardware accelerated Quik for Desktop. So for a couple of years, CineForm didn't get much attention. It was used internally for prototyping, but it received very few public updates.

In 2017, the production market changed again, with new opportunities for the CineForm codec to provide solutions… particularly for greater than 4K sources from premium 360° products, like GoPro's own Fusion and Omni cameras. The new Fusion Studio desktop software defaults to CineForm for all the greater than 4K exports. But unlike GoPro Studio, where CineForm was primarily an internal format, Fusion Studio is not an editor. The CineForm files are meant to be edited in external tools, like Adobe Premiere Pro, After Effects, DaVinci Resolve and HitFilm. While many video tools do have native CineForm support, not all do… and some can’t. This gives GoPro more compelling reasons to make CineForm open source to help support our high resolution cameras.

The CineForm SDK is now open sourced, dual licensed under Apache 2.0 or the MIT license (integrator's choice.) You are now welcome to build new applications around the CineForm SDK, build it into existing applications, and extend its abilities with new performance optimizations and explore new image formats. I would love to see it go into many new places and new markets, but in particular, into popular open source players and transcoders. While the CineForm wavelet engine might be simple, 16 years of coding and optimization have complicated the source. So, to help with new upcoming compression-ists, the open source includes a simplified wavelet compression modeling tool -- simplified CineForm -- for education purposes and some codec design and tuning.

This was another great step for GoPro as it contributes more original projects to the open source community. Learn more about the CineForm SDK open source project on GitHub at https://gopro.github.io/cineform-sdk/

29 comments:

David said...

You can also read my GoPro byline post here: https://gopro.com/news/gopro-open-sources-the-cineform-codec

Chris said...

This is great to see David!

Ariel said...

Thanks a lot for this! I have a few questions (maybe too many, feel free to ignore a few! ):

1. Do new gopro heros output cineform from the camera itself?
2. Is there a plan to support that in the future?
3. Can AutoPano Video pro output cineform in 16 bit bayer raw, log encoded and compressed?
4. Can the gopro vr player play files in cineform in 16 bit bayer raw ?
5. Are you aware of anyone using this SDK to add gstreamer support?
6. Are you aware of video players that run at 120fps from cineform raw (or any of the other formats) on a compatible monitor and nvidia video card running linux?

David said...

Ariel,

1. Hero 5 and 6 Black use VC-5 (basically CineForm RAW) today for RAW Photos -- this is a 12-bit log sensor RAW with 3 to 4:1 compression -- super nice 4-6MBytes RAW photos. However video would be a minimum of 24 times greater at 100MBytes/s or 800Mbs, which is not practical for a microSD based camera. Even if it could handle the data rate, a 64GB card will only hold around 10 minutes. I want this too, it is just not practical.

2. See 1.

3. RAW no, as the sources are not RAW, but deep color yes.

4. Currently there is no way to get CFA Bayer RAW into a 360 file. Bayer RAW implies the lens correction are done after the demoasaic, but stitching can't be done in CFA Bayer space, so inherently all stitched 360 is not RAW.

5. The open source CineForm SDK was only just launched, give a few days before pestering developers to add CineForm support.

6. Sounds doable, but I have limited Linux knownledge.

tobias said...

Do you have a (example) program to decode the GPR files? At the moment there is sadly no program to convert GPR files on Linux. :(

David said...

Tobias,

We hope to soon, either as an independent open source GPR/VC5 decoder, or a small addition to the CineForm SDK. GPR files are DNGs, with a VC-5 compressed RAW essence (which is the same as CineForm RAW with a few header changes.)

Adeel Abbas said...

Great writeup!

Axel Mertes said...

Wow! David - that's amazing news!
Wished that decision would have been there 5 to 10 years ago!
We would have had DCI Cinema using it, or RED cameras, or or or. But it's never to late!

Thanks old mate! :-)

Cheers
Axel

Martin Gartenbau said...

Great news!
Are there any plans to release a standalone cineform windows codec soon from the github project?
At the moment it is hard to find a working solution for Windows7, because GoPro Studio is no longer downloadable from GoPro. And even if you can find an old 2.0.1 to 2.5 version the whole package has to be installed if you need only the codec.
(Btw. I am still using Vegas 12)

David said...

Think Vegas twelve already used the CineForm SDK, it should be okay. For traditional codecs like VFW and Quicktime, all the frameworks have been obsoleted, so there are very few codecs of this type distributed now. It will best for video tool to integrate the CineForm SDK, rather than the binary distribution of old-style codecs. I'm not opposed for a VFW codec binary for tools like VirtualDub and Vegas.

Anonymous said...

Hi David. Wayne here.

Congratulations, great product. I have often had people trying to argue, but my constant recommendation over the years has been cineform and ambarella as leaders in their fields. People just have problems believing things are so great.

I would like to ask, about licencing for camera use?

As for the simplicity of cineform (I haven't examine it yet). I recommend looking for the documents in Adams platform, a maligned super codec. I forget the conclusions (I seem to have some sort of Autism related neurological deterioration, and am not what I used to be) but rememberseeing that it may not have been as bad as people made out. The original handwritten design document mirrors my own efforts from the 80's onwards, and is worth reading. All my work in efficiency through simplicity (after a lot of complex figuring out though). The conceptual approach outlined by him is generations beyond what consumer h265 is currently doing, and is what I was looking at in the Sinclair QL days of the early to mid 1980's (frankly with hardly an idea of how to implement it then). I'm not vouching for their work, but the concept is an interesting place to start something.

Now, vizygig was doing 600mb/s 4kp24 jpeg on an old iphone years ago, and now go much higher. I expect you might find some 1TB cards (especially when moving away from flash) to come which might handle the 4k conform datarate at 6:1+.

We had hoped gopro might do a cineform raw camera in the past. The nvidia credit card area Jetson integra embedded boards are moving to 8k camera inputs with the Xavier series of chips, and frankly a lot of processing power. You could probably run cineform on it at full frame rate. The platform was meant to handle multiple cameras for vision AI on drones. Maybe suitable for a thin personal action, stills, video, drone camera.


An interesting question, that was discussed years ago. Is it possible to get more performance in a simple implementation from cineform by next and previous frame difference (or commonality) inter compression (particularly after multiple frame 3d/4d (which 'd' are we up to now:) ) noise removal and detail restoration)?

Thanks again.

David said...

For 4:2:2 and 4:4:4 in camera encoding we are completely patent free, so hardware vendors are welcome to build CineForm compliant encoding and decoding of all formats except for RAW. The direct using of the CineForm-SDK software, the Apache 2.0 license grants patent rights including RAW for those software products. However for hardware implementations CineForm RAW the licensing would be for SMPTE VC5 which grants RAW patent for custom implementations. Very roughly, the license for VC5/CineForm RAW cameras is $20 per camera in the professional market.

Anonymous said...

Thanks David. BM would be a good company to talk to about that. It is an extremely good figure for other future entrants as well.

Does the 4:4:4/4:2:2 and Raw include all resolutions for video use? Like 4k/uhd and 8k/SHD etc.

Thanks again.

Anonymous said...

Does that mean there can be more updates to the codec for newer OS's? Is there a way to just install it so I can use it with Adobe Media Encoder or just playback files?

I eventually switched to DNxHR because I had stability issues with the codec and my licenses were messed up. Then, on later computers, I had to convert all the CF to DNxHR because I couldn't view it.

I still wish to use CF though. I really need a solid codec with alpha support. I know DNxHR has some implementation but it's difficult to use.

I also miss HD Link - it seemed to convert almost anything to CF!

Maybe I've just misunderstood the way the workflow has needed to change. Anyway, thanks.

David said...

Adobe Media Encoder (all of CC) has natively supported the GoPro CineForm codec for a number of years. The codec in under QuickTime->GoPro CineForm.

SinChris said...

IVe been using Cineform for YEARS back when I could grab the proffesional codec through a university GO Pro Studio Account - I was eiding 1080P footage on my thinkpad like it was nothing.

SO GLAD TO HEAR THIS IS HAPPENING. ProRes always sucked cross platform and Avid is too tight lipped with DNXHbullshit. Ive always archived all of my timelapes generated from high resolution raw stills in cineform and am now glad I did! So much space saved.

Oscar Fernandez said...

This post is so encouraging. Nice work.

Clipping Path Service said...

I m so glad to visit this blog.This blog is really so amazing

Anonymous said...

Wayne again.

Could I suggest that a Blackmagic raw like product might use cineform without need for raw patent.

I have been deliberating that Black Magic Raw may work by encoding the raw values into video in a recoverable way. I've been looking at doing something more advanced for some years. Performing a rough debayer.

Gopro has ASIC design which processing core could be licensed for VC5 to hardware manufacturers, along with a similar Bayer recovery scheme, opening up usefulness for your VC5. Implementation on GPU and on GPU media processing array cores, would enhance acceptability.

BTW Microsd express now supports enough capacity and speed. But as things shrink in depth, something like gopro could use a thicker flash storage pack.

Thank you again David. Wish you well.

Vacuum Cleaner said...

nice blog 👍

ntru said...

Is The Bluoxyn A Recommended Solution?
Yes, of route! The complement like Bluoxyn is a advocated technique that everybody is going to bear in thoughts after they begin understanding it. It is one of the maximum lively and reputed male enhancement nutritional supplements inside the market that no individual should forget about about in any respect due to its excellent modifications that it's going to deliver with its normal dose.
Read More About It >>> https://www.completefoods.co/diy/recipes/bluoxyn-male-enhancement-reviews

brap said...

The People Who Should Not Use Ketogeniks Keto
The makers of this product made positive that the method that they make have to be which encompass truely each person. Meaning that Ketogeniks they superior a product that all the people might be able to use and that they did acquire this set aim. But there live some individuals who were no longer able to healthy in this fold of exclusion. And to be very specific, it's miles essential to inform you that it isn't due to the product but alternatively due to the situations that human beings are in. Following are the examples of folks that can't use Ketogeniks Keto Pills:

Read News >> http://ketogeniksketoweb.mystrikingly.com/

sadishna said...

Debbiesmiracles wanted gain. By exercising on a regular basis you are giving your body the energy boost it needs to ramp up your metabolism rate to start efficiently and effectively losing on a daily basis. I don't about you but that sounds pretty darn good to me. The more you push yourself physically the faster you will see change in your body occur. You may not have to work out with a personal trainer everyday but you do need to do something. Be it g.
https://debbiesmiracles.com/
https://paper.li/e-1581244226#/
https://www.saatchiart.com/art/Video-Debbiesmiracles/1497669/7304751/view

taana said...

Sharktankpedia the body, bringing on fat cells to discharge their put away fat to be copied as vitality. The nourishment you eat amid the day ought to be scorched off through movement. Exercise While You Diet misfortune is about lessening your caloric admission while you increment the calories you smolder. Eating less lessens your caloric admission however practicing helps you blaze more calories. We as a whole realize that to accom.
https://sharktankpedia.org/
https://paper.li/e-1581245426#/
https://www.saatchiart.com/art/Video-Sharktankpedia-Fitness-Blog/1497677/7304815/view

safatini said...


Governmenthorizons ard certified surgeon. Use ASPS (The American Society of Plastic Surgery) to find a board certified surgeon near you. petual and don't react to eating regimen and pharmaceutical, surgery may get to be vital. Today, most specialists choose to perform laparoscopic surgery, since it is negligibly obtrusive and recuperation time is lessened. Blaze On the off chance that you envision yourself shedding pounds and consuming calories amid typical every day action, you will get in shape and muscle to fat quotien.
https://www.governmenthorizons.org/
https://paper.li/e-1581246800#/
https://www.saatchiart.com/art/Video-Governmenthorizons-Fitness-Boys-andGirls/1497691/7304937/view

mideatimers3 said...


Autobodycu and effectively losing on a daily basis. I don't about you but that sounds pretty darn good to me. The more you push yourself physically the faster you will see change in your body occur. You may not have to work out with a personal trainer everyday but you do need to do something. Be it going for a walk, running, swimming, biking, hiking, or any other physical activity you like to do start doing it now. Don't spend another night on the couch watching the latest re-runs. Get your body moving and start losing today. CON.
https://www.autobodycu.org/
https://paper.li/e-1581254466#/
https://www.saatchiart.com/art/Collage-Autobodycu-Best-supplements-all-people/1497795/7305577/view

SrisLawyers said...

Nice blog with valuable information. Thanks for sharing the bog. Keep sharing more valuable blogs to us.. If you know about legal services read our post.
Sex Crimes Lawyer
reckless driving virginia
Conducción imprudente de Virginia

Sruthi Karan said...

Wonderful blog. I happy to read your post and i have bookmarked. Keep doing...
Spousal Support Calculator
Online Solicitation Of A Minor

Dkimlaw said...

Abogado de Accidente De Camión en San Bernardino