Wednesday, October 10, 2012

Protune

If you haven’t heard already, Protune™ firmware and GoPro App for the WiFi BacPac are now available -- get them now.  While my team worked substantially more on developing the GoPro App, this blog is about the origins and design of Protune.

Before I geek out on why and how Protune is so cool, some readers may want to know what it is and does (and may want to skip the rest.)  Protune is a suite of features designed to enhance an even more professional image capture from your GoPro, while still being accessible to every GoPro user.  Protune has the strongest emphasis on image quality by increasing the data-rate (decreasing compression) from an average of 15Mb/s to 35Mb/s.  Small artifacts that can occur in detailed scenes or extreme motion are gone at 35Mb/s.  Next is adding the 24p frame rate (to the existing frame rate options), greatly easing the combination of GoPro footage with other 24p cameras, common to professional markets. Finally the Protune image is designed for color correction; it will start with a flatter look that is more flexible for creative enhancement of the image in post-production. With the latest HERO2 firmware installed, Protune is enabled with the secondary tools menu.

Now for the why and how.

Protune has been a long time coming, and so has this blog entry.  Protune is an acknowledgement that so many GoPro cameras are used for professional content creation – Discovery Channel looks so much like a GoPro channel to me.  Protune is also the first clear influence the CineForm group has had on in camera features, for which we are super proud, yet most of the engineering was done by the super smart camera imaging team at GoPro HQ.  For the novice Protune user, CineForm Studio 1.3 is setup to handle Protune image development, so all users can benefit from this cool new shooting mode. This synergy between the software and camera groups, allows us to push both further. In the old CineForm days (non-GoPro) I would have probably blogged about helping with the design of a new camera log curve, and all the pluses and minus of color tuning, months before we would have had anything to show, but that was before we became part of a consumer electronics company.  Some things must remain secret. Working at CineForm was exciting, but it is nothing compared to the adventures I’ve already had at GoPro, with so much more to come. 

Protune for me started when HERO2 launched.  Here was a camera that I could use in so many ways, yet in certain higher dynamic range scenarios (I shoot a lot of live theatre and was experimenting with placing GoPros around the stage), the naturally punchy image limited the amount of footage I could intercut with other cameras.  It is of course the intercutting of multiple camera types that is of greatest need for the professional user. Note: there is one professional group I know of that exclusively uses GoPro HEROs, and that is our own media team – even though they now use Protune shooting modes. Protune gets you more dynamic range, and I was amazed how much. 

Sensor technology continues to grow, and we are seeing awesome wide dynamic range images coming from premium cameras like ARRI Alexa and even the amazingly affordable Blackmagic Cinema Camera, but as sensor size (really pixel size) shrinks, there is an impact on dynamic range.  Smaller pixels often result in reduced dynamic range, yet so much has changed in so few years.  Back in 2006, CineForm was very much involved with Silicon Imaging and the development of the SI-2K camera, which was highly praised and generally confirmed to have around 11 stops of dynamic range – good enough to be used on the first digitally acquired feature (well, mostly digital) to win Oscar Cinematography and Best Picture awards.  The HERO2 sensor is smaller and has significantly higher pixel count (11MPixel versus SI-2K’s 2MPixel, HERO2 pixels are way smaller), yet we are also seeing a similar dynamic range.  

It was not just five years of sensor technology that made all the difference, it was using a log curve instead of contrast added to Rec709 with 2.2 gamma -- geek speak for calibrating cameras to make the default image look good on your TV.  Making images look great out of the box is the right thing to do for all consumer cameras, and you get just that with HERO2 via HDMI to your TV. Yet TVs do not generally have 11 stops of dynamic range, maybe 9 on a good set, and that is after you’ve disabled all the crazy image “enhancements” TV defaults to having switched on (which typically reduce dynamic range further.)

So why shoot wider dynamic range for something that may only be seen on TV, computer monitor or smart phone (all decreasing in dynamic range)?  The question is somewhat obvious to professional users, as color correction is part of the workflow.  Color correction simply works better with more information from the source for which to choose the output range. Even the average consumer today is more open to color correction of an image thanks to the likes of Instagram filters. The more dynamic range you start with, the better such stylized looks can work.  Our own media team wasn’t using great tools like Red Giant’s Magic Bullet Looks until shooting Protune, which greatly increased the creative flexibility of the GoPro image output.

So why a log curve, rather than just reduced contrast with the regular gamma?  This is a trickier question.  The full dynamic range can be presented with a 2.2 gamma of standard TV, it will look a little bland (flatter or milkier) just as log curves do on a TV without color correction, so it holds no aesthetic advantage over log.  Log curves do have an advantage over gamma curves when your goal is to preserve as much of the source dynamic range for later color correction.

Some imaging basics:  Light hitting the sensor and the sensor’s response to that light, is effectively linear (not the incorrect use of linear to describe video gamma that still seems to be popular.) Linear has the property that as light doubles (increasing one stop), its sensor value doubles.  With an ideal 12-bit sensor, ignoring noise, there are 4096 values of linear light.  After the first detectable level of light brings our ideal sensor from 0 to 1, a doubling of light goes from 1 to 2, and the next stop from 2 to 4, and so on to produce this series 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048 and 4095 of doubling brightness (to the point where the sensor clips.) An ideal 12-bit sensor has a theoretical maximum of 12-stops of dynamic range.  If we were storing this 12-bit data as uncompressed, this is the most flexible data set (for color correction), yet this would be over 1000Mbits/s compared with today’s standard mode 1080p30 mode on HERO2 at 15Mb/s – think how fast your SD card would fill, if it could actually support that fire hose of data. Fortunately it turns out that linear is a very inefficient way of presenting light when humans are involved, as we see brightness changes logarithmically--a stop change is the same level of brightness change to us, whether it is from linear levels 1 to 2 or from 1024 to 2048.  As a result, most cameras map their sensor’s 12, 14, 16-bit linear image, to an 8, 10 or 12-bit output with a log or gamma curve, exploiting that we humans will not notice.  Even the uncompressed mode of the new Blackmagic camera maps its 16-bit linear output and only stores 12-bit with a curve – this is not lossless, but you will not miss it either. Lossless versus lossy is an argument you might have heard me present before, to the same conclusions.

If we remained in linear, converting to 8-bit from 12-bit would truncate the bottom 4 stops of shadows detail, we will notice that.  So a conventional 2.2 gamma curve does the following with its mapping (top 5 stops shown.)
12-bit Linear input
8-bit Gamma 2.2 output
Codes per stop
256
73
19
512
100
27
1024
137
37
2048
187
50
4095
255
88

So gamma curves don’t fully embrace a human visual model, with many more codes used in the brightest stop as compared with the darker stops.  The perfect scenario might be to have the 256 codes divided amongst the usable stops, e.g. 11 stops would be around 23 codes per stop.  Remember, this is for an ideal sensor (i.e. noise free) and this is not going to happen.  The darkest usable stop is mostly noise, whereas the brightest stop is mostly signal, we need a curve to handle the allocation of our code words with this in mind.  

The top 5-stops of the Protune log curve:
12-bit Linear input (idealized)
8-bit Protune output
Codes per stop
256
112
33
512
146
34
1024
181
35
2048
218
37
4095
255
37

While the darkest useable stop have a similar number of code words as the gamma curve, Protune distributes the codes are more evenly over the remaining stops, more code-words are reserved for shadow and mid-tone information. 

While I glossed over this before, again why not just have 23 code words per stop?  This has to do with compression and noise.  Noise is not compressible, at least without it looking substantially different than this input, and the compressor, H.264, CineForm or any other codec, can’t know signal from noise.  So if too many code words represent noise, quality or data-rate has to give.  The Protune curve shown above will produce smaller files, and generally be more color correctable than using fixed code words per stop. We have determined the best curve to preserve dynamic range without wasting too much data to preserve noise.

Side note for other RAW cameras: We have extended our knowledge gained while developing to the Protune curve to calculating of the best log curve for a particular dynamic range. This feature has now been included in the commercial version of CineForm Studio (Windows versions of Premium and Professional), so that the RAW camera shooter, such as from Canon CR2 time-lapse videography, to Blackmagic CinemaDNG files, can optimize the log encoding of their footage.  Of course transcoding to CineForm RAW at 12-bit rather than 8-bit H.264 helps greatly, yet the same evening out of the code-words per stop to applies as it does in the HERO2 camera running Protune.

Protune couldn’t exist as just a log curve applied upon the existing HERO2 image processing pipeline, we had to increase the bit-rate so that all the details of the wider dynamic range image could be preserved. But we didn’t stop there.  As we tuned the bit-rate, we also tweaked the noise reduction and sharpening, turning both down so that much more natural detail is preserved before compression is applied (at a higher data rate required to support more detail.) Automatically determining what is detail and what is noise, is a very difficult problem, so delaying more of these decisions into post allows the user to select the level noise reduction and sharpening appropriate to their production.  I personally do not apply post noise reduction, happy working Protune as it comes from the camera, adding sharpening to taste.  

The CineForm connection:  35Mb/s H.264 H264 is hard to decode, much harder than 15Mb/s. So transcoding to a faster editing format certainly helps, and that comes for free with GoPro CineForm Studio software.  Also, the new Protune GoPro clips carry metadata that CineForm Studio detects and automatically develops to look more like a stock GoPro mode, cool-looking and ready for show.  All these changes are stored as CineForm Activate Metadata, are non-destructive and reversible, all controlled with the free CineForm Studio software.  GoPro is working to get professional features in the hands of the everyday shooter, and the CineForm codec and software is an increasing part of that solution.  

There is so much to this story, but I’m sure I’ve gone on too long already. Thank you for reading.

P.S. Sorry for the lack of sample images, Protune launched while I'm on vacation, my internal connection is way limited at the moment. 

---

Added sample images in the next blog  Why I Shoot Protune -- Always!

51 comments:

Vlad said...

Great article, thanks for the useful explanations !
I still have a question about noise reduction :
Is it possible to automatically apply afterward in post production the same level of noise reduction/sharpening as we can see in standard 15Mbit mode when shooting in Protune mode ? Using Cineform Studio ?

David said...

CineForm Studio does automatically apply a camera equivalent sharpness as Active Metadata. This is destructively controllable and real-time. Unfortunately it is very hard to do the equivalent with denoise (at least a quality version.)

David said...

So denoise is a non-real time process for a user to select. We have used and very much like red giant's Denoiser-II

Unknown said...

Have you looked at Rec 2020 specified for UHDTV? Both 10-bit and 12-bit, linear and log are specified. We think more interesting dynamic ranges are definitely on the way for consumer displays.

David said...

That is cool news. More reasons to shoot log now.

nikol robert said...
This comment has been removed by a blog administrator.
atom said...

ProTune is sweet. Too bad the v198 firmware has so many bugs & missing features :(

3D support :(
1-button mode :(
720p50 :(
"zoom" settings :(

If the camera isn't turned on for 5-10 seconds before I start recording, it'll just record a few seconds and quit.

Anonymous said...

I'll second @atom on some of this - specifically the recording cutoff bug that puts the guesswork back into recording. The long delay (5-10sec) from power-on to record-start is also a hard pill to swallow, rendering 1-button mode fairly useless.

720p30 would be a great space-saver (and maybe battery saver) too, especially for Protune modes...not sure why that was left out.

Otherwise, the quality is superb!

--
leeoniya

Juhani said...

Very good article, and interesting. Lets see what the future brings next :)

Anonymous said...

Hopefully can get my bricked HD2 and another which just refused to update (stuck in v70) to update so I can try out protune.

Alvaro Cova said...

hey dont update, this is not working! my camera is not funcional after the update, im so mad right now!

Shane said...

I am really excited about the new ProTune features and the WiFi Backpack. I bought one the second the app was available.

Unfortunately I have the same experience that many others do - the update turns my camera into a brick. I have to roll back to v70 for the camera to function at all.

I love my Hero 2, but the WiFi backpack is an expensive paperweight and the ProTune features are useless to me because the update is buggy.

When can we expect a functional update?

David said...

Sorry to hear some users are having update issues. I expect this is way less common than any negative news seems on the Internet. If you are having update issues support will help you.

I have shoot many hours of Protune 24p in 3D so I find that bug report odd -- 3D should be fine (even better in Protune.)

The narrow FOV was removed by design, I fought to keep medium FOV. Each new mode adds to the testing time, and Protune added a lot to test, limiting modes was necessary to get the feature launched. 720p50 was cut for the same reasons. Note: Protune does support 1080p25.

Other reported bugs the camera team will examine.

I'm just returning from vacation, I shot many hours of Protune and I will post some examples soon.

Shane said...

It might be less common than it seems, but literally hundreds of users have posted on various forums (and Amazon) that have an issue with the update and support has not been responsive. There is nothing on the GoPro support site that is responsive to the issue.

I'm a software engineer myself, and I understand the stages of software production. I know how big a PITA it can be...but, dude, you've got a serous problem in your code somewhere.

Quintessential Studios said...

Agree with Shane on the first paragraph. Not so sure it's David's code that's the issue, though.

Question as to 24p, why is it not 23.98? As the 30p is all 29.97?

(See twitter question from Editblog on this for more info)

Anonymous said...

Im not having big problems with my GoPro Hero 2 after formatting the SD card... however CineForm will not open files on my XP PC or export on my Windows 7 PC

David said...

I have no idea why the switch from 23.976 to 24.0. Most of the betas were 23.976. I filed the bug. CineForm Studio will transcode 24p to 23.976 which audio correction.

David said...

For CineForm Studio help , email support@cineform.com. All XP, Vista, 7 and Windows 8 systems should be about to run Studio. Make sure you have all system updates installed.

Makofoto said...

Learned a LOT from David's article and everyones comments. I wish I hadn't updated all four of my GoPro2's ... so that I could do a before and after test. To me it seems that the NON ProTune files have worse color balance and more saturation then the previous firmware?

Even before the upgrade I was having a problem with One Touch, which I usually don't use. I lost a lot of takes because the camera was shutting off after about 20+ seconds?!

Some of my initial ProTune tests: http://public.fotki.com/makofoto/work/gopro2-protune-test/

Makofoto said...

I was hoping to use a 3D housing/cable to do a direct comparison between ProTune On and Off, but I'm guessing the sync cable didn't like the fact that ProTune was off on one camera? OR, do I have a bad cable? I had both cameras set to 1080/30 ... and all other settings matched, except ProTune Off/On

Martin said...

ProTune is really cool but misses two settings :

- white balance
- sharpness

Add these two one and it will become the killing firmware all of us are waiting for !

Brian said...

I also have the issue that at the same video rate, the Hero2 stops recording after 20-30 seconds. Likewise, the 5MP 2Hz timelapse option cuts out after several tens of shots.

I just found out that this was resolved (maybe) by changing the class4 SD card for a class10 SD card.

Though now it seems that the battery charging indicator is no longer lit when charging. I am having so many issues with that battery....

David said...

Brian, Yes Class-10 SD media is required for Protune, although 2fps 5MP still should have no issue using Class-4.

David said...

Martin,

Protune is a optimized mode for professional and hobbyists alike. Have separate controls over all features would be a deterrent for many users that we hope will embrace Protune. Sharpness in Protune is turned down as it is easy to add back, it is much harder to remove sharpening artifacts once they are baked at the camera. If you use CineForm Studio, sharpness is very easy to control.

Jessica said...

The narrow, 90deg FOV is gone with ProTune?

The one thing I can't stand about GoPro is the lack of narrow FOV in all modes :(

Adam Miller said...

Thanks so much for this post David! It's great to hear the nitty gritty about Protune and i'm really excited to have it in my hands! My first test looks really good.

http://www.youtube.com/watch?v=vTGoLmkzlf4&hd=1

Since you mentioned that your team also worked on the iOS app I was wondering if you could help me out with something. The "Additional features coming fall 2012" section on the wifi backpack product page says "use your smartphone or tablet as a live video remote—includes video and photo preview and playback." When I read "playback" I was assuming that we would be able to review and watch photos and/or videos that have already been recorded. Is this something that we will see in a future app update or have I misinterpreted the product description?

Thanks again and keep up the awesome work!

-Adam

David said...

Adam,

Yes, there are many more features coming to the GoPro app, playback will be one of them.

Anonymous said...

GoPro should consider using JPEGXR for lossy compression of linear data. Wouldn't be as efficient as H.264, but would be a nice option as a highest quality mode. Some good info on HDR and JXR here:; http://hdview.wordpress.com/2008/08/05/high-dynamic-range-in-hd-view-3/

Jouni said...

Wonderful article, I'm so glad that you guys are working on this.

For me, dynamic range is so important - we shoot aerials with RED Epic and it's a joy to grade in post. The normal dynamic range on the RED Epic is about 13 stops and in HDRx mode (which takes two exposures) it's about 18.5 stops - let me tell you, you can pull out pretty much anything out of the footage that's been under/overexposed.

I am going to buy the Hero 3 when it's available in the UK to experiment, but was wondering if you guys are planning on supporting Cineform RAW codec on the GoPros (or any other cameras you're working on) to retain that RAW latitude of the footage?

Would also be great to have more control of the settings Protune uses for NR and the log curve.

Also - any plans on introducing a dual-exposure HDR mode like the HDRx on RED or the HDR mode in Magic Lantern firmware on Canon? Your sensor/CPU seems to fast enough for this. That way the dynamic range could increase beyond what the sensor offers.

Would be happy to help in any way I can as I really love what you guys are doing.

You can check our reel at http://londonhelicam.co.uk - password is 'helicam'. There are some shots in the reel that are post-processed HDRx shot with RED Epic - should give a visual example of how having high dynamic range opens up new possibilites in post.

Jouni
London Helicam

gary huie said...

I think the protune update would be great if I could record without my hero 2 randomly stopping? At first I just thought it was the 720x60T but today it did it in 960x48 and the first time I recorded was in 1080 and the video would stop after 17 min but st lest it would start again on its own. I've had my hero 2 for less than 2 weeks and did the update before the first time I recorded. I've only recorded 3 times with it but all 3 times it stoped on its own? has anyone else had this happen?

David said...

Gary,

Try another brand of Class-10 SD media, sounds like it is not quite keeping up with the data rate. I just shoot 2 hours of continuous Protune on my HERO2 without issue.

gary huie said...

What brand is recommended? I'm using a sandisk ultra 30mb/s?

David said...

SanDisk should work, as most class-10 should. Just something to try.

gary huie said...

David do you think its possible the protune update maybe causing this?

David said...

Not really. I gather with Protune off your recordings are fine. Protune is high bit-rate, and I have media that doesn't work only in Protune.

gary huie said...

Actually friday it did it in regular 1080, yesterday was the only time I recorded in protune 720x60t and today I recorded in regular 960x48 hoping it was the protune but again it quit on its own.

David said...

Then you have a question for support as it should not be doing that, and it is not likely the media or the Protune firmware.

gary huie said...

That was my next step
Thank you for your time david

Iain Johnson said...

Why does cineform put a green filter over my vids when converting back to mp4. They were shot in protune.

Iain Johnson said...

Also i use vegas movie hd platinum 11 and vegas pro12, shoot mainly mountain bike videos, what is the best color correction to use either if i shot with protune or normal. Im very new to editing etc.

Anonymous said...

Iain, It doesn't make it green here. Please contact cineform@support.com.

Best color correction, I prefer the color correction within CineForm Studio Premium.

Peter said...

I have just about been hitting refresh all day waiting for both the firmware upgrade to my Hero 3 and the Android app, and I have not been disappointed.your article capped it for me. I am a long time raw shooter on my Nikon, and I'm glad to have the capability on the GoPro.

Now maybe e can get it to work on honeycomb on my Samsung 7.7?

Mr.Stanozol said...

And what about the GoPro3 with the 4K recording? :)

Robert Grant said...

Is there anyway to go from Pro Tune's custom gamma curve to an industry standard 10 or 12 bit S-Log file?

bome said...

Great article! I would like to work in CineForm program which is feee from the GoPro website but I'cant.:( Whenever i try it chraseh when I'm converting clips (even 5sec ones)... Edit is workin great (step2) but not converting... my comp specs are; i7 proc, 512gb ssd, 16gb ram and nvidia facotry OC one with 1gb ram.

David said...

bome, that is clearly a question for support. I don't know why your Mac is not working, support can help you.

Groupdmt said...

Color correction by using color gels, or filters, is a process used in stage lighting, photography, television etc......
colour correction

Олег Большаков said...

Hello, David. Why are you all ("professionals") are not telling of how we can really increase the sharpeness of a protuned video so that it could look as detailed (sharp) as unprotuned one? I tried different plugins for AE - but nothing really helps: Sharpen, Digital Film Tools reFine etc. Can you suggest any post-processing tool that can let us reach a result (I mean detalisation exactly) as it is with protune OFF? Without it protuned video is really MUCH blury!

galenleeds said...

Hello David.

I was wondering what differences ProTune makes in battery life on the camera, and also in file capture sizes (since more information is captured, I am guessing that a 3 minute video using Protune will be larger than a 3 minute non-Protune video). Also, I am guessing that since it is capturing more info the camera is working a little harder and using the battery more quickly. Are these false assumptions?

Thank you

David said...

Nothing much changes on battery life. Protune does less work on color and less denoising, but the bit-rate can be more than double. I don't find these changes an issue on battery, only storage.

Phong said...

This is a fantastic article David! Thank you for giving such a detailed insight report on this topic. I'd be interested how a radiometric linearization of the protune curve looks like. I guess one could build their own LUT ising the codelist above, however if converted to floating values where would midgray and white be on a scale that other cameras use (i.e. Alexa). Also do you have any plans of implementing ACES into CineForm?

Thanks a lot David!


Nhat