Uli Plank – DIGITAL PRODUCTION https://digitalproduction.com Magazine for Digital Media Production Mon, 09 Dec 2024 08:04:06 +0000 en-US hourly 1 https://digitalproduction.com/wp-content/uploads/2024/09/cropped-DP_icon@4x-32x32.png Uli Plank – DIGITAL PRODUCTION https://digitalproduction.com 32 32 236729828 DaVinci Resolve Version 19 – Speech recognition https://digitalproduction.com/2024/09/14/davinci-resolve-version-19-speech-recognition/ Sat, 14 Sep 2024 08:00:00 +0000 https://digitalproduction.com/?p=144904 Wenig hilfreich war eine Normalisierung der Lautstärke.
The public beta version has been available for some time and once again offers numerous new functions. Blackmagic Design (BM for short) is still releasing improved versions every few weeks versions at intervals of a few weeks, but many of the new features are already relatively stable and allow initial tests.
]]>
Wenig hilfreich war eine Normalisierung der Lautstärke.

Until the final version is released, we are concentrating here on improvements of functions that were already available in version 18 but were not yet particularly mature. One of these is the transcription from audio to text.

In our article on 18.5 and 18.6 of DaVinci Resolve (DR for short) in DP 23:06, we already dealt in detail with the then new speech recognition. Unfortunately, we had to admit that there were still considerable comprehension problems. As the competition, such as Adobe with Premiere Pro, is not sleeping in this area either, this time we want to find out whether anything has changed.

In any case, a new feature is the optional recognition of the person speaking, to whom you can subsequently assign a name. Initially, we left the setting in the programme on “Auto” in the hope that the AI would find out the language itself. Unfortunately, this was a disappointment, because in a feature film with clearly understandable German dialogue, the program only stated that a foreign language was present and reported the type of music and a few noises. Incidentally, this was done in English, even though we had switched the interface from DR to German. You would expect the AI to orientate itself to the language setting of the operating system, or at least that of the GUI.

Mit „Auto“ wurde die Sprache nicht erkannt und ein bisschen Englisch halluziniert.
With “Auto”, the language was not recognised and a bit of English was hallucinated.


Only after selecting the language to be recognised in the project settings did we get a more usable result. However, after closer inspection, we again found the hallucinated, multiple repetitions of a sentence that was basically recognised correctly. These also appeared elsewhere in the film without dialogue, often before the actually existing sentence. There were also occasional omissions, as in version 18, with even slightly longer sections of dialogue occasionally being ignored. During a passage of jazz music with a small cast, without any dialogue at all, we got completely nonsensical text mixed with Cyrillic letters (or should that be Greek?).

Subtitles

As we have had relatively good results in the past with the “Create Subtitles from Audio” function in the timeline menu, this was our next test. To our surprise, this is obviously not the same AI, as the language is recognised correctly here even when set to “Auto” and the results tend to be better than with transcription. Where there were misinterpretations, these were not identical to those from the transcription.
To ensure that it was not just the language, we repeated the test with a French film. The results were largely comparable in terms of error rate and type of error, and the differences in language identification between transcription and subtitles were also the same. What was astonishing, however: If the film was in French but the language for subtitles was set to German, the AI can translate subtitles reasonably well. Where the text was recognised correctly, punctuation and spelling were also surprisingly good, e.g. subordinate clauses or questions were rendered correctly.
As well as being accessed via the icon at the top, the transcription can be accessed by right-clicking on clips or a timeline in the Media Pool. It is irritating that after closing the text window, selecting “Transcribe” in the menu opens the window again, as if the analysis had to be repeated. A new feature here is that text can be imported from subtitles or exported with speaker information as a file in .srtx format. Logically, the errors are identical to those in the transcription window.

Erkannte Sprecher können mit den Untertiteln exportiert werden.
Recognised speakers can be exported with the subtitles.

Identify speakers

The menu also contains the new “Speaker recognition” function. After correct language selection, the AI needed less than 5 minutes for a feature film of just under one and a half hours on a MacBook Pro with pure text recognition, with this additional function it was a good 7 minutes. Unfortunately, the results were not outstanding in this respect either. Even male and female voices were not always correctly separated in short dialogues. In longer text passages, there was still confusion between speakers of the same gender, even between young and older voices.

Bisweilen ist die AI verwirrt – nur die negierte Variante kam einmal vor.
Sometimes the AI is confused – only the negated variant occurred once.


In addition, the AI generated far more speakers in the list than were actually involved, especially for short dialogue passages of the same person. On the other hand, the text of a completely new person is often assigned to a person who has already appeared before. The AI had difficulties recognising the same person, especially when changing the tone of voice to express different emotions, as well as when whispering and shouting with room reverberation, even if the text content was recognised correctly. Overall, the new function is still not very useful under these conditions.

Neu ist die optionale Erkennung der jeweiligen Person.
The optional recognition of the respective person is new.


In all tests, we noticed that speech recognition seemed to run much faster at the beginning, with up to 28 times real time being reported. After a short time, the display dropped to considerably lower values, in some cases less than ten times. We therefore also tested the possibility that a short clip could be analysed better than a longer one – unfortunately to no avail, there were no differences.

In diesem verwirrten Dialog gab es eigentlich zwei Sprecherinnen.
There were actually two speakers in this confused dialogue.

Filtering with AI

We had deliberately presented the AI with fully mixed feature films as an endurance test, because even in earlier versions, a commentary in clearly spoken English without any music or atmosphere was recognised almost perfectly. A trial with conventional normalisation only brought us various “Muffled Speaking” messages in the subtitle function, but DR now offers two neural filters for speech called “Voice Isolation” and “Dialogue Leveler”. When set to 100%, the first can actually remove most of the music and atmosphere from speech, but should generally be reduced somewhat as it can distort the sound.

Wenig hilfreich war eine Normalisierung der Lautstärke.
Normalising the volume was not very helpful.


The Dialogue Leveler is primarily suitable for improving the intelligibility of speakers when the volume varies greatly. We also tested it for transcription, although the AI often recognised whispered speech surprisingly well, for example. On the first attempt, the speech recognition seemed to fail completely after activating both filters, but this is probably due to the beta and could not be reproduced. The second time, everything ran smoothly and it only took a little longer than the pure transcription.

Die neuen Filter für Sprache leisten Erstaunliches – aber wenig für die Transkription.
The new filters for speech do an amazing job – but not much for transcription.


Loud music or noises come through in the Voice Isolator as incomprehensible snippets of speech, which are occasionally interpreted as such by the AI. Otherwise, these experiments hardly brought any improvements for the transcription, obviously the AI already does this quite well internally. The errors were only sometimes different, but not significantly less frequent. So you can save yourself the additional computing time in this respect. Regardless of this, both filters can be very useful, especially for documentary film under difficult conditions. They work quite quickly even on modest hardware.

Comment

BM is obviously running its mouth with the following marketing statement: “Due to recent advances in AI and expert system technologies, it’s become possible to get remarkably accurate and perfectly timed subtitles of spoken text using DaVinci Resolve’s Create Subtitles from Audio function.” We were unable to verify this, at least for German and French. Even the position in time is not always accurate, and you still need to check the AI results carefully.
As the errors with identical source material are largely the same as those in version 18, we do not expect any outstanding improvements here either. In addition, DR only recognises a dozen or so languages, while Whisper (openai.com/index/whisper/) understands around 100. However, you would have to install this yourself, as it is open source. The new speaker recognition is also not very useful as long as it is too often unable to correctly identify people in fast dialogues. One wonders whether BM could not improve this by correlating it with the visual person recognition that has been available for some time.

]]>
DIGITAL PRODUCTION Wenig hilfreich war eine Normalisierung der Lautstärke. 144904
BMD Micro Color Panel https://digitalproduction.com/2024/09/09/blackmagic-design-micro-color-panel/ Mon, 09 Sep 2024 13:46:25 +0000 https://digitalproduction.com/?p=144806 Das Micro Color Panel wird mit einem iPad zur kleinsten Grading-Station.
In addition to some new cameras, the Australians have also released a revised version of their smallest grading panel. Unlike the new camera models, it was already available for us to test.
]]>
Das Micro Color Panel wird mit einem iPad zur kleinsten Grading-Station.

However, it only works with the new version 19 of DaVinci Resolve (DR19 for short), which was only available as a public beta version at the time of testing.

The new panel no longer comes bundled with a studio license at that price. Blackmagic Design (BM for short) has presumably realised that only a few buyers of a panel do not yet have one. However, even with a separate license, the price is lower than its predecessor.

Build quality

While the older model had a lower shell made of light metal and thus weighed a good 3.5 kilograms, the new one is made of plastic and weighs just under 1.2 kg. It has also become significantly smaller; at around 36 x 19 x 4 cm, it is only slightly wider than our 16″ MacBook Pro. It is obviously aimed at digital nomads, which is also emphasized in the advertising by the combination with an iPad.
There is a recess in the rear edge, which is not primarily used as a shelf for writing utensils, but is perfectly matched to an iPad in terms of tilt angle and balance. Even the additional cut-out in the centre serves this purpose, as otherwise you would have to remove the tablet to switch an app. However, iPad owners who order the panel straight away will be disappointed: there is no 19 beta version for iOS, only for Windows, Mac OS and Linux. It will only work with the iPad in the final version.
The material doesn’t look ‘cheap’, unlike the older version of the Tangent Wave, for example. But the matt surface is quite sensitive to fingerprints. The tactile feedback of the keys feels even better than on the predecessor with its somewhat spongy rubber keys. The trackballs are no worse, but the outer ring has become much lower. The raised outer edge on the older model may look a little less elegant, but it prevents the trackball from being accidentally twisted. The ring is held in place magnetically and makes it easy to remove the trackballs for cleaning.

Compared to its predecessor, many functions have been added.


Although the rotary controls at the back have been regrouped, they fulfill the same tasks as before. They work with optical coding and therefore have no end stop. Furthermore, the respective value is reset by pressing the button, but the perceived feedback is somewhat less clear. The rotary knobs have also been lowered, which in turn makes it easier to access an iPad. Anyone who previously had a blind grip on the positions for the subdued lighting in the studio will swear at the beginning because you often miss the mark.
Compared to a Mini Panel, all secondary curve functions such as Hue vs. Hue, control of the qualifiers, the colour warper and for Dolby Vision Trim are missing. In addition, there are the small screens with the corresponding information to keep track of the wealth of functions. On the other hand, the Mini is considerably more expensive and requires much more space on the work surface, especially in terms of depth.

Power and connection

The Micro Color Panel can only be connected to the computer via USB-C (with an adapter if necessary) or Bluetooth. An iPad can only be used sensibly via Bluetooth, as the only USB-C connection is usually required for external mass storage devices. With a wireless connection, the panel is powered by an internal battery, for which there is unfortunately no main switch (as with the Speed Editor).
When DR19 is not running on the computer, the panel should automatically go into sleep mode with minimal power consumption. With the first version of the firmware, this did not really work, so that the battery was drained quite fast when the computer was nearby. With firmware 2.2, the battery lasts longer, but not as long as in the Speed Editor. In contrast, we found it easier to connect the panel to a PC via Bluetooth straight away. Under Mac OS Sonoma, we first had to delete several other connections before the panel responded.
Unfortunately, it is no longer possible to delete the entire list in the system settings, as was the case with earlier versions of Mac OS, so you will have to experiment. This problem occurred even though no Logitech devices, which are notorious for such problems, were connected. It was simply an Apple keyboard and mouse (albeit slightly older ones), a Wacom tablet and sound output devices. After successfully connecting to the panel, we were able to reconnect everything without any problems.

Operation

The confusion of experienced users applies even more to the buttons. There are now several more of them, which previously only existed on the more expensive Mini Panel. In addition, most of them have two or three assignments. The separate buttons for the trackballs have been omitted; there is now only one group in the centre with triple assignment. Switchers will often reset the wrong value here too, until it got back into their muscle memory. The remaining buttons are now arranged symmetrically on both sides, and important buttons such as Bypass, Disable, Loop etc. have been moved to the left.
Right-handers will certainly have a different opinion on this point than left-handers, but overall you will hardly have to reach for the keyboard and mouse – the comprehensive multiple assignment is both a blessing and a curse. Even the trackballs and rings have additional functions thanks to the switch buttons at the bottom left, especially when creating Power Windows without a mouse or drawing tablet. There is also no need to switch to the tab for tracking, as this can be triggered directly with the playback buttons using the Shift key. Hold down the right Shift key to control the scaling of clips (input sizing). This is extremely welcome if size adjustments transferred via XML from other editing software are not recognized correctly and quick corrections are required. However, if you want to adjust several parameters simultaneously here, or with Power Windows, you will still have to keep one finger on the Shift key all the time.
You also have to crank for far too long to adjust the soft edges or the inclination of a selection, for example. However, the sensitivity of these operating elements can be adjusted in the user settings under “Control Panels”. You will probably also want to change the brightness of the button lighting there, as it is initially set to ‘full power’. This is clearly too much for any reasonable room lighting when grading. The labelling of the rotary controls is not illuminated.
Some buttons indicate their activation by being coloured green, including OFFSET and CURSOR. Due to the lack of a fourth trackball, the right trackball becomes the corresponding controller when the offset is activated, while the other two outer rings control the colour temperature and tint. Cursor allows you to control the cursor in the display window, e.g. for the colour picker or the control points in the colour warper. The cursor is also visible on the client screen, so that you can point to something. BYPASS and DISABLE, on the other hand, light up red as a warning when activated.
Incidentally, our test device came with German keyboard labelling, and the panel is also available in other languages – we have already seen it in French. You don’t necessarily have to like this, as the German language unfortunately forces you to use rather cryptic abbreviations on the compact device. “WDHRST.” instead of REDO or “SCHLB. HINZUF.” (add keyframe) instead of the already abbreviated “ADD KEYFRM” do not really make orientation any easier. Why translate keyframe literally at all? Node is not translated either. However, we were assured that you can also get the panel in English from German dealers.
Even in English, the buttons only offer space for the standard assignment; you have to learn the respective function or hang it on the wall. The supplied quick start guide, which was translated into easy-to-understand German in our sample, should be kept in a nearby place for the time being. This is obviously the manufacturer’s intention, as it is protected by a plastic surface. You can find a very clear alternative from Simon Lamarre-Ledoux at imgur.com/mK13lkg to print out, and Mr. AlexTech has a video for beginners.

Points of criticism

Some of the mappings are a little unfortunate. For example, you have to spread your fingers a lot between the left shift key and OFFSET to switch to log wheels, as the LOG key no longer exists. This means you only get visual feedback on the screen. And why can’t you switch directly to HDR wheels with the right shift key plus offset? After all, they can be operated with the panel after reaching for the mouse.
Some people on the Internet also complain about the presence of the AUTO COLOR button, saying that it is superfluous for users of such a panel. I wouldn’t agree with that, because with a little background knowledge it can be quite useful. Examples can be found at youtu.be/NLT-Ozou20w by Alex Jordan (Learn Colour Grading).

The toggle button can also be used to copy grades from previous clips in the timeline.
But why can’t you simply hold the toggle buttons by double-clicking, like the capitalisation on iOS, and then report this back in green? When creating Power Windows, this would prevent cramps in the left hand and the rotary knobs could then do a lot more, e.g. when keying. Resetting the values for Power Windows is only possible globally, not for individual parameters.
The “USER” button still has no function; the manufacturer intends to add its own commands later. In areas other than the colour page, at least the play and full screen buttons work, even if this does not replace a speed editor. We tested the new panel under Sonoma 14.6.1 and Beta 6 with a MacBook Pro. The firmware was 2.2, and even that may still be in beta testing to some extent.

Selbst die Verpackung wurde eingedeutscht.
Even the packaging has been Germanised.

Comment

Despite some criticism, the new panel is hard to beat in terms of functionality, transportability and price, while at the same time offering good workmanship. We can recommend it even to beginners, because grading is all about speed, as our eyes adapt to color changes in a short amount of time. You have to learn to master such a panel like a musical instrument in order to become a really good color grader.

]]>
DIGITAL PRODUCTION Das Micro Color Panel wird mit einem iPad zur kleinsten Grading-Station. 144806
Film with a Phone? https://digitalproduction.com/2024/06/19/film-from-the-fon/ Wed, 19 Jun 2024 09:43:52 +0000 https://digitalproduction.com/?p=146483
It's almost 10 years since Austin Reza shot a film for Bentley's luxury cars: on an iPhone 5S. At the time, it was still in stylish black and white and with high contrasts - no doubt also to prevent artefacts from becoming too obvious. Back in 2011, Park Chan-wook and Park Chan-kyong from South Korea even shot the fantasy horror "Night Fishing" on an iPhone 4 in colour and won a Golden Bear for the best short film. Apple itself, on the other hand, has generally shot its advertising films on cameras such as the Arri Alexa.
]]>

But now the “Scary Fast” event for the release of Apple’s M3 computers was filmed entirely on the iPhone, as the credits alone reveal. On YouTube you can find the “Behind the
Scenes…”, where it also becomes clear that the filming and technical effort was actually the same as always(is.gd/iphone_bts). But the dollies, cranes and the drone were actually just sitting on a new iPhone 15 Pro Max. In addition, the predominant visual theme was “black”, matching one of the new laptop colours.

The EU is to blame

This doesn’t mean that this iPhone is a low-light wonder (it isn’t), but that you can set the light to high contrast and then grade it to black. The decisive factor is that recording in ProRes with Log is now available. Apple would certainly have liked to hold on to an in-house connection for longer. But the EU has demanded that all smartphones that come onto the market here in 2024 must have USB-C. Apple has taken the plunge and has already installed the corresponding socket in the new series this year.

Data transfer

In the early productions with iPhones, the employees must have had incredible patience. Recordings had to be transferred via WiFi or the interface with the euphemistic name “Lightning” (or its 30-pin predecessor), which is very slow by today’s standards. Two short test clips of around 750 MB took over ten minutes via “AirDrop”, via USB-C the process was
the process was barely measurable in seconds.

Initially, there were still rumours floating around the web that, apart from charging the battery via USB-C, Apple might restrict the use of external devices to its own offerings or only allow its own cables at pharmacy, sorry, Apple prices.
None of this has materialised: A standard SSD connected via a normal data cable (not just a charging cable, of course) works without any restrictions. With the iPad, we still had to complain that FCP-X cannot work directly from connected storage devices (DP 23:05). On the iPhone, on the other hand, Apple does not force internal recording, as already available apps show. The Apple event was obviously recorded with the help of the free Camera App from Blackmagic (BM for short), Apple even thanks us in the credits.

Storage media

It is clear that Apple also wants to satisfy its shareholders: the new options are limited to the two top models. With the “Pro”, you also have to pay an extra 130 euros for 256 GB of internal memory to get full functionality with the memory-intensive ProRes even without an SSD. This is the basic configuration for the Pro Max. The internal memory is really fast, with 1.6 gigabytes per second (GB/s) when writing and reading. Measured with Diskio by Sergii Khliustin, a free app with witty comments – is.gd/diskio.

Externally, only regular USB-C is supported, no Thunderbolt. However, with a good SSD, this is completely sufficient to record ProRes 422 HQ in UHD with up to 60 fps directly, for which you only need around 200-225 MB/s. Initially, the app even listed ProRes 4444, but that was probably too much for the system. Even on an Arri Alexa, this codec is not always selected, and 422 HQ is completely sufficient as a source for the iPhone. In UHD at 60 fps, it needs just under 800 GB of space for one hour.

A Samsung T5 SSD, for example, as external storage shows around 373 MB/s throughput when writing, depending on the app being measured. Even a T7, which is not recommended for BM cameras, ran for two hours on the iPhone at 25 fps. The free OWC Drive Speed app can actually directly measure the suitability for various codecs, resolutions and the achievable frame rate. However, it already warns of picture dropouts,
without us being able to detect these in practice for the respective media, so this app is probably not yet fully developed.

Even a T5 is still relatively large for an iPhone, although thanks to MagSafe it can be easily attached to the back with a magnetic ring. Some people wonder whether a smaller flash drive with USB-C would also work. Samsung has relatively fast models for this, currently with a maximum of 256 GB. According to the company (we didn’t have one to test), these should be able to read 400 MB/s, but only write 110. This information is always given with the small addition “up to” and only refers to the 256 GB model. This should still be sufficient for ProRes 422 HQ with 25 fps, but ProRes 422 LT could also go up to 60 fps. Unfortunately, such sticks are often not compatible with the mobile phone case.

We have also not tested the idea of connecting a really fast SDXC card with a small USB-C adapter. But a Sony “Tough” or comparable cards with 300 MB/s should work with a really fast adapter. A card reader can of course also be useful for backups on the iPhone when working with conventional cameras. If you want to edit the material on the PC, you should pay attention to the formatting of the media: The iPhone does not recognise NTFS, but exFAT works up to 2 TB.

Be careful, there is a risk of data loss! The “classic” HFS or the new APFS, which the iPhone naturally supports, are better. One of these formats would therefore also be advisable for transferring to the PC. Under Windows there are programmes such as “HFS for Windows” or “APFS for Windows” from Paragon for using these formats. Of course, the iPhone also uses these two formats.

Power and heat

A fully charged Pro Max still showed 66 per cent charge remaining after one hour at 60 fps, and 42 per cent after two hours of recording in ProRes 422 HQ at 25 fps in UHD, with the T5 SSD also being supplied with power. The SSD absorbed the greater proportion at 25 fps, at 60 fps it is the other way round. The iPhone 15 Pro still had 58 per cent after one hour at 60 fps, the camera app needed the larger share, the SSD about 2/3 of it. This was the app from Blackmagic. This value is not the same for all apps, as we will see in their tests. These are amazing runtimes, but the iPhone does get very warm, despite iOS 17.1.1, which has essentially eliminated the initial heat problems. You should definitely remove any silicone case, we got overload messages at 60 fps on the Pro Max after a few minutes at an ambient temperature of 27 degrees. Another tip from BM was to switch off the overlays when recording, i.e. to “wipe them away”. This allowed the Pro Max to run smoothly even in slightly warmer conditions.

At 60 fps with the case and overlays, the iPhone got too warm.

The iPhone 15 Pro may seem more attractive for video with its 3x lens and slightly lower price. However, its battery is smaller; in stand-alone mode with SSD, 20 per cent charge was still left after two hours. Unfortunately, the smaller housing also cools less well, and despite the above cooling measures, 60 fps was over after a few minutes. At 25 or 30 fps, the Pro also lasted without overheating and the app consumed less power in relation to the SSD. Very fast SSDs, such as a Samsung 980 Pro in an NVMe enclosure, require too much power and only run on both iPhones with an external power supply via USB-C hub. You don’t necessarily have to take a hub with you to externally power the iPhone itself, as there are already power banks for MagSafe.

Frame rate and synchronisation

With a runtime limited only by power and storage medium, another question arises: What about maintaining the frame rate and thus the picture-sound synchronisation over a longer period of time? In Apple’s film, no statement was longer than 20 seconds without a cut, which any smartphone can actually manage without becoming asynchronous. But unfortunately, practically all of today’s smartphones do not provide a fixed frame rate, but a variable one (VFR = Variable Frame Rate). Although DaVinci Resolve (DR for short) simply reports a highly precise 60,000 under “Shot Frame Rate”, the iPhone is unfortunately no different. Before recording, we switched to aeroplane mode and closed all other apps to rule out any interference.

However, a further measurement showed that this was not necessary. In the free MediaInfo, the information is slightly more correct, showing a frame rate between 59.940 and 60.181 fps – that’s a deviation of far less than one per cent! A longer recording with a film gate showed that the variable frame rate in DR is relatively unproblematic. The sound ran continuously one frame early for over 30 minutes (at 60 fps!), only at one point did we have the impression that it tended towards two frames. Since in DR the picture occasionally moved one frame ahead of the TC flap, which we kept continuously in the picture, the editing software probably does react to VFR. This is usually manageable, but the iPhone is probably not quite suitable for uninterrupted concert recordings lasting several hours.

The prism-folded 120mm is a technical masterpiece.

Cameras and lenses

A few years ago, the statement “My smartphone has three cameras and a LiDAR scanner” would probably have been met with incredulous smiles. But there are actually three lenses here, each with its own sensor, just like before. In fact, most of it is even identical to the iPhone 14, only the signal processing has been further developed. The ultra wide-angle lens with 13 mm, the regular 24 mm and the 77 mm in the iPhone Pro are identical to their predecessors.

Only the 120 mm in the Pro Max is really new and a very clever design with a multiple prism that basically folds a relatively long focal length into the flat body. To us, such a design seems potentially more durable than a miniaturised zoom lens like the competition (usually referred to as a periscope lens), as there are no moving parts other than the sensor. It is therefore not an optical zoom at all, as is sometimes incorrectly written. Please note that in the following we use the term KB full frame for the photographic full frame of 24×36 mm, because experienced film cameramen tend to mean the S-35 full frame. However, its diagonal is not always the same, but depends on the respective widescreen format. The information provided by Apple and in the apps refers to the equivalent focal length of KB full frame. The main lens 1× has a real 6.86 mm with f 1.78 and the ultra wide-angle 0.5 has a real 2.22 mm at f 2.2, the 3× has 9 mm with f 2.8 and the 5× has a real 15.66 mm, also at f 2.8.

The sharpness right into the corners and the lack of vignetting or distortion are partly due to software.

The iPhones also offer a zoom across the lenses, but this is based on continuous digital scaling. Unfortunately, this shows a disturbing jump when switching to telephoto due to the parallax of the separate cameras. In addition, even with sophisticated algorithms, the sharpness drops slightly when the resolution of the sensor falls below 48 megapixels (mpx), which is particularly noticeable on the way to 5x. This is a disadvantage that longer focal lengths also have with conventional cameras: The 77 mm only focuses up to 60 cm at close range, the 120 mm only up to 1.3 m.

You have to be careful here: The iPhone should be set to “Keep setting” for macro control. Otherwise, only the small flower indicates that the macro range is switched to the ultra-wide angle camera, although the original lens is still selected. However, this only has 12 mpx, unlike the standard camera. This automatic function can be deactivated. With the default setting, the photos from the 24 mm lens are downscaled to 24 mpx and recorded in HEIC, the full 48 mpx are only available in RAW (DNG format). The main camera also delivers the best quality for video thanks to downsampling. And for the selfies: The front camera has 2.69 mm at f 1.9, so it looks like a 15 mm, but only has just under 8 megapixels (actually photo cells = Sensel).

Photography

The DP is not explicitly aimed at photographers, but in view of the different resolutions of the sensors, a few comments seem appropriate. In our test of the Sony A7SIII (DP 21:05), we already proved that megapixels aren’t everything. Just as the Sony “only” offers 12 mpx, images from the iPhone would also be perfectly suitable for an A4 page. For large posters, however, we would generally recommend professional cameras, even if Apple sees it differently in its own advertising.

A section of a photo from the main camera in RAW Max – and the same section from the Sony A7IV at a similar focal length.

It should be noted that the sensor has an aspect ratio of 4:3, which is even closer to the square than the usual 35mm with 3:2. The HEIC photos are very compact at around 2.8 MB, whereas DNG requires over 67 MB for the same subject. A section in 2,400×2,400 pixels from the iPhone at 120 mm (equivalence) with the same section from the Sony A7 IV with approximately the same focal length differs primarily, of course, in the real bokeh, which is not simulated here on the iPhone either. The slightly different colours can of course be adjusted, as both are raw images with plenty of colour depth.
The resolution of the test chart in 48 mpx from the 24 mm of the iPhone is excellent and shows no moiré, but slight colour noise even at ISO 320.

The test panel from the 24 mm in DNG shows excellent resolution

The counterpart in standard setting as HEIC with 24 mpx shows less noise, but colour moiré and clear edge sharpening. The modern HEIC compression is very high quality in itself, but unfortunately Apple applies all the “image enhancements” to these photos that the fast A17 processor is capable of, such as smoothing and sharpening contours or local colour tone mapping. Edge sharpening in particular looks just as distracting as it does on amateur-class drones. Of course, Apple also knows that the sharpening of details in faces is usually undesirable. Accordingly, skin tones tend to be smoothed out, which often gives them the infamous “plastic look”.

In compact HEIC, on the other hand, light moire and artificial sharpening are used.

The design of these functions is certainly based on comprehensive market analyses, as the vast majority of users will simply take photos with their mobile phones and presumably find all this attractive. Professionals who don’t want to be deprived of the opportunity to tweak their pictures themselves will therefore have to work with RAW Max (as it is called in the menu) and plan the necessary space for it, either conveniently and expensively in the iPhone or more cheaply on an external storage device. However, raising your eyebrows at “computational photography” in general is not entirely appropriate. After all, most professional cameras have been correcting various lens errors in the background for years, primarily vignetting and distortion. This means that lenses can be optimised in other respects without being sinfully expensive.

Night shots from the iPhone can be impressive – the sky was black to the eye.

The iPhone goes beyond this and achieves amazing things in two areas: bokeh and low-light. The ability to use the (actually relatively coarse) information from the LiDAR to subsequently influence and target the depth of field was already available on a number of predecessor models. But Apple has significantly improved the algorithms and, at least in photography, the results can now only be recognised as fake with intensive pixel peeping, even with hair and other difficult edge zones.

Ein 24 mm Glas ohne digitale Korrektur zeigt auf der Sony Vignettierung und Verzeichnung.
A 24 mm lens without digital correction shows vignetting and distortion on the Sony.

The results in RAW (DNG) are impressively good, while artefacts can be seen in HEIC. However, we also photographed the test panel at 320 ISO for the wide angle and 500 for the telephoto. Vignetting or chromatic aberrations are hardly recognisable and the distortion correction for 24 mm is also better than with the uncorrected professional lens. Even moiré is hardly noticeable at 48 mpx from 24 mm and the resolution is even slightly higher, as the A7IV only has 33. At 77 mm or 120 mm, the A7IV has a recognisable advantage over the 12 mpx of the iPhones, although it was at a slight disadvantage in terms of speed each time. A 13 mm was not available to us for the full frame, so no comparison was made.

Die Skalierung führt zu leichtem Moiré, das aber bei üblichen Motiven kaum auffällt.
The scaling leads to slight moiré, but this is hardly noticeable with normal subjects.

The Sony’s HIF images look cleaner because, apart from subtle noise filtering in extremely low light, everything else can be deactivated. In low light, the iPhone does all by itself what, in principle, any camera on a tripod can do: A series of photos that are then combined to reduce noise. Red has been offering something similar for 10 years (see DP 13:03). With the iPhone, this also works with good stabilisation from reasonably steady hands, but you can get even more out of it with a tripod. Some even recommend this for astrophotography; there is even a free app for this, AstroShader, which can also save raw images.

Die Sony ist im Vollbild praktisch frei von Moiré.
The Sony is practically free of moiré in full frame.

Now, we have too much light pollution near a city for a great starry sky, but you can focus on the lights of the city itself instead. It’s impressive what such a tiny chip can do with the help of a computer. The tree in the foreground was barely distinguishable from the night sky with the naked eye. Apple is clearly making a virtue of the A17’s computing power out of the necessity of the tiny sensor and calculating far better images than you would expect from a mobile phone. However, these calculations take time and cannot be used for video. So how do the 15s perform when filming? We use the Sony A7 IV (see DP 22:05) as a reference here, even if some people seriously compare the images with those from an Arri Alexa in grading(is.gd/quazi_iphone_arri).

Film: Resolution

When filming from the high-resolution 24 mm sensor in UHD, the iPhone not only scales down the 48 million photocells (aka Sensel), but also crops slightly from the 4032 horizontal pixels available per photo to 3840. So if Apple had only added a few more Sensel, the iPhone could even deliver the 4096 pixels of the DCI cinema format. There would be more than enough vertical pixels anyway. Other formats are possible with third-party apps; we will go into this in more detail in the respective apps. The same applies to the Ultra-WW and the telephoto lenses, which logically have less oversampling, but are also slightly cropped.

The downsampling of the 48 mpx at least offers the possibility of noise reduction by integrating the neighbouring pixels or a digital 2× zoom without loss of resolution. However, when zooming through to 5× telephoto, not only does the parallax error mentioned above occur, but the resolution also collapses beyond 2×. Up to 3× on the iPhone 15 Pro, these effects are correspondingly less noticeable. A 28 mm and a 35 mm are still simulated quite well by both iPhones. This is comparable to Sony’s digital zoom in the camera, which works with similarly good algorithms, but cannot deliver the full resolution beyond around 1.7.

The resolution of the iPhones is also reduced at the higher levels of stabilisation. “Standard” only uses optical stabilisation by mechanically tracking the sensor, while “Cinematic” and “Extreme” additionally track a reduced area of the sensor surface. Unfortunately, “Extreme” in particular also shows overshoots – or is this intended to simulate the inertia of a Steadicam? Be that as it may, a small gimbal can do this better. Another disadvantage of the two higher stabilisations are massive deviations in the frame rate in the first second; the iPhone is obviously “stressed” here (thanks to Kurt Friis Hansen for these measurements).

The other sensors with 12 mpx naturally also offer less resolution reserve when filming, but are completely sufficient for UHD. For comparison: On S-35, the 5× lens of the Pro Max would correspond to about 75 mm, the 3× to about 48 mm. Because a suitable prime lens was not available for the Sony, we used a very high-quality zoom lens. However, this actually put the A7IV at a slight disadvantage in terms of speed. In all shots, the film images of the iPhones on the test chart show slight moiré and some loss of resolution due to scaling; the Sony obviously has better downsampling. The differences are small at 24, 77 and 120 mm and the moiré can practically only be seen on the test chart. Other cameras only deliver better results with very good downsampling of images in 6K or more.
It is obvious that a lot of calculations are also carried out on the images in log, as the charts show no vignetting and hardly any distortion, just like in photography. This is not to be expected with such lenses without software correction. The very high-quality Zeiss C/Y zoom on the Sony shows more optical errors due to the lack of software correction. It should be understandable that a lens with a real focal length of just under 16 mm (the 5×) without software tricks does not offer such a differentiated depth of field as a real 120 mm for KB full frame (see above). All lenses definitely show some breathing during focus shifts – if Apple doesn’t simulate this as well 😉

Film: Compression

Here, too, the comparison with the Sony A7IV, which only records 10 bit with 4:2:2 as H.265 for maximum quality, ProRes is not available without an external recorder. Our endurance test was a water surface in the rain, which can really torture any GOP codec (such as H.264/265). The iPhone only recorded a good 7 MB/s at 60 fps in UHD, while the same motif was recorded in ProRes 422 HQ at 213 MB/s. The Sony A7 IV records this codec at 50 fps with 25 MB/s. Nevertheless, the typical artefacts, such as block formation, can hardly be seen in the correctly exposed motif on the iPhone. However, this does not only speak in favour of the quality of the encoder alone, as the finest details are already reduced before compression.

This becomes clear with a shot of the same subject that is underexposed by just one f-stop and corrected in DR. Despite 10 bit and 4:2:2, the signal in H.265 already shows signs of banding and is overall much less “full” and more coarsely structured than the version in ProRes. The same applies to noise, which is also filtered more heavily before compression and then “muddied” by the codec. The noise remains much finer and more “organic” in ProRes. It is therefore less disturbing in itself and is also more suitable for careful filtering in post-production. Consequence: If you don’t have enough storage space on location, H.265 is perfectly usable in Log, but should be exposed as correctly as possible. ProRes is considerably more tolerant, while Cinema P3 can record better H.265 at higher data rates (see p. 128).

Film: Motion display

We can be brief here and take the liberty of quoting the values from CineD (is.gd/cined_test) because our reference fan was not available. For UHD, our colleagues measured 5.3 milliseconds for the sensor at 24 mm, and even 5 ms at 77 mm and 120 mm. Our subjective impression from our own tests agrees with this. These are excellent values, also in comparison to conventional film cameras. The jelly images of older iPhones are definitely a thing of the past and action is the order of the day.
However, the Bentley film mentioned at the beginning was still noticeable with the jerking of fast movements at only 24 fps. A look at the individual images shows that they lack any motion blur. Even these tiny sensors are too light-sensitive today to comply with the filmic rule for sufficient motion blur in daylight. After all, mobile phones primarily use the time for exposure control, because the aperture is fixed. Without an ND filter, the only option is to regulate the ISO value, which the iPhones also make intensive use of. But this has consequences for the usable contrast range.

Film: Dynamic range and colours

The HDR shots from an iPhone with the corresponding screen are very impressive at first glance, but on closer inspection the sharpening becomes unpleasant. Added to this is the local tone mapping, which is particularly noticeable with intensely coloured light sources and soft colour gradients in the moving image. Only recording in Apple Log avoids these annoyances, which is why we limited ourselves to it for dynamic measurement. Apple Log is excellently tuned, it does better justice to the iPhone than an A7 IV with its S-Log3, because even for 10 bit it is still very flat. Incidentally, in the current version 18.6.4, DR understands the Apple Log with the colours in Rec 2020 with automatic colour management.

Beim Sonnenuntergang 
hält das iPhone noch mit.
The iPhone still keeps up with the sunset.

The amazing night vision in photos ends for the iPhone when filming, as there is no time for the exposure tricks from photography. We also used the Sony A7 IV for comparison. It is not quite as sensitive to light as the A7S and also works with noise reduction in low light. Compared to the iPhone, however, it shows the richness of detail that a full-frame sensor can achieve when collecting light.

However, we came across an interesting phenomenon that suggests multiple exposures in the highlights. When shooting at 50 fps in Log with a longer exposure time of up to 1/80 second, the highlights are consistently limited to around 70 per cent. If you now go to at least 1/96, the values expand into the space above. Hardly any further information can be found about this function, only CinemaP3 refers to it as EDR (Enhanced Dynamic Range).

In the documentation for developers, the term is primarily used for playback on HDR displays. Red has been offering something similar for recording under the name “HDRx” for some time, but probably does not hold a patent on this (as with the compression of RAW in the camera). Obviously, two (perhaps even more) readouts of the sensor are combined here as soon as the time until the next single image is sufficient. 7.5 milliseconds is obviously not enough, but 10 ms is fine. This is less noticeable at 25 fps, as the usual exposure time of 1/50 with 20 ms is easily sufficient. Here the effect already starts at 1/40, at 1/30 the remaining time is still too short – which mathematically corresponds to the behaviour at 50 fps with just under 7 ms. The resulting distribution across the log curve can be easily controlled using ND and ISO.

So wenig „Rauschteppich“ bei ganz kleinem Sensor ist nur mit NR möglich.
So little “noise carpet” with a very small sensor is only possible with NR.


At the other end of the scale, it is noticeable that there is practically no recognisable noise. You can simply cover the lens with manual exposure and the line drops to zero. So there is obviously a lot of filtering going on in the camera, because such a small sensor cannot be so low-noise despite all the technical progress. This becomes quite clear when shooting at high ISO values with short exposure times, as the lower mid-tones are already very noisy. Medium ISO with longer exposure times using ND looks much better.

So there is no “correct”, native ISO value that always fits. The iPhone makes too many adjustments for this, even with Log. It all depends on where the important details are: If you want to bring out the highlights, e.g. add structure to a cloudy sky, a higher ISO value is appropriate. EDR reduces the contrast in the highlights, but the details are there. If the shadow areas are the most important thing, on the other hand, you should go for a low ISO in order to provide the sensor with enough light, then the highlights will be compressed, but largely without clipping.

Einen Korrekturspielraum von 5 Blenden würden
A correction range of 5 f-stops would

In our exposure series with EDR, 2 f-stops overexposure and 3 f-stops underexposure were still easily correctable, the colours become weaker, but do not derail. Incidentally, they are excellent: especially in the critical area between red and yellow and thus with the skin tones, the iPhone shows a precision landing. Noise is visible in dark areas, but is fine-grained in ProRes and can be controlled in post. We worked with DR’s colour management and only corrected brightness and contrast. We measured an extremely high-contrast motif with a cloudy sky and molleton in the shadowy foreground with 9.5 f-stops – this would still be usable in post (the blue tone on the blacks is scattered sky blue).

9,5 Blenden haben wir zwischen Molton im 
Schatten und der hellsten Wolke gemessen.
we measured 9.5 f-stops between Molton in the shadow and the brightest cloud.


After all these observations, I must also note that Gerald Undone and Patrick Tommaso’s categorisation of ISO 1,100-1,400 as the native value should be treated with caution. The two did not explain whether they used ND filters or varied the exposure time(is.gd/undone_iphone). But DXOmark also assumes 1,250 to 1,480. I tend to favour ISO 200-800 for noise reduction and use EDR with sufficiently long exposure times. Of course, ND filters are the order of the day in bright light, otherwise the typical staccato films from action cams will result.

Film: Equipment

Basically, the only indispensable accessory for good pictures is an ND filter. But a variable ND consists of two polarising filters and has the corresponding side effects, from the infamous “X” to altered colours (warmer ones in our case). Conventional ND filters are better. We were unable to observe any noticeable IR contamination with an inexpensive set, even at 10 f-stops; Apple probably has this well under control. Filters can be attached with a simple clamp mount, but this should not cover too much of the interface on the side of the screen. In addition, such a mount does not fit well with the usual transparent silicone covers, as stray light can then enter from behind. Such covers are also not conducive to cooling in warm environments.

If it’s not too warm, a thin, black cover would be useful. If you want it more perfect, you’ll be back on the road to “Frankenrig” at some point, because the accessories industry has immediately adapted to this. But there are alternatives that can still be put in your pocket as a protective frame without all the accessories. The company Beastgrip has been involved in the development since the Bentley advert and also produced the universal cage that was used for Apple’s event. Their “Beastcage” will soon be released in specific versions for the iPhone 15 Pro and Pro Max and offers a very sophisticated system, including filters, macro lens and anamorphic lenses. Another provider is Freewell with the “Sherpa” – the name of course says it all, especially for the iPhone. It works with clever magnetic holders for filters or lenses.

Freewell not only offers filters, but also anamorphic lens attachments. These are available in both 1.33 and 1.55 and at Freewell optionally with streaks in classic blue as well as in “gold”, i.e. warm yellow. However, the streaks are very pronounced, including the typical side effects such as edge blurring and distortion. However, the 1.55s in particular offer much more horizontal image area, even if they are not easy to align. There is no gain in resolution associated with the anamorphic lenses in ProRes Log. When recording film, the full area of the 4:3 sensor can only be used with H.265 in 8-bit HDR or Rec 709 and then shows all the typical “blurring” effects.

Be careful if you want to use existing filters with a larger diameter via the step-up ring: the 24 mm main camera can then vignette. In addition, the stabilisation of the iPhone does not work with anamorphic lenses and sometimes the iPhone “sees” the adapter as an object and goes into macro mode. So is this more for wannabe Spielbergs? The only filters you really need are NDs and possibly a circular polarising filter, as neither of these can be replaced in post. There are excellent software solutions for everything else, such as “Scatter”.

Aber Vorsicht bei 
maximaler Einstellung!
But be careful with maximum setting!

Advantages and disadvantages

So where do we stand with such a powerful camera phone? Well, first of all, an “always with you” camera is definitely better than no camera at all. In low light, a good hybrid camera like the Sony A7 IV is still clearly superior and you have an almost infinite choice of lenses, but all of this is still much more expensive, even compared to Apple’s prices. If you know your camera really well, you can
blindly with the viewfinder and at the same time see the subject in isolation without distraction. Nowadays, the monitor can usually be swivelled and rotated, but the iPhone’s monitor is still visible even in sunlight.
But you don’t always have all those nice lenses with you. Dust on the sensor when changing lenses is also not an issue with mobile phones. The iPhone is even waterproof, where larger cameras can only withstand splashing water. And you hardly stand out with it, whereas a “real” camera with a longer lens might be eyed suspiciously or even rejected at the entrance. The iPhone 15 can achieve cinema quality in the documentary sector and would certainly have been a dream come true for authors such as Michael Winterbottom with “In This World” or Wim Wenders with “Buena Vista Social Club” – these films were released to the cinema in large parts by DV.

Commentary

Apple has managed an amazing balancing act with the iPhone 15 Pro: 99 per cent or more of users get nice pictures without any great expertise and professionals get a camera that, with the right app, can do far more than a conventional smartphone. The pig called “game changer” is driven through the digital village almost every day – mostly ephemeroptera (mayflies). But the Pro models of the iPhone 15 could actually signal a paradigm shift, similar to the Canon 5D Mark II 15 years ago. Just as that marked the beginning of hybrid photo/film cameras, the iPhone is now a working tool for professionals. And unlike a standard camera, you can even use it to make phone calls..

]]>
DIGITAL PRODUCTION 146483
SynthEyes at BorisFX https://digitalproduction.com/2024/06/07/syntheyes-at-borisfx/ Fri, 07 Jun 2024 17:49:00 +0000 https://digitalproduction.com/?p=144243
3D camera tracking, also known as "match moving" in the professional scene, is now offered by every better software for video editing or 3D. But when we compared a few such popular 3D trackers in DP 18:01, SynthEyes was already our high-end reference for good reason.
]]>

The programme was developed largely single-handedly by Dr Russell Andersson over more than 20 years and has long been established not only in the film industry, but also in architecture and forensics.

It has now been incorporated into the BorisFX portfolio. As with Mocha Pro – originally from Imagineer – you can expect this programme to be maintained and developed further. Compared to the remaining competitors 3DEqualiser and PFTrack, the price is quite attractive (Boujou disappeared a few years ago). A good reason for a more comprehensive test!

Mit Rotoscoping-Masken bekommt ihr schwierige Clips in den Griff.
Rotoscoping masks help you get to grips with difficult clips.

Interface and operation

At first glance, the GUI looks a little old-fashioned, but don’t let that fool you. The fact that the drop-down menus are usually not sticky and you have to hold the mouse button down until you select the desired function can also be a little irritating. You should forget about the option to switch to languages other than English. It is based on an AI that sometimes makes you go “Huh, what’s that, please?” and where German text often doesn’t fit into the field. The assistant called “Synthia” is also not very helpful at first glance and initially responds stubbornly with “Sorry, I don’t understand.” This is not Siri or Alexa, but rather an assistant for comprehensive automation through scripts in SynthEyes with defined commands. Russ shows how this works here: is.gd/automationsyntheyes.

Detail­lierte Tool-Tipps helfen beim Einstieg, aber die Übersetzung passt nicht überall.
Detailed tool tips help you get started, but the translation doesn’t fit everywhere.

But that’s it for the criticism, because there is not only a manual with over 800 pages in English, but also various video tutorials that Russ himself has published over the years. A few of the most important ones have already been published on his own YT channel “Boris FX Learn”, even if Russ’s are a little older. This one on masking moving objects using rotoscoping should be very useful for many: is.gd/rotomasking, or this one on mesh building: is.gd/meshbuilding. New additions are is.gd/solving_AE and is.gd/lens_AE, which help with getting started and transferring to After Effects.

In addition to the actual manual, there are nine further PDFs on specific topics such as Planar Tracking or Camera Calibration under “Help”. Comprehensive tool tips and warnings if you make mistakes complete the whole thing. Otherwise, the wealth of functions with 13 tabs and many hidden, additional windows may seem overwhelming at first glance, but right under the first tab “Summary” there is a large, green button called “Auto” at the top left. You can try it out at the beginning, even if experienced professionals turn up their noses at it. Nothing prevents you from refining the results yourself. We also unleashed this function on our more than 20 test clips to see how difficult they were for the programme. There was only one that was not successfully calculated, even if the automatic function sometimes failed to achieve the magic value for precision below 1.0 pixels straight away.

Performance

Precision and speed are of course the most important criteria in everyday production. There are three steps in the process: The identification of suitable image points for tracking, here called “blips”, their tracking over a number of frames as 2D tracking and finally the “solver”, which is the calculation of the spatial relationships for the camera and the scene. Anyone who has ever worked with a point tracker will know what suitable shots should look like: lots of depth of field, little motion blur and good contrasts. For 3D, the parallax of a freely moving camera must also be taken into account.

Selbst bei diesem Clip stand der Testkörper auf Anhieb stabil.
Even with this clip, the test body was stable straight away.
Auch mit Wald und Fluss hatte die Software keine Probleme.
The software also had no problems with forests and rivers.
Die klassische 3D-Teekanne wird als einer der Testkörper geboten.
The classic 3D teapot is offered as one of the test objects.

SynthEyes can also process panning shots from a tripod, but then you only get a circular horizon. We used shots from a drone with a relatively small sensor, hand-held shots with an iPhone 15 and some from a Sony A7IV with a wide-angle lens, also hand-held. As experience has shown that even simple trackers can cope well with shots from an urban environment, we didn’t want to make it so easy for the professional software. We mainly flew over cultivated fields with repetitive structures and natural shapes such as trees, bushes and water, with a few houses thrown in for good measure. Simple trackers usually have difficulties with such motifs.

After Effects und Fusion hatten mit mehreren
Testclips Probleme.
After Effects and Fusion had problems with several test clips.

Not so SynthEyes: With the exception of one, all clips were successfully calculated with “Auto” and could then be brought to values below 1.0 with just a few interventions, in the majority we even came close to 0.5 without much “manual work”. The duration of the clips was between just under one and five minutes, the resolution was UHD. All clips took less than their own runtime on a modest MacBook M1 Pro. The longest, which primarily contained trees, flowing water and sky, was calculated in 3:45, with around 7,500 images. The precision was 0.75 at the first attempt without any intervention. That is clearly professional level!

So gute Werte bekommt man schon mit ein paar zusätz­lichen Trackern.
You can achieve such good values with just a few additional trackers.
Der Graph Editor zeigt die Dauer und Präzision der Tracker über die Zeit.
The Graph Editor shows the duration and precision of the trackers over time.

The 10 CPU cores were fully occupied, on a computer with more “steam” the programme would hardly leave time for a sip of coffee. For comparison: The camera tracker in Fusion took over 13 minutes with the standard setting for a clip of 1:30, but achieved a very good precision of 0.23. After Effects took over 20 minutes for the same clip and achieved 0.47 pixels. These are good values, but with these times you wouldn’t want to do much fine-tuning if a scene is more difficult. In this case, SynthEyes took a good minute and achieved 0.84, but offers plenty of potential for fine-tuning.

Selbst mit repetitiven Strukturen kommt das Tracking zurecht.
The tracking can even cope with repetitive structures.

Lens data

With early 3D trackers, it was a common recommendation to enter the exact size of the chip and the focal length of the lens. Unfortunately, however, lens manufacturers are not always very precise with the focal length. It is even more difficult today with the chip size. The absolute size can usually be found out, but it is much more difficult with the actual area used. With CMOS sensors, a smaller section is often used, either for higher image frequencies or for internal lens correction. It is usually not possible to find precise information on this.

For our drone, for example, DJI specifies a field of view of 84 degrees diagonally and gives a full-frame equivalent for the focal length of 24mm. However, this applies to photos on the full chip area with an aspect ratio of 4:3. The chip area used for video in UHD is almost impossible to determine, as the small lens is certainly also used for calculations – in any case, the image hardly shows any distortion. With the iPhone, it is generally the case that lens errors are factored out so that the real values are also largely unknown here. It is therefore very helpful that SynthEyes can calculate the necessary distortion correction itself. In the case of the drone, it arrives at just under 70 degrees, which is not unrealistic.

In order not to disturb this calculation, you should never activate internal stabilisation. Both the shifting of the image on the chip and the optical stabilisation, where the entire sensor is moved, would constantly shift the centre of the lens. Mechanical means, such as a gimbal or Easyrig, are of course permitted because the camera itself moves. Blurring, be it a shallow depth of field or too much motion blur, is also not a good starting point for match moving. It is better to add both based on the tracking information in post-production, and SynthEyes can also take care of the stabilisation.

Equalisation of lenses

SynthEyes offers highly complex algorithms for rectification, Standard Radial 4th-order is recommended for spherical lenses, but there are also algorithms for anamorphic lenses. “Read or Cry” should be taken seriously, because if you use this feature incorrectly, the results will tend to be worse. It is an iterative process in which the parameters should be worked through from top to bottom. To do this, you have to switch from “Automatic” to “Refine” in the “Solver” window at the top left after the first tracking and click “Go!” once after each step. In our 90-second clip, one step took less than a second, and this alone almost halved the deviation to 0.48.

Even a rolling shutter is corrected reasonably well, but is generally not a good prerequisite for camera tracking. With the drone or the iPhone 15, this value was low, but with the full-frame camera, this adjustment still brought quite a lot of improvement. In a further step, you could increase the number of trackers and remove the less good ones, but this was not necessary here. We had no difficulty getting below the magic 1 in any of the test clips.

Die Objektiventzerrung sollte Schritt für Schritt erfolgen.
Lens equalisation should be carried out step by step.

With anamorphic lenses in particular, the question then arises as to whether we should keep the clip in its corrected form after compositing or undo the correction. As a rule, the distortions are not removed in the final result, but are added again after compositing. SynthEyes also has a solution for this, which can be found under “Lens Workflow”. This means that both options are available, and it is particularly easy to use scripts for After Effects, as shown in the second clip for beginners (see above). In this case, SynthEyes even warns you if you have not taken this aspect into account.

Bei der Übergabe an externe Software entscheidet ihr selber,
ob die Entzerrung erhalten bleiben soll.
When transferring to external software, you decide yourself whether the equalisation should be retained.

Input and output

In addition to precision and speed, a solution for professionals must of course be very versatile in order to fit into any workflow. Right from the start, we were pleased to see how many video formats are accepted. This even includes material in HEVC 10-bit 4:2:2 from hybrid cameras with up to 8K, which is something that the standalone version of Fusion does not recognise. Of course, ProRes is also no problem, but DNxHR only works in a MOV, not in MXF. Image sequences commonly used in animation and VFX, e.g. EXR, are read without any problems, as are DNG, JPEG or PNG series.

Der Image Preprocessor kann LUTs laden oder direkt das Bild korrigieren.
The image preprocessor can load LUTs or correct the image directly.

Of course, RAW from Arri, Red or Blackmagic is accepted, but currently no MXF clips from the Arri Alexa S-35 or Sony’s Venice and FX9. If necessary, you can switch the preprocessor in front of it, which is almost like a small grading app, but can also read in and memorise LUTs for adjustment. We used this with the iPhone material for Apple Log. With the D-Cinelike from the drone, which is not a real log, we also tested whether a contrast enhancement would produce better results. Here, however, the low-contrast original immediately delivered slightly better precision.

In Blender kommt alles mit Hilfe von Python an.
Everything arrives in Blender with the help of Python.

The list of 25 output formats is really impressive. This ranges from long-lost software such as Shake to practically every important compositing and 3D software today. For After Effects there is a Javascript, for Blender, Cinema4D and some other Python scripts that start the respective software and transfer the entire model including camera and video clip – it couldn’t be more convenient. Meshes that you have created and textured in SynthEyes on the basis of trackers can also be transferred. Not only for a moving camera, but also for moving objects. It is even possible to transfer them to photogrammetry software such as Metashape.

Meshes bearbeitet man in der Perspektivdarstellung.
Meshes can be edited in the perspective view.

Features

The support does not end with conventional video, SynthEyes can also handle 360-degree VR or stereo 3D, the latter in particular is explained in detail in the manual. It can limit the search area for trackers with a simple chromakeyer, but can also use imported alpha masks for this purpose. Even zoom shots can be used, but are more critical in terms of tracking precision. If data is available for this, you can enter keyframes in the lens equalisation, e.g. for zooms.

Integration?

At this point, it will be exciting to see how the company will deal with the new acquisition. It was already announced in the interview (see DP 23:06) that SynthEyes will also be available as a plug-in, although no time horizon has yet been set. With Mocha Pro, BorisFX has shown that it is not simply expanding its portfolio, but that the strengths of the new software are being utilised in other products. This has also started here, as the superior lens correction can already be found in Mocha 2024. Currently, “Camera Solve” in Mocha is still a little slower, even considerably so for very long scenes.

Mocha 2024 nutzt bereits die Stärken von Synth­Eyes.
Mocha 2024 already utilises the strengths of SynthEyes.

You can define a ground plane, a coordinate origin and scaling based on a known distance between two tracker points. This already works very well for static scenes with a moving camera; moving or deformable objects are in the works. For problematic scenes, the correction options in SynthEyes are much more comprehensive. In Mocha, you are still dealing with a largely automated process in which you can only change the number of trackers and the blip size.

If a “Power Mesh” was defined before camera tracking, this automatically becomes a 3D mesh. You can export this when set to “Single Mesh”, but so far it is still untextured. The export is not as versatile as in SynthEyes, but in addition to some popular target programmes, the increasingly popular USD format is supported and others are in the works.

Auch für den Export wurde in Mocha 2024 schon gesorgt.
Export has also been taken care of in Mocha 2024.
Importierte USD-Modelle sind in SynthEyes platzierbar, werden aber nicht gerendert, sondern dienen allein der Visualisierung.
Imported USD models can be placed in SynthEyes, but are not rendered and are used solely for visualisation.

More can certainly be expected in this area, as Mocha could teach SynthEyes a thing or two in the creation of tracked patches if the products do not completely merge. For example, masks created directly in SynthEyes are less useful elsewhere as they show hard contours. Their creation is also less elegantly solved than working with Mocha 2024, which has been improved again with “Extrapolate Track”. This means that Mocha tracking no longer stops if a few images need to be skipped. SynthEyes can already deliver trackers to Particle Illusion, but with the new “Inherit Velocity” function in version 2024, it could even add the initial movement to the particles.

In our extensive tests, we didn’t find a single scene that couldn’t be tracked with a little fine-tuning, and there weren’t even any crashes. The annual licence now costs 295 US dollars, for shorter projects the programme is available for 49 US dollars per month, and there is also a permanent licence for 595 US dollars. For comparison: PFTrack costs 1,125.00 per year, 3DEqualiser 65.00 for one week. In addition to the tutorials already mentioned at Boris FX Learn, to which more will surely be added, we can warmly recommend Matt Merkovich at Track VFX.

Comment

With SynthEyes, Boris FX has once again acquired a little piece of cream that may eventually merge with Mocha Pro. It comes with the usual exemplary product documentation including bug tracking and workarounds. Experienced SynthEyes artists would certainly be happy if a “Classic” version were to be retained in the event of any further development of the GUI, as is the case with Mocha.

]]>
DIGITAL PRODUCTION 144243
Halation and diffusion – On the hunt for the ultimate film look https://digitalproduction.com/2024/01/30/halation-and-diffusion-on-the-hunt-for-the-ultimate-film-look/ Tue, 30 Jan 2024 16:49:00 +0000 https://digitalproduction.com/?p=149404 What do you do when digital cameras and modern lenses are too good? It's simple: you use software to simulate the technical weaknesses of analogue film or the filters that experienced cameramen use in such cases. We took a closer look at "Halation" in DaVinci Resolve (DR for short) and FilmConvert Nitrate. Plus "Scatter", which digitally replicates conventional diffusion filters]]>

The term is derived from the word “halo”, which in itself means “halo”. In German this would generally be translated as “Überstrahlung”, but in the context of opto-chemical film it is referred to as “Lichthof”. The incident light, or what is left of it, is reflected by the film carrier on the back (usually polyester, formerly cellulose fibre)
Backing (usually polyester, formerly cellulose acetate) and scattered in the process. To minimise the effect, there is even a halation protection layer behind the colour-sensitive film layers. However, this cannot suppress all light emitted by strong, localised reflections or light sources; a certain amount is always reflected back diffusely.

As the last colour layer is sensitised to red, the halo appears primarily reddish. If the light also reaches the middle layer for green, the colour tone shifts towards orange to a warm golden yellow. The halation will hardly become bluish, as there is a yellow filter behind the first, blue-sensitive layer, which suppresses blue to protect the deeper layers. In a more subtle way, however, parts of the incident light can be scattered in the respective film layer itself and contribute to halation. These forms of light scattering do not exist with digital sensors.

This is what the halation would look like without the image.

Areas of application

This is what the above scene looks like with halation from FilmConvert Nitrate.

So why simulate the technical problem called “halation”? One reason could be a “time piece”, i.e. a film plot that takes place at the time of analogue film and should look as if it was recorded at that time. Another example would be digitally recorded material that is to be combined with archive material. But there are also good reasons to use it for contemporary material. If you allow the diffusion to affect the entire image in a weaker form, it can soften contrasts and create a dreamy, romanticised look – more on this in the other filters.

Depending on the camera, digital sensors have the unpleasant characteristic of showing hard clipping when overexposed. The best cameras with a high dynamic range try to conceal this with a very flat transition into the highlights. But at some point, overexposure always reaches the point where only an area of sterile white or a single colour (the opposite colour to the one already clipped) is displayed.

Film reacts more beautifully in this respect, as its sensitivity flattens out more and more in the vicinity of overexposure for purely physical reasons. The scattering then contributes to not producing a sharply defined, bright spot. Simulated halation in post-production can also make sharply defined highlights look more pleasing in some cases. But if, in extreme cases, these are even surrounded by artefacts, as in our example from an iPhone with its unsolicited interventions, you have to resort to more drastic means such as Keys and Power Windows.

A sunset can look quite banal when recorded digitally.

On-board tools or plug-ins

Halation from DaVinci Resolve can beautify it.

You have always been able to recreate Halation yourself with a clever combination of parallel nodes in colour grading. You can certainly find some suitable power grades to download by googling. But I recommend the tutorial in the German-language “colour channel” by Tim Kersken(is.gd/tim_halation), which is also worth watching. He not only explains the phenomenon in more detail, but also shows step by step how to achieve the corresponding result digitally. There are (as is so often the case in DR) other ways to achieve your goal, but he makes a very important point here: you should temporarily work with linear light when building your own.

In the more recent versions of DaVinci Resolve, Halation is also available as OFX, but only in the paid studio version. The free version is enough to try it out, then there is just a watermark over it
Watermark over it. In addition, FilmConvert from New Zealand now also offers Halation as a supplement to FilmConvert Nitrate (see DP 2002). In any case, the effect should only be used very discreetly – why else would film have a halation protection layer? This is why we can hardly make very subtle effects visible here, especially in the print medium, and have usually exaggerated them somewhat.

Halation hardly helps with artefacts from a mobile phone.

DaVinci Resolve vs. FilmConvert

We are therefore interested in the operation and rendering speed, the rest you should try out with the test versions on your own motifs. But where do you put the in-house effect in grading? Logically, it should come right after the Color Space Transform (CST), or in the case of automatic colour management at the beginning, as it comes from the camera in the case of film. But Darren Mostyn rightly recommends
rightly recommends placing it at the end so as not to influence other processes such as keying(is.gd/darren_halation). You should use a wide colour space for this, we have set this to “Timeline” because we have it set to DaVinci Wide Gamut.

Darren also introduces the halation of Dehancer Pro in the video, whereby the name “Dehancer” is charmingly ironic, almost the opposite of “to enhance”, which would mean improve in English. You can also purchase Dehancer for shorter periods of time if you only need its features for one project. FilmConvert including Halation, on the other hand, is somewhat cheaper as a permanent licence.

FilmConvert has very similar settings to DR, sometimes under a different name. The main difference here is that FilmConvert has its own colour management and should therefore be adjusted to your camera. You no longer even have to download the necessary camera profiles separately from the Internet, but can do this directly from the filter interface. You may miss the specific setting of the blue channel here, but as already described above, this does not actually contribute to halation.

An effect filter with correspondingly extensive settings makes it easier.
FilmConvert also simulates a classic film type.
The additional Highlight Colour Recovery can reduce clipping.

There is also a choice between speed and precision, so we have compared the rendering. FilmConvert is 60 per cent faster when set to “Performance” and still 20 per cent faster when set to “Balanced”; it only takes a little longer at optimum quality. Depending on the scene, the differences are only noticeable with a lot of pixel peeping. Unlike DR, it hardly slows down at all with additional film emulation and film grain. The film grain can also be realistically adjusted according to brightness ranges, which would be much more complicated in DR.

Scatter

This series of filters(videovillage.com/scatter/) simulates the effect of numerous diffusion filters used by professionals in front of the camera to tame overly ‘snappy’ lenses and the merciless sharpness and contrast of digital cameras. This is nothing new: even in the early days, cameramen experimented with Vaseline on a glass plate or tightly stretched nylon stockings in front of the lenses. Especially after emulsions and lenses became so good that they no longer portrayed the delicate skin of the leading actress in a flattering way.

There were even lenses in photography that enabled targeted undercorrection of spherical aberration. The 85 mm f 2.8 Varisoft from Minolta was a pioneering achievement. It can be infinitely adjusted from full sharpness, where it is very well corrected, to a very dreamy look. The image is not simply blurred, but the diffusion is softly superimposed on the sharp basic image. The successor was the Minolta AF 2.8/100 mm Soft Focus for cameras with autofocus from Minolta and Sony.

The Minolta 85 mm f 2.8 Varisoft is a special portrait lens from the late seventies.

Since the analogue era, manufacturers such as Tiffen have been offering entire series of corresponding filters with graduated intensity. This is necessary in order to adjust the diffusion specifically to the lighting, focal length and aperture, as all these factors influence its effect. But it’s not as cheap as a woman’s stocking. Among the most popular for digital cameras are Pro-Mist and Black Pro-Mist. However, their use requires a lot of experience on the part of the camera operator, because if you get the intensity of the filter wrong, it’s like putting salt in soup: you can’t get it out again.

This problem is solved by the filters in Scatter, which are carefully modelled on physical reality. When a colourist like Eric Weidt is not too shy to use Scatter at the suggestion of the cinematographer for David Fincher’s “The Killer”, that says a lot. In post, such filters can be perfectly adjusted to the wishes of the camera and director under optimal viewing conditions (a QLED wall in Weidt’s case) instead of being fixed during the shoot. A permanent licence for Scatter is not cheap, but this applies even more to complete physical filter sets. However, Scatter can also be licensed for shorter periods on a project-specific basis.

Here too, we can only recommend using the test period offered with your own recordings. The manufacturer quite rightly advertises multiple advantages: no loss of light, no unwanted reflections or spots in the bokeh. In addition, the filters are adjustable, including animation of the parameters, can be masked and can be applied to CGI in exactly the same look. What’s more, the image is not affected during the shooting process if it is required in parallel for editing with VFX. Scatter runs on all platforms in DR and does not even require its studio version on MacOS or Windows (only with Nvidia).

It is important for the realistic effect that you use the “Scene-Referred” filters. In itself, this should be logical, as the originals are also aimed directly at the scene. In principle, they also work behind brightness or contrast changes or tone mapping, but then they are no longer physically correct. The filter profiles were calibrated for a focal length of 50 mm on S-35, i.e. for a typical portrait lens (like 80 mm on 35 mm). Other focal lengths can be set, but are only calculated approximately.

The image from FilmConvert with the film simulation alone.
And here with additional halation.

More information can be found in the freely accessible online manual at videovillage.com. The filters are technically quite demanding, rendering BRAW material in UHD with additional Highlight Colour Recovery to reduce overexposed highlights required around 2.6 times the output without the filters. Real-time playback requires a reasonably powerful computer. But the even more realistic fog that DR can deliver with Depth Map and animated Fast Noise requires much more processing power than “Fog” or “Smoque” in Scatter.

Comment

When it comes to real filters, cameramen basically only need ND, polarisers and, on some cameras, IR cut filters. Only these cannot be adequately simulated, everything else today is just as good in terms of quality and more flexible in post-production.

This is what our trendy café looks like without filtering.

Even the light scattering from the evening sky at the top right is simulated as “Black Pro-Mist” of strength 1/2.
]]>
149404