Introduction
AI (Artificial Intelligence) and photo culling is becoming more and more mainstream with different software vendors proposing apps to photographers to ease their photo culling and perform it faster. Of course, there is no silver bullet and besides the fact each apps have strengths and limits as usual, there are critics and some are relevant as already listed in this blog post.
At Camera Futura, the company behind one of these apps (Futura Photo, please read the disclaimer below, I am the founder of this company), the Quality Assurance process requires testing the apps with dozens of thousands of images, from real photoshoots of real photographers. I wanted to calculate in terms of % how many images were automatically discarded and how much relevant it was, in average.
Indeed, there are different metrics but basically, a photographer should not post-process more than typically 5% of the total number of images for a given shoot. This number is arguable, but it means most of the images should be discarded.
So, after the culling done by the apps, how many are already discarded ? How well it will help the photographer ?
Calculation
The calculation has been done on a set of photo shoots (a couple of dozens) being part of this Quality Assurance process, as they are real shoots from real photographers. They are of different kinds of photography:
- Wedding,
- Landscapes,
- Macro,
- Wildlife,
- Travel,
- Sports.
In total, this means the calculation is done based on a 15’000 photos test.
Results
With Future Photo, AI can cull typically 50 to 55% of what the photographer must discard. It can do it with a False Positive rate of less than 10% (Each “false positive”, wrongly detected image, means obliging the photographer to manually override the AI).
Of course, these numbers will vary, depending on the photoshoot, but this is an average which is proving AI photo culling works efficiently: AI can be compared to an assistant capable of preforming more than half of the job and is right almost all the time :
For 100 images, it is right 90 times, and wrong 10 times.
How did I come to these results ? Let’s do some (basic) maths…
Detailed results
(You can jump to the conclusion if you are not interested in the maths)
Exact number of images tested: 14’537
How many must be discarded: 90% of 14’537: 13’083
Total identified by Futura Photo to be discarded: 10’095
Bridge: 13’083-10’095=2’988 : images that are no duplicates of anything, not really missed in terms of technical terms (acutance and exposure) but just not interesting. This part is not accessible to AI unless you start to implement a rule “not interesting picture” which is not trivial at all but not out of reach that being said. Anyway, it exists and must be considered.
AI in Camera Futura can discard 10’095/13’083 * 100 = 77% of images to be discarded, that does not sound bad at all to me.
But reality is different, there are “false positive” (FP: wrongly detected to be discarded, painful) and “false negative” (FN: missed to be detected).
Below the FP rates per rule. I am also adjusting the duplicates FP as some images are wrongly chosen. How much ? Typically, by experience, 10% (1 for 10, meaning 9 are true positive).
FP rate measured: (218*0+493*0.2+0*31+7.6*1’972+7’616*(10+1.2)+207*18.4+320*2.8)/(3’241+7’616) = 9.7 % (we considered the whole set of duplicates of course for this calculation)
FN rate measured: (218*17+493*4.3+0*31+10.5*1’972+7’616*16.7+207*22.2+320*0)/(3241+7’616)=14.5%
This means whereas 77% of images are flagged as “discarded”, it is more: 77-14.5=62.5 % and 9.7% wrongly flagged. For the calculation I will subtract this number as it will require a manual override from the user: 62.5-9.7=52.8 %
This means “a little bit less than half of the photos to be discarded are actually discarded” and on top of them, “a little bit less than 10% has been wrongly discarded”.
Conclusion
Based on the above tests and calculations:
Let’s say you have a photoshoot of 1’000 images:
You will need to discard typically (at least) 900 of them.
AI will find 540, and will be right for 490 of them, wrong for 50 photos.
That’s why it is worth investigating how AI can help you. Furthermore, keep it mind it can do more than just cull the images, but also organize your shoot by detecting panoramas and time lapse members, and align them if needed for the latter.
Disclaimer
I am the founder of Camera Futura, the company behind Futura Photo