[londons_explore]

All modern smartphones have masses of proprietary stuff to turn gigabytes of raw data from the sensor into a beautiful photo.

If you want an opensource phone, you end up with crappy photos unless you have a camera team of 500 people to rewrite that stuff, and a big team of lawyers to deal with all the patent licensing you're going to have to do for the techniques you use.

[morsch]

I'm not sure that it's truly all down to software, but as far as it is, you don't have start at zero -- there are excellent, competitive raw photo processors available as free software, like darktable and RawTherapee. Still a lot of work, though, no doubt.

[londons_explore]

heh - no.

Camera stacks on modern phones process many raw frames to make a single photo - sometimes as many as 100.

Collecting those frames is done in a second or so before and after the user clicks the shutter button. Thats a data rate of ~15 Gigabits - far faster than most phones can write to storage - and anyway would you be happy for each photo to take a few gigabytes of storage space until it was processed?

Therefore, the raw data needs to be processed (or at least preprocessed) in realtime. You can't store it and do it later.

And to do that, you typically need custom silicon - all the big phone manufacturers have ASICs dedicated to image processing.

And to make the trouble even worse, you often need to use data from some of the raw frames to adjust camera parameters for the next frame - for example adjusting amplifier gains. See patent US9196027B2 for an example of the sort of thing that is done 'in the loop'.

[londons_explore]

If you wanted to do this 'opensource', your best bet is to try and get the GPU to do the work. You'd need hardware/HAL support for streaming raw data at ~15Gbits to GPU accessible memory, and after that you can probably do most of the rest using software without hardware/kernel involvement.

At a minimum, the GPU needs to be able to generate image scale pyramids, align them using optical flow, have a sensor noise model to detect misalignment, and take a running-mean of every non-misaligned pixel in each frame seen. This technique only needs enough ram for a handful of frames.

That should get you a good chunk of the way to a decent image.

As usual, the actual math is quite simple in python, but when you need it to run at 15 Gbits you'll be spending a really long time optimising assembly code for whatever GPU you're using...

[verall]

The SoC in the fairphone has an ISP, they can just use that.

[londons_explore]

The ISP is locked down and can only run qualcomm-code. Perhaps with the right licensing agreement they'd let you run your own code on it.