Overview

Libmegapixels is a library for accessing V4L2 cameras in Linux. It’s the camera set-up code originally written for Megapixels but split off and improved as a library to make it easier to debug and integrate.

The way you’d normally open a webcam in Linux and capture frames is roughly:

  • Open a /dev/video* file descriptor

  • Run the VIDIOC_QUERYCAP ioctl to check the device has the V4L2_CAP_VIDEO_CAPTURE flag and check which methods of capturing frames are valid.

  • Run the VIDIOC_S_FMT and VIDIOC_G_FMT ioctls a few times to narrow down a format to use.

  • Set-up mmapped video capture on the device and run VIDIOC_STREAMON to start the capture

  • Get frames from the kernel

This works for UVC webcams you usually encounter on Linux devices since these are relatively simple devices.

If you try to use the camera on a modern phone the whole workflow is completely different. Instead of dealing with just a video device you also have to deal with a /dev/media* device for setting up hardware pipelines and then deal with various /dev/v4l-subdev* devices to configure all the nodes in that media pipeline.

One (or more) of the nodes in that media pipeline will be the regular old /dev/video* device again that will provide your application with the frames, but due to the way the pipelines work a lot of the normal auto-configuration things applications do no longer work.

In the new pipelines the video device only deals with getting the video frames into userspace. Due to this the video device does not actually know what formats and resolutions are valid so using the old ioctls to query this information is useless.

One of the v4l-subdev devices will be representing the sensor in the device, this does know what modes are available but to know that mode will work you need to make sure all the nodes in the pipeline can run at that mode and all these nodes need to be manually configured. But in reality it’s even more complicated becaues this only describes what resolutions and modes the drivers for these components support and it does not account for limitations in the actual hardware like not all the MIPI lanes being connected between the sensor and the SoC limiting the possible bandwidth, or even more basic bandwidth limitations due to the length of PCB traces.

Using libmegapixels

The solution for this mess in Megapixels which is now transferred to a library is using config files that provide the pipeline information. This hardcodes a series of known working modes and also provides the metadata for producing high quality pictuers from the data coming from the pipeline.

The general use of libmegapixels is this:

#include <libmegapixels.h>


// Create the empty device config object
libmegapixels_devconfig *config = {0};
libmegapixels_init(&config);

// Find the config file path for this device
char configpath[PATH_MAX];
int ret = libmegapixels_find_config(configpath);

// Load the config
libmegapixels_load_file(config, configpath);

// Additionally load UVC cameras with autodetection
libmegapixels_load_uvc(config);

printf("Found %d cameras\n", config->count);

// Open the first camera
libmegapixels_camera *camera = config->cameras[camera_id];
libmegapixels_open(camera);

printf("Camera has %d modes\n", camera->num_modes);

// Select the first mode
libmegapixels_mode *mode = camera->modes[mode_idx];
struct v4l2_format format = {0};
libmegapixels_select_mode(camera, mode, &format);

// Now you can do regular V4L2 things on the camera with the
// supplied FDs. video_fd for the /dev/video device.
ioctl(camera->video_fd, VIDIOC_QUERYCAP, &cap)

Libmegapixels replaces most of the original V4L2 setup code and provides a filled in struct v4l2_format to get all the exact mode information.

To pick a camera and mode you’d iterate over the found cameras and then over the mode structs to find something that matches the needs of the application.

The camera struct provides the FDs for the various devices you might need:

  • The video_fd field has the FD for /dev/video* for actually getting the frames.

  • The sensor_fd field is for the /dev/v4l-subdev* device that represents the sensor. This is for setting camera V4L2 controls while capturing.

  • The media_fd field is for the /dev/media* device that has the pipeline for the current camera. This is mostly for the libmegapixels internals.

The config files

The libmegapixels library ships with configuration files for some devices. The device configuration can be stored in multiple locations so it can be overridden by packagers and end users.

/usr/share/megapixels/config/%model.conf

This is the location where the config files from packages are stored. This is also where the build script from libmegapixels will place the internal config files here.

/etc/megapixels/config/%model.conf

This is the place for extra and/or override config files

$cwd/config/%model.conf

The config is loaded from the current working directory to make testing and debugging the code easier and to run it from the root of the git repository.

The %model argument in the path is referring to the device-tree compatible names of the device. This can be found in /proc/device-tree/compatible. This stores the name of the hardware seperated by null-bytes in decreasing specificity order. Libmegapixels will check all of them in order for every location above.