Introduction

I had this project where I needed to stream the video+audio to a backend RTMP Server, and also needed to do a local recording, along with some ML Model inferencing locally.

The usual Flutter Camera package: pub.dev/packages/camera does not provide a way to do all three of the above mentioned tasks.

So what next? I've explored different camera library packages for flutter, namely flutter_better_camera, rtmp_publisher and camera_awesome

And neither of them do all three.

The crazy thing is none of the packages allow simultaneous recording and image byte stream to flutter end.

Solutions

So there were a few solutions I tried so solve this problem

1. Do everything using just the Image Byte Stream 😓

   At the end of day, I need to do atleast video+audio+inference and send it to the backend server. So what if I get the audio stream (using the audio_streams package) and image stream (using the camera package)?

   I'll simply use the image stream, do my inferencing, combine it with the audio and send it over to the backend using a socket connection!

   There are a few problems with this approach

   • Audio and Video Sync Issues
   • The video I stream to the backend will also need to be played, and I've sending audio and image frame by frame, there has to be muxer involved. I couldn't find a good way of muxing in flutter
   • The audio is not continuous, so to the end user it wouldn't make sense of what's happening.
   • The FPS was not that good
   • Also how would I record it to a file without using a media container? This involves a lot of processing on flutter, and I couldn't find a library for this.

2. Modify the flutter's camera library itself 👿

   This was my last resort, where I would go through the entire source code of flutter's camera module and modify it such that it can do all my task. I'll need to write platform specific code (Android only) with Java/Kotlin (which i hate). It would be a nightmare, but this was the only solution for me.

   For Android there's this RTMP/RTSP library: rtmp-rtsp-stream-client-java which can stream the camera video to backend, and also has an API to record the video!

   Add a method call handler for my api

             "startVideoStreamingAndImageStream" -> {
                Log.i("SatyajitCustomStuff", call.arguments.toString())
                var bitrate: Int? = null
                if (call.hasArgument("bitrate")) {
                    bitrate = call.argument("bitrate")
                }
   
                camera!!.startVideoStreamingAndImageStream(
                    call.argument("url"),
                    bitrate,
                    imageStreamChannel,
                    result)
            }
   

   And add the handler in Camera.kt

    fun startVideoStreamingAndImageStream(url: String?, bitrate: Int?, imageStreamChannel: EventChannel, result: MethodChannel.Result) {

        if (url == null) {
            result.error("fileExists", "Must specify a url.", null)
            return
        }
        try {
            // Setup the rtmp session
            if (rtmpCamera == null) {
                currentRetries = 0
                prepareCameraForStream(streamingProfile.videoFrameRate, bitrate)

                // Start capturing from the camera.
                createCaptureSession(
                    CameraDevice.TEMPLATE_RECORD,
                    Runnable { rtmpCamera!!.startStream(url) },
                    rtmpCamera!!.inputSurface,
                    imageStreamReader!!.surface
                )

//                createCaptureSession(
//                    CameraDevice.TEMPLATE_RECORD,
//                    Runnable { rtmpCamera!!.startStream(url) },
//                    imageStreamReader!!.surface)

            } else {
                rtmpCamera!!.startStream(url)
            }

            imageStreamChannel.setStreamHandler(
                object : EventChannel.StreamHandler {
                    override fun onListen(o: Any?, imageStreamSink: EventSink) {
                        setImageStreamImageAvailableListener(imageStreamSink)
                    }

                    override fun onCancel(o: Any?) {
                        imageStreamReader!!.setOnImageAvailableListener(null, null)
                    }
                })

            result.success(null)
        } catch (e: CameraAccessException) {
            result.error("videoStreamingFailed", e.message, null)
        } catch (e: IOException) {
            result.error("videoStreamingFailed", e.message, null)
        }
    }

The important part to notice here is that we pass in 2 input surfaces, rtmpCamera!!.inputSurface and imageStreamReader!!.surface, in which the latter is used to stream images to the flutter end.

Also in camera.dart I added the invoker for above

  Future<void> startVideoStreamingAndImageStream(
      String url, {onLatestImageAvailable? onAvailable,
      int bitrate = 1200 * 1024, bool? androidUseOpenGL}) async {
    if (!value.isInitialized! || _isDisposed) {
      throw CameraException(
        'Uninitialized CameraController',
        'startVideoStreaming was called on uninitialized CameraController',
      );
    }
    if (value.isRecordingVideo!) {
      throw CameraException(
        'A video recording is already started.',
        'startVideoStreaming was called when a recording is already started.',
      );
    }
    if (value.isStreamingVideoRtmp!) {
      throw CameraException(
        'A video streaming is already started.',
        'startVideoStreaming was called when a recording is already started.',
      );
    }
    if (value.isStreamingImages!) {
      throw CameraException(
        'A camera has started streaming images.',
        'startVideoStreaming was called while a camera was streaming images.',
      );
    }

    try {
      await _channel.invokeMethod<void>(
          'startVideoStreamingAndImageStream', <String, dynamic>{
        'textureId': _textureId,
        'url': url,
        'bitrate': bitrate,
      });
      value =
          value.copyWith(isStreamingVideoRtmp: true, isStreamingPaused: false);
    } on PlatformException catch (e) {
      throw CameraException(e.code, e.message);
    }

    const EventChannel cameraEventChannel =
        EventChannel('plugins.flutter.io/camera_with_rtmp/imageStream');
    _imageStreamSubscription =
        cameraEventChannel.receiveBroadcastStream().listen(
      (dynamic imageData) {
        if (onAvailable != null)
          onAvailable(CameraImage._fromPlatformData(imageData));
      },
    );
  }

Now finally i need to record the video too

    fun startVideoRecordingAndStreaming(filePath: String, url: String?, bitrate: Int?, result: MethodChannel.Result) {
        if (File(filePath).exists()) {
            result.error("fileExists", "File at path '$filePath' already exists.", null)
            return
        }
        if (url == null) {
            result.error("fileExists", "Must specify a url.", null)
            return
        }
        try {
            // Setup the rtmp session
            currentRetries = 0
            prepareCameraForRecordAndStream(streamingProfile.videoFrameRate, bitrate)

            createCaptureSession(
                    CameraDevice.TEMPLATE_RECORD,
                    Runnable {
                        rtmpCamera!!.startStream(url)
                        rtmpCamera!!.startRecord(filePath)
                    },
                    rtmpCamera!!.inputSurface
            )
            result.success(null)
        } catch (e: CameraAccessException) {
            result.error("videoRecordingFailed", e.message, null)
        } catch (e: IOException) {
            result.error("videoRecordingFailed", e.message, null)
        }
    }

And that's it! I was able to stream video+audio to RTMP backend + get a image byte stream in flutter end and also record video !

The inferencing was done using TfLite plugin.

Notes

• Use the following code to convert the YUV Image to JPEG/PNG

  class ImageUtils {
  /// Converts a [CameraImage] in YUV420 format to [imageLib.Image] in RGB format
  static imageLib.Image? convertCameraImage(CameraImage cameraImage) {
    if (cameraImage.format.group == ImageFormatGroup.yuv420) {
      return convertYUV420ToImage(cameraImage);
    } else if (cameraImage.format.group == ImageFormatGroup.bgra8888) {
      return convertBGRA8888ToImage(cameraImage);
    } else {
      return null;
    }
  }
  
  /// Converts a [CameraImage] in BGRA888 format to [imageLib.Image] in RGB format
  static imageLib.Image convertBGRA8888ToImage(CameraImage cameraImage) {
    imageLib.Image img = imageLib.Image.fromBytes(cameraImage.planes[0].width!,
        cameraImage.planes[0].height!, cameraImage.planes[0].bytes!,
        format: imageLib.Format.bgra);
    return img;
  }
  
  /// Converts a [CameraImage] in YUV420 format to [imageLib.Image] in RGB format
  static imageLib.Image convertYUV420ToImage(CameraImage cameraImage) {
    final int width = cameraImage.width!;
    final int height = cameraImage.height!;
  
    final int uvRowStride = cameraImage.planes[1].bytesPerRow!;
    final int uvPixelStride = cameraImage.planes[1].bytesPerPixel!;
  
    final image = imageLib.Image(width, height);
  
    for (int w = 0; w < width; w++) {
      for (int h = 0; h < height; h++) {
        final int uvIndex =
            uvPixelStride * (w / 2).floor() + uvRowStride * (h / 2).floor();
        final int index = h * width + w;
  
        final y = cameraImage.planes[0].bytes![index];
        final u = cameraImage.planes[1].bytes![uvIndex];
        final v = cameraImage.planes[2].bytes![uvIndex];
  
        image.data[index] = ImageUtils.yuv2rgb(y, u, v);
      }
    }
    return image;
  }
  
  /// Convert a single YUV pixel to RGB
  static int yuv2rgb(int y, int u, int v) {
    // Convert yuv pixel to rgb
    int r = (y + v * 1436 / 1024 - 179).round();
    int g = (y - u * 46549 / 131072 + 44 - v * 93604 / 131072 + 91).round();
    int b = (y + u * 1814 / 1024 - 227).round();
  
    // Clipping RGB values to be inside boundaries [ 0 , 255 ]
    r = r.clamp(0, 255);
    g = g.clamp(0, 255);
    b = b.clamp(0, 255);
  
    return 0xff000000 |
    ((b << 16) & 0xff0000) |
    ((g << 8) & 0xff00) |
    (r & 0xff);
  }
  
  static void saveImage(imageLib.Image image, [int i = 0]) async {
    List<int> jpeg = imageLib.JpegEncoder().encodeImage(image);
    final appDir = await getTemporaryDirectory();
    final appPath = appDir.path;
    final fileOnDevice = File('$appPath/out$i.jpg');
    await fileOnDevice.writeAsBytes(jpeg, flush: true);
    print('Saved $appPath/out$i.jpg');
  }
  }
  

  • There was the weird bug for me, I was trying to print the base64 encoded image onto console and me not realizing that there is a line limit of 1024 bytes for console write. I was thinking my image is corrupted, so I had to write the base64 to a file and use that to verify the image.