Skip to content

Commit

Permalink
* Provide new FFmpegFrameGrabber(InputStream) and `FFmpegFrameReco…
Browse files Browse the repository at this point in the history
…rder(OutputStream)` constructors (issue #95)

 * Make `FrameFilter`, `FrameGrabber`, and `FrameRecorder` implement `Closeable` to let us try-with-resources
  • Loading branch information
saudet committed Dec 27, 2016
1 parent ef71876 commit 16050a2
Show file tree
Hide file tree
Showing 7 changed files with 271 additions and 10 deletions.
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@

* Provide new `FFmpegFrameGrabber(InputStream)` and `FFmpegFrameRecorder(OutputStream)` constructors ([issue #95](https://github.com/bytedeco/javacv/issues/95))
* Make `FrameFilter`, `FrameGrabber`, and `FrameRecorder` implement `Closeable` to let us try-with-resources
* Fix potential crash when recording audio with `FFmpegFrameRecorder`
* Add `OpenKinect2FrameGrabber` to capture images with libfreenect2 ([pull #584](https://github.com/bytedeco/javacv/pull/584))
* Add `OpenKinectFrameGrabber.grabIR()` and stabilize `RealSenseFrameGrabber` ([pull #585](https://github.com/bytedeco/javacv/pull/585))
Expand Down
102 changes: 102 additions & 0 deletions platform/src/test/java/org/bytedeco/javacv/FrameGrabberTest.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
/*
* Copyright (C) 2016 Samuel Audet
*
* Licensed either under the Apache License, Version 2.0, or (at your option)
* under the terms of the GNU General Public License as published by
* the Free Software Foundation (subject to the "Classpath" exception),
* either version 2, or any later version (collectively, the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
* http://www.gnu.org/licenses/
* http://www.gnu.org/software/classpath/license.html
*
* or as provided in the LICENSE.txt file that accompanied this code.
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.bytedeco.javacv;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import org.bytedeco.javacpp.Loader;
import org.bytedeco.javacpp.indexer.UByteIndexer;
import org.junit.Test;

import static org.bytedeco.javacpp.avutil.*;
import static org.junit.Assert.*;

/**
* Test cases for FrameGrabber classes. Also uses other classes from JavaCV.
*
* @author Samuel Audet
*/
public class FrameGrabberTest {

@Test public void testFFmpegFrameGrabber() {
System.out.println("FFmpegFrameGrabber");

File tempFile = new File(Loader.getTempDir(), "test.mkv");
try {
FFmpegFrameRecorder recorder = new FFmpegFrameRecorder(new FileOutputStream(tempFile), 640, 480);
recorder.setFormat("matroska"); // mp4 doesn't support streaming
recorder.setPixelFormat(AV_PIX_FMT_BGR24);
recorder.setVideoCodecName("libx264rgb");
recorder.setVideoQuality(0); // lossless
recorder.start();

Frame[] frames = new Frame[1000];
for (int n = 0; n < frames.length; n++) {
Frame frame = new Frame(640, 480, Frame.DEPTH_UBYTE, 3);
UByteIndexer frameIdx = frame.createIndexer();
for (int i = 0; i < frameIdx.rows(); i++) {
for (int j = 0; j < frameIdx.cols(); j++) {
for (int k = 0; k < frameIdx.channels(); k++) {
frameIdx.put(i, j, k, n + i + j + k);
}
}
}
recorder.record(frame);
frames[n] = frame;
}
recorder.stop();
recorder.release();

FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(new FileInputStream(tempFile));
grabber.start();

for (int n = 0; n < frames.length; n++) {
Frame frame = frames[n];
Frame frame2 = grabber.grab();
assertEquals(frame.imageWidth, frame2.imageWidth);
assertEquals(frame.imageHeight, frame2.imageHeight);
assertEquals(frame.imageChannels, frame2.imageChannels);

UByteIndexer frameIdx = frame.createIndexer();
UByteIndexer frame2Idx = frame2.createIndexer();
for (int i = 0; i < frameIdx.rows(); i++) {
for (int j = 0; j < frameIdx.cols(); j++) {
for (int k = 0; k < frameIdx.channels(); k++) {
int b = frameIdx.get(i, j, k);
assertEquals(b, frame2Idx.get(i, j, k));
}
}
}
}
assertEquals(grabber.grab(), null);
grabber.stop();
grabber.release();
} catch (Exception e) {
fail("Exception should not have been thrown: " + e);
} finally {
tempFile.delete();
}
}

}
72 changes: 70 additions & 2 deletions src/main/java/org/bytedeco/javacv/FFmpegFrameGrabber.java
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright (C) 2009-2015 Samuel Audet
* Copyright (C) 2009-2016 Samuel Audet
*
* Licensed either under the Apache License, Version 2.0, or (at your option)
* under the terms of the GNU General Public License as published by
Expand Down Expand Up @@ -50,13 +50,17 @@
package org.bytedeco.javacv;

import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.nio.Buffer;
import java.nio.ByteBuffer;
import java.util.HashMap;
import java.util.Map.Entry;
import org.bytedeco.javacpp.BytePointer;
import org.bytedeco.javacpp.DoublePointer;
import org.bytedeco.javacpp.IntPointer;
import org.bytedeco.javacpp.Loader;
import org.bytedeco.javacpp.Pointer;
import org.bytedeco.javacpp.PointerPointer;

import static org.bytedeco.javacpp.avcodec.*;
Expand Down Expand Up @@ -120,6 +124,9 @@ public FFmpegFrameGrabber(File file) {
public FFmpegFrameGrabber(String filename) {
this.filename = filename;
}
public FFmpegFrameGrabber(InputStream inputStream) {
this.inputStream = inputStream;
}
public void release() throws Exception {
synchronized (org.bytedeco.javacpp.avcodec.class) {
releaseUnsafe();
Expand Down Expand Up @@ -170,7 +177,7 @@ void releaseUnsafe() throws Exception {
}

// Close the video file
if (oc != null && !oc.isNull()) {
if (inputStream == null && oc != null && !oc.isNull()) {
avformat_close_input(oc);
oc = null;
}
Expand All @@ -185,12 +192,62 @@ void releaseUnsafe() throws Exception {
frame = null;
timestamp = 0;
frameNumber = 0;
filename = null;

if (inputStream != null) {
try {
inputStream.close();
} catch (IOException ex) {
throw new Exception("Error on InputStream.close(): ", ex);
} finally {
inputStream = null;
inputStreams.remove(oc);
if (avio != null) {
if (avio.buffer() != null) {
av_free(avio.buffer());
avio.buffer(null);
}
av_free(avio);
avio = null;
}
if (oc != null) {
avformat_free_context(oc);
oc = null;
}
}
}
}
@Override protected void finalize() throws Throwable {
super.finalize();
release();
}

static HashMap<Pointer,InputStream> inputStreams = new HashMap<Pointer,InputStream>();

static class ReadCallback extends Read_packet_Pointer_BytePointer_int {
@Override public int call(Pointer opaque, BytePointer buf, int buf_size) {
try {
byte[] b = new byte[buf_size];
InputStream is = inputStreams.get(opaque);
int size = is.read(b, 0, buf_size);
if (size < 0) {
return 0;
} else {
buf.put(b, 0, size);
return size;
}
}
catch (Throwable t) {
System.err.println("Error on InputStream.read(): " + t);
return -1;
}
}
}

static ReadCallback readCallback;

private InputStream inputStream;
private AVIOContext avio;
private String filename;
private AVFormatContext oc;
private AVStream video_st, audio_st;
Expand Down Expand Up @@ -431,6 +488,17 @@ void startUnsafe() throws Exception {
for (Entry<String, String> e : this.options.entrySet()) {
av_dict_set(options, e.getKey(), e.getValue(), 0);
}
if (inputStream != null) {
if (readCallback == null) {
readCallback = new ReadCallback();
}
oc = avformat_alloc_context();
avio = avio_alloc_context(new BytePointer(av_malloc(4096)), 4096, 0, oc, readCallback, null, null);
oc.pb(avio);

filename = inputStream.toString();
inputStreams.put(oc, inputStream);
}
if ((ret = avformat_open_input(oc, filename, f, options)) < 0) {
av_dict_set(options, "pixel_format", null, 0);
if ((ret = avformat_open_input(oc, filename, f, options)) < 0) {
Expand Down
71 changes: 69 additions & 2 deletions src/main/java/org/bytedeco/javacv/FFmpegFrameRecorder.java
Original file line number Diff line number Diff line change
Expand Up @@ -51,13 +51,16 @@
package org.bytedeco.javacv;

import java.io.File;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.Buffer;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.DoubleBuffer;
import java.nio.FloatBuffer;
import java.nio.IntBuffer;
import java.nio.ShortBuffer;
import java.util.HashMap;
import java.util.Map.Entry;
import org.bytedeco.javacpp.BytePointer;
import org.bytedeco.javacpp.DoublePointer;
Expand Down Expand Up @@ -149,6 +152,18 @@ public FFmpegFrameRecorder(String filename, int imageWidth, int imageHeight, int
this.video_pkt = new AVPacket();
this.audio_pkt = new AVPacket();
}
public FFmpegFrameRecorder(OutputStream outputStream, int audioChannels) {
this(outputStream.toString(), audioChannels);
this.outputStream = outputStream;
}
public FFmpegFrameRecorder(OutputStream outputStream, int imageWidth, int imageHeight) {
this(outputStream.toString(), imageWidth, imageHeight);
this.outputStream = outputStream;
}
public FFmpegFrameRecorder(OutputStream outputStream, int imageWidth, int imageHeight, int audioChannels) {
this(outputStream.toString(), imageWidth, imageHeight, audioChannels);
this.outputStream = outputStream;
}
public void release() throws Exception {
synchronized (org.bytedeco.javacpp.avcodec.class) {
releaseUnsafe();
Expand Down Expand Up @@ -204,9 +219,10 @@ void releaseUnsafe() throws Exception {
}
video_st = null;
audio_st = null;
filename = null;

if (oc != null && !oc.isNull()) {
if ((oformat.flags() & AVFMT_NOFILE) == 0) {
if (outputStream == null && (oformat.flags() & AVFMT_NOFILE) == 0) {
/* close the output file */
avio_close(oc.pb());
}
Expand Down Expand Up @@ -238,12 +254,53 @@ void releaseUnsafe() throws Exception {
swr_free(samples_convert_ctx);
samples_convert_ctx = null;
}

if (outputStream != null) {
try {
outputStream.close();
} catch (IOException ex) {
throw new Exception("Error on OutputStream.close(): ", ex);
} finally {
outputStream = null;
outputStreams.remove(oc);
if (avio != null) {
if (avio.buffer() != null) {
av_free(avio.buffer());
avio.buffer(null);
}
av_free(avio);
avio = null;
}
}
}
}
@Override protected void finalize() throws Throwable {
super.finalize();
release();
}

static HashMap<Pointer,OutputStream> outputStreams = new HashMap<Pointer,OutputStream>();

static class WriteCallback extends Write_packet_Pointer_BytePointer_int {
@Override public int call(Pointer opaque, BytePointer buf, int buf_size) {
try {
byte[] b = new byte[buf_size];
OutputStream os = outputStreams.get(opaque);
buf.get(b, 0, buf_size);
os.write(b, 0, buf_size);
return buf_size;
}
catch (Throwable t) {
System.err.println("Error on OutputStream.write(): " + t);
return -1;
}
}
}

static WriteCallback writeCallback;

private OutputStream outputStream;
private AVIOContext avio;
private String filename;
private AVFrame picture, tmp_picture;
private BytePointer picture_buf;
Expand Down Expand Up @@ -329,6 +386,16 @@ void startUnsafe() throws Exception {
throw new Exception("avformat_alloc_context2() error:\tCould not allocate format context");
}

if (outputStream != null) {
if (writeCallback == null) {
writeCallback = new WriteCallback();
}
avio = avio_alloc_context(new BytePointer(av_malloc(4096)), 4096, 1, oc, null, writeCallback, null);
oc.pb(avio);

filename = outputStream.toString();
outputStreams.put(oc, outputStream);
}
oc.oformat(oformat);
oc.filename().putString(filename);

Expand Down Expand Up @@ -702,7 +769,7 @@ void startUnsafe() throws Exception {
}

/* open the output file, if needed */
if ((oformat.flags() & AVFMT_NOFILE) == 0) {
if (outputStream == null && (oformat.flags() & AVFMT_NOFILE) == 0) {
AVIOContext pb = new AVIOContext(null);
if ((ret = avio_open(pb, filename, AVIO_FLAG_WRITE)) < 0) {
release();
Expand Down
Loading

11 comments on commit 16050a2

@Perryoo
Copy link

@Perryoo Perryoo commented on 16050a2 Oct 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Saudet,

I hit the memory leak issue when I implement the video and audio stream with the current feature.
The implementation steps(take video for example) for me:

  1. Create the input and output stream

    PipedOutputStream viPos = new PipedOutputStream();
    PipedInputStream viPis = new PipedInputStream();
    viPos.connect(viPis);

  1. Write the stream to output stream in the main thread

   viPos.write(inbuf, 0, inlen);

  1. Grabber holds the inputstream object and recorder pushes the stream to rtmp with FFmpegFrameRecorder
    The pseudocodes are attached below:

......
FFmpegFrameRecorder recorder = new FFmpegFrameRecorder(rtmpURL, 1280, 720);
recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);  // AAC for audio
recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264); // H264 for video
....set the options for the recorder......
recorder.start();

......
   // a sub-thread thread to process the video frames
     new Thread(()->{
    FFmpegFrameGrabber grabber1 = new FFmpegFrameGrabber(viPis);
            Frame frame1 = null;
            try
            {
                recorder.start();

                while ((frame1 = grabber1.grabImage()) != null )
                {
                    recorder.record(frame1);
                } 
            } catch (Exception e)
            {
                e.printStackTrace();
            } 
        .......
    }.start();

I reproduce the issue via opening the video files.
I want to know whether I lost some codes when calling the related classes/methods or not.

Can you give me some suggestions for the issue?

Thanks in advance.

@saudet
Copy link
Member Author

@saudet saudet commented on 16050a2 Oct 6, 2018 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Perryoo
Copy link

@Perryoo Perryoo commented on 16050a2 Oct 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which method should I call?
recorder.release() or grabber.release().
And when should I call it? I have to hold the inputstream all the time.

@saudet
Copy link
Member Author

@saudet saudet commented on 16050a2 Oct 6, 2018 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Perryoo
Copy link

@Perryoo Perryoo commented on 16050a2 Oct 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Saudet.

The two objects "recorder" and "grabber" need to be enabled all the time to process the streams.
So, it's hard for me to stop and freee the resource for the two objects.
Even if I don't free the resource of the two objects, it should leak the resource of the two objects occupied once only.
Why does it leak more and more?

Need I free the resource of the frame object("frame1" in my codes above) in the "while" loop?
How?

Thanks.

@saudet
Copy link
Member Author

@saudet saudet commented on 16050a2 Oct 6, 2018 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Perryoo
Copy link

@Perryoo Perryoo commented on 16050a2 Oct 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It maybe
I commented the codes above and find that it seems to be fine.

If the usage of the stream for javacv in my first comments is fine, I can investigate more about my codes.

Thanks for your response.

@saudet
Copy link
Member Author

@saudet saudet commented on 16050a2 Oct 6, 2018 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Perryoo
Copy link

@Perryoo Perryoo commented on 16050a2 Oct 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Saudet,
I export the dump file with jmap tool, and analyse the memory for the codes with MAT(Eclipse Memory Analyzer tool).

Its "Leak Hunter" shows that:


One instance of "java.io.BufferedInputStream" loaded by "" occupies 536,870,968 (87.56%) bytes. The instance is referenced by org.bytedeco.javacv.FFmpegFrameGrabber @ 0x838f53f0 , loaded by "sun.misc.Launcher$AppClassLoader @ 0x80a99560". The memory is accumulated in one instance of "byte[]" loaded by "< system class loader >".


It seems to be that the memory is not free in BufferedInputStream object(constructed with PipedInputStream object I set) in FFmpegFrameGrabber.

Can you have a look please?
Or give me some suggestions?

Thanks.

@saudet
Copy link
Member Author

@saudet saudet commented on 16050a2 Oct 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes, that's a limitation of using InputStream with FFmpeg. We basically have to load the whole file in memory: https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/FFmpegFrameGrabber.java#L751
Some formats like Matroska can do better, so if you don't need to seek, you could try to reduce that value.

@Perryoo
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, thanks Saudet.
Let me have a try, and investigate how to fix it better.

Please sign in to comment.