Informikon

I have modified the Picture-In-a-Picture sample and added a custom image compositor object to implement a version of the Cube sample from the SDK in C#. Once again, the code can be found in a zip file.

I have described the steps to produce the CustomICTypeLib.cs in the "readme.txt" file of the download. So I'll concentrate on the implementation of the IVMR9ImageCompositor interface in C#. The documentation on this interface (beside the Reference section of the DirectShow documentation) consists of the following paragraph: (taken from the section "VMR Filter Components")

Image Compositor: The Image Compositor is a COM object that performs the actual blending of the input streams onto a single DirectDraw or Direct3D surface provided by the allocator-presenter. The VMR provides a default image compositor that enables applications to perform 2-D alpha-blending effects. Applications can provide a custom image compositor to enable other 2-D and 3-D effects, such as applying textures to portions of the image, per-pixel alpha blending, mapping the image to stationary or moving 3-D objects, and so on.

The remaining information has to be gleaned from the Cube sample, but it was written for the VMR-7. So now I'll try to fill some holes on how we can use this interface by implenting, in C#, a version of this Cube sample. The Cube sample prompts the user for three video files and play these files on a rotating cube.

After creating a filter graph and adding a VMR9 renderer object to it, we have the following code:

        
// configure the vmr for three streams in windowless mode
fc = (IVMRFilterConfig9)vmr;  
//VMRMode_Windowless  = 0x00000002
fc.SetRenderingMode( 0x00000002 );
fc.SetNumberOfStreams( 3 );
// tell the vmr where to display the streams
wc = (IVMRWindowlessControl9)vmr;
wc.SetVideoClippingWindow( panel1.Handle );
 
   

This code access the IVMRFilterConfig9 of the vmr renderer object and set its mode to "windowless" and the number of streams to 3. Then, after accessing the IVMRWindowlessControl9 interface of the renderer, we set the "display" window to a panel (which fill the client area of our form). Then comes the following code:

        
// create our image compositor object
MyIC myIc = new MyIC();
// access the image compositor interface and
// tell the vmr (through its filter config) about it
ic = (IVMRImageCompositor9)myIc;
fc.SetImageCompositor( ic );                    
 
   

We create our custom MyIC image compositor object and call the SetImageCompositor method on the IVMRFilterConfig9 interface of the renderer object. We'll come back later to the implementation of our custom image compositor object. After this code, we use "Intelligent Connect" to build the rest of the graph. Then we set the video position using the following code:

        
// sets the video position to occupy the whole panel
wc.GetVideoPosition( out tagRectSrc, out tagRectDst );
tagRectDst.Top = 0; tagRectDst.Left = 0; 
tagRectDst.right = panel1.Size.Width;
tagRectDst.bottom = panel1.Size.Height;
wc.SetVideoPosition( ref tagRectSrc , ref tagRectDst );
 
   

We call the methods GetVideoPosition and SetVideoPosition of the IVMRWindowlessControl9 interface to retrieve and set the source and destination rectangle for the video stream. Then, we use as before, the media control and event interfaces to configure and run the graph.

We are now ready to look at the implementation of our custom image compositor class which starts as follows:

  
public class MyIC : IVMRImageCompositor9
{
  // our device and vb variables
  private Device device = null;
  private VertexBuffer vb = null;
  private float angle = 0.0f;
  // we remember if we have texture surface using these
  bool [] isTexture;
 
   

Our class implements the IVMRImageCompositor9 interface which consists of four methods. We declare a device and vertex buffer object for our cube, and an array of boolean values to store if the surfaces handed over by the VMR to our component are, in fact, texture resources. The original C++ Cube sample create a "mirror" texture and copy the surface content to this texture surface if the VMR doesn't provide the image compositor with texture resources. My version doesn't provide this functionality (and doesn't display anything if one is handed over non-texture resources). We also declare two arrays of three surfaces and textures that will be needed when we composite our three streams into a final image.

We are now ready to look at the method InitCompositeDevice of the image compositor interface. The code store away the device and create a vertex buffer for a texture cube.

 
//
// call to prepare to display frames, we save the device
// and create our vb
//
public void InitCompositionDevice ( IntPtr pD3DDevice )    
{
  if( pD3DDevice == IntPtr.Zero ) return;
  device = new Device( pD3DDevice );
  device.DeviceReset += new EventHandler( resetHandler );
  resetHandler( this, null );
}
 
   

The creation of the vertex buffer and the setting of the camera is quite standard. So we'll skip to the code for TermCompositionDevice which is empty, hence the marshaller will return S_OK to the caller (that's also the implementation in the C++ version). Then, we have another method of the image compositor interface:

    
//
// i save the texture information passed in by the mixer 
//
public void SetStreamMediaType ( uint dwStrmID , ref AM_MEDIA_TYPE pmt , int fTexture )
{
isTexture[(int)dwStrmID] = (fTexture == 0) ? false : true;
}
 
   

The third argument to this method is a boolean value that indicates if the surface handed by the VMR (or more precisely, the mixer component) to the compositor is a texture; so we store away this value.

Now, we arrive at the main method of the image compositor interface which starts with:

    
//
// the pVideoStreamInfo contains references to our textures, draw these
// on a cube
//
public void CompositeImage ( IntPtr pD3DDevice , IntPtr pddsRenderTarget , 
ref AM_MEDIA_TYPE pmtRenderTarget , long rtStart , long rtEnd , uint dwClrBkGnd , 
IntPtr pVideoStreamInfo , uint cStreams )
{
 
   

I'll let you look at the meaning of the different arguments in the doc (since that's all there is in the documentation) and I'll describe the important one: the pVideoStreamInfo argument. According to the doc, this argument is a pointer to an array of VMR9VideoStreamInfo structures. The first field of this structure is a pointer to IDirect3DSurface9 interface (marshal as an IntPtr by the .Net runtime). So after, some standard managed DirectX code (clearing the screen, setting the camera and stream source and beginning the scene), we have the following:

      
// draw the cube with its video texture  
for( i = 0; i < 3; i++ ) 
{ 
  // this code works only on 32-bit architecture (64 = size of VMR9VideoStreamInfo)
  IntPtr arrPtr = Marshal.ReadIntPtr( pVideoStreamInfo, i*64 );
  if( isTexture[i] ) 
  {
    DrawBox(angle / (float)Math.PI, angle / (float)Math.PI * 2.0f, 
      angle / (float)Math.PI / 4.0f, 0.0f, 0.0f, 0.0f, arrPtr, i ); 
  }
}
 
   

We call ReadIntPtr to read the IDirect3DSurface9 pointers (who are 64 bytes apart in the native heap, where 64 is the size of the structure on a 32-bit machine). And then we call the DrawBox function passing the surface pointer and the loop index values for the last two arguments. The DrawBox function is coded as:

    
private void DrawBox(float yaw, float pitch, float roll, float x, float y, float z, IntPtr t, int i )
{
  try 
  {
    device.Transform.World = Matrix.RotationYawPitchRoll(yaw, pitch, roll) * 
                             Matrix.Translation(x, y, z);      
    // get our surface object
    surfaces[i] = new Surface( t );   
    GC.SuppressFinalize( surfaces[i] );
    // get our texture from the surface
    textures[i] = (Texture)surfaces[i].GetContainer( IID_IDirect3DTexture9 );      
    GC.SuppressFinalize( textures[i] );
    // set the texture and draw
    device.SetTexture(0, textures[i] );      
    device.DrawPrimitives(PrimitiveType.TriangleList, i*12, 4);
  }
  catch( Exception ) 
  {
    // when we run out of surfaces objects, call collect
    GC.Collect();
  }
}
 
   

I didn't come up with this on my first attempt! So let's go over it. We get the surface from the pointer passed in, then we call GetContainer like we did in the custom A/P sample that I wrote awhile ago (since this is the only way I found to access the texture from a surface in a DirectShow graph). We call GC.SuppressFinalize because the Mixer component, of the VMR, is not too happy if we get rid of the interfaces too quickly (like I did, for example, in the custom A/P sample). The problem now is we are going to run out of DirectX resources (more precisely here, textures and surfaces), so I call GC.Collect when I run out of these. This is not elegant one bit (and it's inefficient), but it's is the only way I found so far to make this work.

It's a pity that the image compositor interface is so poorly documented. Because it provides a relatively simple way to create output for sophisticate DirectShow applications.

---

Bookmark it...