How to use WebCam in XNA

In our previous sample we have seen how to use Microphone device. Now we will be talking about Webcam.

For Webcam we have several alternatives.Com Registration and DirectShow.

I have chosen the DirectShow way! Let me remind one thing: DirectShowLib says No Warranty what happens to your webcam or computer,so I shall tell you so :)

Its much more easier and no-com register at all.

Ok lets start!

Create a new XNA 4.0 Windows Game Project(i named it WebCamXNA)

Add these namespaces and references to your project.

System

System.Drawing

DirectShowLib

using System.Windows.Forms;

using DirectShowLib;

using System.Drawing;

Then add Capture.cs file to your project there are lots of comments so no need to explain further :) but if you need me to explain,please write a comment under this article so i would be more helpful;

Which has this structure:

using System;

using System.Drawing;

using System.Drawing.Imaging;

using System.Collections;

using System.Runtime.InteropServices;

using System.Threading;

using System.Diagnostics;

using System.Windows.Forms;

using DirectShowLib;

namespace WebCamXNA

{

internal class Capture : ISampleGrabberCB, IDisposable

{

#region Member variables

/// <summary> graph builder interface. </summary>

private IFilterGraph2 m_FilterGraph = null;

// Used to snap picture on Still pin

private IAMVideoControl m_VidControl = null;

private IPin m_pinStill = null;

/// <summary> so we can wait for the async job to finish </summary>

private ManualResetEvent m_PictureReady = null;

private bool m_WantOne = false;

/// <summary> Dimensions of the image, calculated once in constructor for perf. </summary>

private int m_videoWidth;

private int m_videoHeight;

private int m_stride;

/// <summary> buffer for bitmap data. Always release by caller</summary>

private IntPtr m_ipBuffer = IntPtr.Zero;

#if DEBUG

// Allow you to "Connect to remote graph" from GraphEdit

DsROTEntry m_rot = null;

#endif

#endregion

#region APIs

[DllImport("Kernel32.dll", EntryPoint="RtlMoveMemory")]

private static extern void CopyMemory(IntPtr Destination, IntPtr Source, [MarshalAs(UnmanagedType.U4)] int Length);

#endregion

// Zero based device index and device params and output window

public Capture(int iDeviceNum, int iWidth, int iHeight, short iBPP, Control hControl)

{

DsDevice [] capDevices;

// Get the collection of video devices

capDevices = DsDevice.GetDevicesOfCat(FilterCategory.VideoInputDevice);

if (iDeviceNum + 1 > capDevices.Length)

{

throw new Exception("No video capture devices found at that index!");

}

try

{

// Set up the capture graph

SetupGraph( capDevices[iDeviceNum], iWidth, iHeight, iBPP, hControl);

// tell the callback to ignore new images

m_PictureReady = new ManualResetEvent(false);

}

catch

{

Dispose();

throw;

}

/// <summary> release everything. </summary>

public void Dispose()

{

#if DEBUG

if (m_rot != null)

{

m_rot.Dispose();

}

#endif

CloseInterfaces();

if (m_PictureReady != null)

{

m_PictureReady.Close();

}

// Destructor

~Capture()

{

Dispose();

}

/// <summary>

/// Get the image from the Still pin. The returned image can turned into a bitmap with

/// Bitmap b = new Bitmap(cam.Width, cam.Height, cam.Stride, PixelFormat.Format24bppRgb, m_ip);

/// If the image is upside down, you can fix it with

/// b.RotateFlip(RotateFlipType.RotateNoneFlipY);

/// </summary>

/// <returns>Returned pointer to be freed by caller with Marshal.FreeCoTaskMem</returns>

public IntPtr Click()

{

int hr;

// get ready to wait for new image

m_PictureReady.Reset();

m_ipBuffer = Marshal.AllocCoTaskMem(Math.Abs(m_stride) * m_videoHeight);

try

{

m_WantOne = true;

// If we are using a still pin, ask for a picture

if (m_VidControl != null)

{

// Tell the camera to send an image

hr = m_VidControl.SetMode(m_pinStill, VideoControlFlags.Trigger);

DsError.ThrowExceptionForHR( hr );

}

// Start waiting

if ( ! m_PictureReady.WaitOne(9000, false) )

{

throw new Exception("Timeout waiting to get picture");

}

catch

{

Marshal.FreeCoTaskMem(m_ipBuffer);

m_ipBuffer = IntPtr.Zero;

throw;

}

// Got one

return m_ipBuffer;

}

public int Width

{

get

{

return m_videoWidth;

}

public int Height

{

get

{

return m_videoHeight;

}

public int Stride

{

get

{

return m_stride;

}

/// <summary> build the capture graph for grabber. </summary>

private void SetupGraph(DsDevice dev, int iWidth, int iHeight, short iBPP, Control hControl)

{

int hr;

ISampleGrabber sampGrabber = null;

IBaseFilter capFilter = null;

IPin pCaptureOut = null;

IPin pSampleIn = null;

IPin pRenderIn = null;

// Get the graphbuilder object

m_FilterGraph = new FilterGraph() as IFilterGraph2;

try

{

#if DEBUG

m_rot = new DsROTEntry(m_FilterGraph);

#endif

// add the video input device

hr = m_FilterGraph.AddSourceFilterForMoniker(dev.Mon, null, dev.Name, out capFilter);

DsError.ThrowExceptionForHR( hr );

// Find the still pin

m_pinStill = DsFindPin.ByCategory(capFilter, PinCategory.Still, 0);

// Didn't find one. Is there a preview pin?

if (m_pinStill == null)

{

m_pinStill = DsFindPin.ByCategory(capFilter, PinCategory.Preview, 0);

}

// Still haven't found one. Need to put a splitter in so we have

// one stream to capture the bitmap from, and one to display. Ok, we

// don't *have* to do it that way, but we are going to anyway.

if (m_pinStill == null)

{

IPin pRaw = null;

IPin pSmart = null;

// There is no still pin

m_VidControl = null;

// Add a splitter

IBaseFilter iSmartTee = (IBaseFilter)new SmartTee();

try

{

hr = m_FilterGraph.AddFilter(iSmartTee, "SmartTee");

DsError.ThrowExceptionForHR( hr );

// Find the find the capture pin from the video device and the

// input pin for the splitter, and connnect them

pRaw = DsFindPin.ByCategory(capFilter, PinCategory.Capture, 0);

pSmart = DsFindPin.ByDirection(iSmartTee, PinDirection.Input, 0);

hr = m_FilterGraph.Connect(pRaw, pSmart);

DsError.ThrowExceptionForHR( hr );

// Now set the capture and still pins (from the splitter)

m_pinStill = DsFindPin.ByName(iSmartTee, "Preview");

pCaptureOut = DsFindPin.ByName(iSmartTee, "Capture");

// If any of the default config items are set, perform the config

// on the actual video device (rather than the splitter)

if (iHeight + iWidth + iBPP > 0)

{

SetConfigParms(pRaw, iWidth, iHeight, iBPP);

}

finally

{

if (pRaw != null)

{

Marshal.ReleaseComObject(pRaw);

}

if (pRaw != pSmart)

{

Marshal.ReleaseComObject(pSmart);

}

if (pRaw != iSmartTee)

{

Marshal.ReleaseComObject(iSmartTee);

}

else

{

// Get a control pointer (used in Click())

m_VidControl = capFilter as IAMVideoControl;

pCaptureOut = DsFindPin.ByCategory(capFilter, PinCategory.Capture, 0);

// If any of the default config items are set

if (iHeight + iWidth + iBPP > 0)

{

SetConfigParms(m_pinStill, iWidth, iHeight, iBPP);

}

// Get the SampleGrabber interface

sampGrabber = new SampleGrabber() as ISampleGrabber;

// Configure the sample grabber

IBaseFilter baseGrabFlt = sampGrabber as IBaseFilter;

ConfigureSampleGrabber(sampGrabber);

pSampleIn = DsFindPin.ByDirection(baseGrabFlt, PinDirection.Input, 0);

// Get the default video renderer

IBaseFilter pRenderer = new VideoRendererDefault() as IBaseFilter;

hr = m_FilterGraph.AddFilter(pRenderer, "Renderer");

DsError.ThrowExceptionForHR( hr );

pRenderIn = DsFindPin.ByDirection(pRenderer, PinDirection.Input, 0);

// Add the sample grabber to the graph

hr = m_FilterGraph.AddFilter( baseGrabFlt, "Ds.NET Grabber" );

DsError.ThrowExceptionForHR( hr );

if (m_VidControl == null)

{

// Connect the Still pin to the sample grabber

hr = m_FilterGraph.Connect(m_pinStill, pSampleIn);

DsError.ThrowExceptionForHR( hr );

// Connect the capture pin to the renderer

hr = m_FilterGraph.Connect(pCaptureOut, pRenderIn);

DsError.ThrowExceptionForHR( hr );

}

else

{

// Connect the capture pin to the renderer

hr = m_FilterGraph.Connect(pCaptureOut, pRenderIn);

DsError.ThrowExceptionForHR( hr );

// Connect the Still pin to the sample grabber

hr = m_FilterGraph.Connect(m_pinStill, pSampleIn);

DsError.ThrowExceptionForHR( hr );

}

// Learn the video properties

SaveSizeInfo(sampGrabber);

ConfigVideoWindow(hControl);

// Start the graph

IMediaControl mediaCtrl = m_FilterGraph as IMediaControl;

hr = mediaCtrl.Run();

DsError.ThrowExceptionForHR( hr );

}

finally

{

if (sampGrabber != null)

{

Marshal.ReleaseComObject(sampGrabber);

sampGrabber = null;

}

if (pCaptureOut != null)

{

Marshal.ReleaseComObject(pCaptureOut);

pCaptureOut = null;

}

if (pRenderIn != null)

{

Marshal.ReleaseComObject(pRenderIn);

pRenderIn = null;

}

if (pSampleIn != null)

{

Marshal.ReleaseComObject(pSampleIn);

pSampleIn = null;

}

private void SaveSizeInfo(ISampleGrabber sampGrabber)

{

int hr;

// Get the media type from the SampleGrabber

AMMediaType media = new AMMediaType();

hr = sampGrabber.GetConnectedMediaType( media );

DsError.ThrowExceptionForHR( hr );

if( (media.formatType != FormatType.VideoInfo) || (media.formatPtr == IntPtr.Zero) )

{

throw new NotSupportedException( "Unknown Grabber Media Format" );

}

// Grab the size info

VideoInfoHeader videoInfoHeader = (VideoInfoHeader) Marshal.PtrToStructure( media.formatPtr, typeof(VideoInfoHeader) );

m_videoWidth = videoInfoHeader.BmiHeader.Width;

m_videoHeight = videoInfoHeader.BmiHeader.Height;

m_stride = m_videoWidth * (videoInfoHeader.BmiHeader.BitCount / 8);

DsUtils.FreeAMMediaType(media);

media = null;

}

// Set the video window within the control specified by hControl

private void ConfigVideoWindow(Control hControl)

{

int hr;

IVideoWindow ivw = m_FilterGraph as IVideoWindow;

// Set the parent

hr = ivw.put_Owner(hControl.Handle);

DsError.ThrowExceptionForHR( hr );

// Turn off captions, etc

hr = ivw.put_WindowStyle(WindowStyle.Child | WindowStyle.ClipChildren | WindowStyle.ClipSiblings);

DsError.ThrowExceptionForHR( hr );

// Yes, make it visible

hr = ivw.put_Visible( OABool.True );

DsError.ThrowExceptionForHR( hr );

// Move to upper left corner

Rectangle rc = hControl.ClientRectangle;

hr = ivw.SetWindowPosition( 0, 0, rc.Right, rc.Bottom );

DsError.ThrowExceptionForHR( hr );

}

private void ConfigureSampleGrabber(ISampleGrabber sampGrabber)

{

int hr;

AMMediaType media = new AMMediaType();

// Set the media type to Video/RBG24

media.majorType = MediaType.Video;

media.subType = MediaSubType.RGB24;

media.formatType = FormatType.VideoInfo;

hr = sampGrabber.SetMediaType( media );

DsError.ThrowExceptionForHR( hr );

DsUtils.FreeAMMediaType(media);

media = null;

// Configure the samplegrabber

hr = sampGrabber.SetCallback( this, 1 );

DsError.ThrowExceptionForHR( hr );

}

// Set the Framerate, and video size

private void SetConfigParms(IPin pStill, int iWidth, int iHeight, short iBPP)

{

int hr;

AMMediaType media;

VideoInfoHeader v;

IAMStreamConfig videoStreamConfig = pStill as IAMStreamConfig;

// Get the existing format block

hr = videoStreamConfig.GetFormat(out media);

DsError.ThrowExceptionForHR(hr);

try

{

// copy out the videoinfoheader

v = new VideoInfoHeader();

Marshal.PtrToStructure( media.formatPtr, v );

// if overriding the width, set the width

if (iWidth > 0)

{

v.BmiHeader.Width = iWidth;

}

// if overriding the Height, set the Height

if (iHeight > 0)

{

v.BmiHeader.Height = iHeight;

}

// if overriding the bits per pixel

if (iBPP > 0)

{

v.BmiHeader.BitCount = iBPP;

}

// Copy the media structure back

Marshal.StructureToPtr( v, media.formatPtr, false );

// Set the new format

hr = videoStreamConfig.SetFormat( media );

DsError.ThrowExceptionForHR( hr );

}

finally

{

DsUtils.FreeAMMediaType(media);

media = null;

}

/// <summary> Shut down capture </summary>

private void CloseInterfaces()

{

int hr;

try

{

if( m_FilterGraph != null )

{

IMediaControl mediaCtrl = m_FilterGraph as IMediaControl;

// Stop the graph

hr = mediaCtrl.Stop();

}

catch (Exception ex)

{

Debug.WriteLine(ex);

}

if (m_FilterGraph != null)

{

Marshal.ReleaseComObject(m_FilterGraph);

m_FilterGraph = null;

}

if (m_VidControl != null)

{

Marshal.ReleaseComObject(m_VidControl);

m_VidControl = null;

}

if (m_pinStill != null)

{

Marshal.ReleaseComObject(m_pinStill);

m_pinStill = null;

}

/// <summary> sample callback, NOT USED. </summary>

int ISampleGrabberCB.SampleCB( double SampleTime, IMediaSample pSample )

{

Marshal.ReleaseComObject(pSample);

return 0;

}

/// <summary> buffer callback, COULD BE FROM FOREIGN THREAD. </summary>

int ISampleGrabberCB.BufferCB( double SampleTime, IntPtr pBuffer, int BufferLen )

{

// Note that we depend on only being called once per call to Click. Otherwise

// a second call can overwrite the previous image.

Debug.Assert(BufferLen == Math.Abs(m_stride) * m_videoHeight, "Incorrect buffer length");

if (m_WantOne)

{

m_WantOne = false;

Debug.Assert(m_ipBuffer != IntPtr.Zero, "Unitialized buffer");

// Save the buffer

CopyMemory(m_ipBuffer, pBuffer, BufferLen);

// Picture is ready.

m_PictureReady.Set();

}

return 0;

}

This is our class that will capture our WebCam device.

How will we use it?

Open your Game1.cs class and write these code:

Add these namespaces:

using System.Runtime.InteropServices;

using System.Windows.Forms;

using System.Drawing;

using System.Drawing.Imaging;

Add these variables:

private System.Windows.Forms.Button button1;

private System.Windows.Forms.PictureBox pictureBox1;

private System.ComponentModel.Container components = null;

private System.Windows.Forms.PictureBox pictureBox2;

private Capture cam;

IntPtr m_ip = IntPtr.Zero;

Create a Dispose Function:

protected override void Dispose(bool disposing)

{

if (disposing)

{

if (components != null)

{

components.Dispose();

}

base.Dispose(disposing);

if (m_ip != IntPtr.Zero)

{

Marshal.FreeCoTaskMem(m_ip);

m_ip = IntPtr.Zero;

}

Add these codes inside Initialize function which loads 2 picturebox,a button and a camera

this.button1 = new System.Windows.Forms.Button();

this.pictureBox1 = new System.Windows.Forms.PictureBox();

this.pictureBox2 = new System.Windows.Forms.PictureBox();

((System.ComponentModel.ISupportInitialize)(this.pictureBox1)).BeginInit();

((System.ComponentModel.ISupportInitialize)(this.pictureBox2)).BeginInit();

// button1

this.button1.Location = new System.Drawing.Point(368, 80);

this.button1.Name = "button1";

this.button1.Size = new System.Drawing.Size(75, 40);

this.button1.TabIndex = 0;

this.button1.Text = "Click";

this.button1.Click += new System.EventHandler(this.button1_Click);

// pictureBox1

this.pictureBox1.BorderStyle = System.Windows.Forms.BorderStyle.FixedSingle;

this.pictureBox1.Location = new System.Drawing.Point(16, 256);

this.pictureBox1.Name = "pictureBox1";

this.pictureBox1.Size = new System.Drawing.Size(320, 240);

this.pictureBox1.TabIndex = 1;

this.pictureBox1.TabStop = false;

// pictureBox2

this.pictureBox2.BorderStyle = System.Windows.Forms.BorderStyle.FixedSingle;

this.pictureBox2.Location = new System.Drawing.Point(16, 8);

this.pictureBox2.Name = "pictureBox2";

this.pictureBox2.Size = new System.Drawing.Size(320, 240);

this.pictureBox2.TabIndex = 2;

this.pictureBox2.TabStop = false;

Control.FromHandle(Window.Handle).Controls.Add(pictureBox1);

Control.FromHandle(Window.Handle).Controls.Add(pictureBox2);

Control.FromHandle(Window.Handle).Controls.Add(button1);

const int VIDEODEVICE = 0; // zero based index of video capture device to use

const int VIDEOWIDTH = 320; // Depends on video device caps

const int VIDEOHEIGHT = 240; // Depends on video device caps

const int VIDEOBITSPERPIXEL = 24; // BitsPerPixel values determined by device

cam = new Capture(VIDEODEVICE, VIDEOWIDTH, VIDEOHEIGHT, VIDEOBITSPERPIXEL, pictureBox2);

Create a Button Click event

private void button1_Click(object sender, System.EventArgs e)

{

Cursor.Current = Cursors.WaitCursor;

// Release any previous buffer

if (m_ip != IntPtr.Zero)

{

Marshal.FreeCoTaskMem(m_ip);

m_ip = IntPtr.Zero;

}

// capture image

m_ip = cam.Click();

Bitmap b = new Bitmap(cam.Width, cam.Height, cam.Stride, PixelFormat.Format24bppRgb, m_ip);

// If the image is upsidedown

b.RotateFlip(RotateFlipType.RotateNoneFlipY);

pictureBox1.Image = b;
Cursor.Current = Cursors.Default;
}

Update your UnloadContent function as shown below:

protected override void UnloadContent()

{

cam.Dispose();

if (m_ip != IntPtr.Zero)

{

Marshal.FreeCoTaskMem(m_ip);

m_ip = IntPtr.Zero;

}

And now run the project and see what happens!

Picturebox1 displays webcam output,Picturebox 2 will show your picture when clicked the button.

Here looking at camera :)

And now I have took a photo of mine and showed it on Picturebox2 :)

Nice! isnt it? Ok not me perhaps but the project itself :) :)

We have added Webcam functionality to our game but whats the point?

What can you do with Webcam in a game?

You can develop Game-Version of ChatRoulette.in which 2 players view their camera in picturebox1 and picturebox2.And in the right corner they can play nice games such as Chess,Card Games(Hearts,FreeCell) or anything you can imagine ;)

And that would be a great game and even celebs can play with you :)

Allright I showed you today how to use webcam functionality in games with a nice demo.

Hope you like it!

See you in other articles ;)

Cheers

9 Comments

Join the conversation! Your thoughts help the community grow.

Janis Stolzenwald7y ago
Hi, thank you so much for your example, it works really well. I faced a resolution problem though. When I set the constants for VIDEOWIDTH/HEIGHT to HD i.e. 1280/720 then the video becomes super lagy. is there a way to improve that?
Greg Ivanov14y ago
Thank you for your example, but in windows XP functions described above taking sreen it does not work (provided by exception "Timeout waiting to get picture"), if it can be replaced by another way? Best regards
Brian Singh14y ago
Great article. I am running your code on Windows XP with DirectX 9.0c. I am getting the following error: No combination of intermediate filters could be found to make the connection. [VFW_E_CANNOT_CONNECT (0x80040217)] The error is thrown at...Capture -> SetupGraph() if (_mVidControl == null) { /* THROWS EXCEPTION [VFW_E_CANNOT_CONNECT (0x80040217)] */ hr = _mFilterGraph.Connect(_mPinStill, pSampleIn); DsError.ThrowExceptionForHR( hr ); // works no exception hr = _mFilterGraph.Connect(pCaptureOut, pRenderIn); DsError.ThrowExceptionForHR( hr ); } else { // works no exception hr = _mFilterGraph.Connect(pCaptureOut, pRenderIn); DsError.ThrowExceptionForHR( hr ); /* THROWS EXCEPTION [VFW_E_CANNOT_CONNECT (0x80040217)] */ hr = _mFilterGraph.Connect(_mPinStill, pSampleIn); DsError.ThrowExceptionForHR( hr ); } Seems the issue is with connecting the Still Pin. Connecting the Capture Pin works. How can I resolve this issue? Thanks in advance. PS The same code on Windows 7 with DirectX 11 works fine.
Vineet Kumar Saini15y ago
Hi Ibrahim ,I want to know How to use WebCam in C#.NET,Please tell me....
Mehrdad15y agoedited
Hi, and tnx for your working!! code. I want to capture webcam content and pass it to Image Processing block in my game, and not showing on Control. I just need a Bitmap. what should i do?
Robert15y ago
Thank you for a great example of using the webcam in XNA. I was curious how difficult it would be to adpat this code so that it accepts a webcam feed through a TCP connection?
- Ibrahim Ersoy15y ago
  @Cem: Your welcome! @aditya: Im not sure if this method works with WebBrowser.But i'd advice dont store image itself in a database.Just store its path. @Robert: I havent tried it actually.But i will share the results with you when i did.
aditya gangwar16y ago
i am working in c#.net with web application, how to integrate webcam in web pages and how to store image in sql database
Cem Olcay16y ago
Helal olsun valla calisti, aklima -sizinde dediginiz gibi- bir cok proje gelmeye basladi bile, cok tesekkur ederim :)