Optical Flow

• Feature Based Methods :- Track features such as corners, etc across all the frames. Hence they gave you apparent motion (Motion in the image).

• Direct or Dense Methods :- Recover motion for each pixel of image space-temporal image brigthness variation. Suitable for videos and image variation is small.

  • Lucan-Kannade Methode is used.

• Optical Flow :- Optical flow is the pattern of apparent motion of image objects between two consecutive frames caused by the movemement of object or camera.

  • Consider a pixel I(x,y,t) in first frame (Check a new dimension, time, is added here. Earlier we were working with images only, so no need of time). It moves by distance (dx,dy) in next frame taken after dt time. So since those pixels are the same and intensity does not change, we can say,

I(x,y,t)=I(x+dx,y+dy,t+dt)

• • Then take taylor series approximation of right-hand side, remove common terms and divide by dt to get the following equation:

  • Above equation is called Optical Flow equation. In it, we can find fx and fy, they are image gradients. Similarly ft is the gradient along time. But (u,v) is unknown. We cannot solve this one equation with two unknown variables. So several methods are provided to solve this problem and one of them is Lucas-Kanade.

  • Lucas-Kanade method We have seen an assumption before, that all the neighbouring pixels will have similar motion. Lucas-Kanade method takes a 3x3 patch around the point. So all the 9 points have the same motion. We can find (fx,fy,ft) for these 9 points. So now our problem becomes solving 9 equations with two unknown variables which is over-determined. A better solution is obtained with least square fit method. Below is the final solution which is two equation-two unknown problem and solve to get the solution.[ ( Check similarity of inverse matrix with Harris corner detector. It denotes that corners are better points to be tracked.)

  • So from user point of view, idea is simple, we give some points to track, we receive the optical flow vectors of those points. But again there are some problems. Until now, we were dealing with small motions. So it fails when there is large motion. For Larger motion :-

    • Iterative Refinement :- This is used when linear approximation of taylor is not valid and higher order terms should also be used.

    • Coarse-to-refinement This is used when pixel motion is not small.

      • We use hierarchical LK here:-

      • 

      • 

      • 

      •