|
The comments in this post strongly validate the need for reliable video processing and understanding with VLMs. While you can use Gemini or other local VLMs, the real challenge is token efficiency, accuracy, and coverage. For example, how do you make a VLM “watch” a 2-hour or 4GB video without losing context or meaning? Video transcript alone can be sufficient for basic workflows needing no visual context. But when deep contextual understanding is required, e.g., self-driving, security analysis, warehouse tracking, etc., you’ll need more advanced methods like keyframe sampling, clipping, chunking, and shots+transcript. You can explore the different encoding strategies we designed for efficient video processing and understanding here: https://vlm-run.github.io/mm/encoders/#video. FYI, the repo is now public, and contributions are welcome. |