hackathon project to production, and the iterative process

My primary reason for building this project is a rather personal one. In software, it is easy to find yourself building primarily for an internship or your resume. This time, however, I was building from a childhood passion for photography and cinematography, creating a solution for something I genuinely cared about.

The point of this tech blog is to show that it is never too late to return to a project, proving that software development is an iterative process. Read on to understand how, months after the hackathon, I rebuilt part of FrameShift's architecture to overcome an intrinsic limitation in the system and make it work in production.

FrameShift is a tool that lets you segment any object in any frame, edit it by repositioning, enlarging, recolouring, or transforming it entirely, and automatically propagate that change across every subsequent frame. With just a few clicks, it can remove small visual imperfections and turn them into seamless edits through one end-to-end workflow.

The ultimate goal is low-latency change propagation, convincing spatiotemporal relationships between frames, and a deeper understanding of video object segmentation and vision models. The project's iterations through RIFE, image-generation models, and SAM 2 will each become a separate article.

Client-side, FrameShift runs on Next.js and TypeScript. A custom canvas renderer draws the active frame, and a timeline scrubber handles frame navigation and real-time previews.

Editor state lives in a single useEditorState hook. Instead of five separate loading spinners scattered across components, one unified polling system tracks every async operation: object detection, segmentation, editing, refinement, and video propagation. It reports each stage back to the UI.

// one poller for every long-running backend op
async function pollJob(kind: OpKind, id: string) {
  while (true) {
    const res = await fetch(`/api/${kind}/${id}`);
    if (res.status === 202) {            // still working
      const { stage } = await res.json();
      setStage(kind, stage);             // e.g. "segmenting" -> "propagating"
      await sleep(800);
      continue;
    }
    return res.json();                   // done
  }
}

On the backend, FastAPI runs the whole video-processing and AI pipeline. It follows a service-based layout so the route handlers stay thin and the logic stays testable:

backend/controllers/   http route handlers
backend/models/        dataclasses
backend/services/      logic layer
backend/utils/         helper utils

Everything starts with decoding. FFmpeg turns the uploaded video into individual JPEG frames, then OpenCV normalizes dimensions, aspect ratios, pixel formats, and frame rates so every downstream stage gets consistent input.

def extract_frames(video_path: str) -> list[Path]:
    subprocess.run([
        "ffmpeg", "-i", video_path,
        "-vf", "fps=source",                 # keep the clip's native rate
        f"{self.frames_dir}/frame_%05d.jpg",
    ], check=True)
    return self.normalize(sorted(self.frames_dir.glob("*.jpg")))

The user clicks once. YOLOv11 detects the object nearest that click and hands over a bounding box, which seeds SAM 2. SAM 2 takes the point and the box, segments the object, and then propagates that mask across the rest of the frames, so one click tracks the object through the whole clip.

async def segment(self, frame: np.ndarray, click: Point) -> Mask:
    box = self.yolo.detect(frame, near=click)          # bounding box
    return self.sam2.prompt(frame, point=click, box=box)

def propagate(self, frames: list[np.ndarray], init: Mask) -> list[Mask]:
    # sam 2 carries the mask forward across every frame
    return self.sam2.propagate_in_video(frames, init)

In the first version, the pipeline split on what the edit actually was. Semantic transformations such as “make it a sports car” go through Gemini; direct edits such as blur, delete, and recolour skip generation entirely.

if edit.is_semantic:
    frames = await self.generative_path(edit, masks)
else:
    frames = self.direct_path(edit, masks)

KEYFRAME_STRIDE = 10   # generate 1 in 10 to cut latency + cost

async def generate_keyframes(self, frames, masks, prompt):
    idxs = range(0, len(frames), KEYFRAME_STRIDE)
    edited = await asyncio.gather(*[
        self.gemini.edit(frame=frames[i], mask=masks[i], prompt=prompt)
        for i in idxs
    ])
    return dict(zip(idxs, edited))

def fill_gaps(self, keyframes: dict[int, np.ndarray]) -> list[np.ndarray]:
    out = []
    for a, b in pairwise(sorted(keyframes)):
        out.append(keyframes[a])
        # shared rife model -> segments run one after another
        out += self.rife.interpolate(keyframes[a], keyframes[b], n=b - a - 1)
    return out

def delete_object(self, frame: np.ndarray, mask: np.ndarray) -> np.ndarray:
    # reconstruct whatever was behind the object
    return cv2.inpaint(frame, mask, 3, cv2.INPAINT_TELEA)

One of the largest pitfalls came from the most instinctive part of the original system. If generating every frame was expensive, generating a few frames and interpolating the gaps felt like the obvious answer. Before replacing that design, I wanted to understand why it failed instead of swapping models until the artefacts became less visible.

An image generation call is a fresh stochastic draw. Even when I sent Gemini two neighbouring frames with almost identical prompts, each call began from different noise and could settle on a slightly different object. The colour could shift, an edge could move, or a detail could appear in one keyframe and disappear in the next. The model understood my instruction, but it had no memory that these were meant to be two moments in the life of the same edited object.

SAM 2 had a different kind of memory. It tracked visual appearance and location through its memory bank, but that was not the same as remembering the semantic identity of Gemini's new object. It could keep telling me where the selected car was, but it could not insist that Gemini preserve the exact headlights, paint, and body shape it generated ten frames earlier.

That distinction became central to the rebuild. Gemini decided what the edit should look like. SAM 2 decided where the object was. Neither model, on its own, guaranteed that the generated appearance would remain identical through time.

Whichever path ran, the SAM 2 mask does the final assembly. It acts as a stencil: edited pixels go inside the object region, and the original video stays untouched everywhere outside it.

This is alpha compositing, the same over operation used by image editors. For every pixel, the calculation is final = generated × mask + original × (1 - mask). A value of one keeps the generated pixel, while zero keeps the original pixel. I feather the boundary over a few pixels instead of using a strictly binary edge, which avoids the hard, aliased look of a pasted cut-out. Keeping the mask as a lossless PNG preserves those intermediate alpha values through the pipeline.

def composite(self, original, edited, mask):
    # mask = 1 inside the object, 0 outside
    return edited * mask + original * (1 - mask)

Cloudinary handles heavier image transformations such as recolouring, background removal, restoration, and upscaling, while Gemini owns the generative object transforms. Finally, FFmpeg combines the completed frames with the original audio into the finished video.

def encode(self, frames_dir, audio_src, out):
    subprocess.run([
        "ffmpeg", "-framerate", str(self.fps),
        "-i", f"{frames_dir}/frame_%05d.jpg",
        "-i", audio_src,
        "-map", "0:v", "-map", "1:a", "-c:a", "copy",   # frames + original audio
        out,
    ], check=True)

Every edit is a long-running job, so nothing blocks the HTTP response. When the user hits generate, the frontend captures the active frame as a PNG, sends it with the prompt, and gets an instant job ID back.

@post("/edit")
async def create_edit(self, request: Request):
    payload = await request.json()
    job_id = await self.job_service.enqueue(payload)    # fires a background task
    return json({"job_id": job_id}, status=202)         # instant return

The job runs in the background while the frontend polls a status endpoint. Each poll reports the current stage until the video is ready.

@get("/edit/{job_id}")
async def edit_status(self, job_id: str):
    job = await self.job_service.status(job_id)
    if job.status == "processing":
        return json({"status": "processing", "stage": job.stage}, status=202)
    return json({"status": "done", "video_url": job.video_url})

RIFE assumes its two endpoints are glimpses of the same scene caught during one continuous motion. That assumption was exactly what broke in FrameShift. Two independently generated keyframes were not two photographs of one object moving. They were two plausible versions of the requested object.

RIFE could not know that. It estimated a smooth transition anyway, inventing a motion field that turned appearance A into appearance B. When the two generations disagreed, that transition appeared as smearing, ghosting, stretching, or features from both versions briefly occupying the same frame.

• Gemini understood the semantic instruction, but not temporal identity across independent calls.
• RIFE produced temporally smooth transitions, but it interpolated pixels without understanding the object.
• SAM 2 constrained the region of the edit, but it could not make the pixels inside that region describe one stable design.

The mask could hide interpolation outside the object, but it could not repair an unstable object inside it. This was the fundamental mismatch in the first architecture.

The solution was to stop asking the image model to redraw the object throughout the clip. I now generate the edit once on an anchor frame, then transport those exact pixels through the video using optical flow. Generation defines appearance once. Motion is handled separately.

Optical flow is a dense motion field between two frames. Every pixel receives a vector (u, v) describing where that point moved. The classical starting point is brightness constancy: I(x, y, t) = I(x + u, y + v, t + 1). A first-order expansion gives Iₓu + Iᵧv + Iₜ = 0. That leaves one equation with two unknowns at every pixel, which is the aperture problem. Rather than solving that underdetermined system directly, I use RAFT to estimate the dense field.

from torchvision.models.optical_flow import Raft_Large_Weights, raft_large

weights = Raft_Large_Weights.DEFAULT
raft = raft_large(weights=weights, progress=False).eval().to(device)

@torch.inference_mode()
def flow(img_a, img_b):
    # img_a and img_b: (1, 3, H, W), with H and W divisible by 8
    img_a, img_b = weights.transforms(img_a, img_b)
    return raft(img_a, img_b, num_flow_updates=12)[-1]
    # (1, 2, H, W): horizontal and vertical displacement per pixel

To render a target frame, I use a backward warp. Each output pixel looks up the coordinate it came from in the anchor image. PyTorch'sgrid_sample performs that lookup after the pixel coordinates have been normalized to the range from negative one to one.

import torch.nn.functional as F

def warp(field, displacement):
    _, _, height, width = field.shape
    yy, xx = torch.meshgrid(
        torch.arange(height, device=field.device, dtype=field.dtype),
        torch.arange(width, device=field.device, dtype=field.dtype),
        indexing="ij",
    )
    base = torch.stack((xx, yy), dim=0).unsqueeze(0)
    source = base + displacement

    gx = 2 * (source[:, 0] + 0.5) / width - 1
    gy = 2 * (source[:, 1] + 0.5) / height - 1
    grid = torch.stack((gx, gy), dim=-1)

    return F.grid_sample(
        field, grid, mode="bilinear", padding_mode="zeros",
        align_corners=False,
    )

The direction convention matters. RAFT returns motion from its first input to its second, so I estimate each backward pair with the later frame first and the earlier frame second. The resulting field maps frame k back to frame k - 1, which is exactly what the backward sampler needs.

I also avoid warping the previous edited output into the next frame. Repeating that process would resample already softened pixels and accumulate blur. Instead, I compose the pairwise flows into one target-to-anchor displacement, then sample the pristine anchor exactly once for every output frame.

# backward_flows[k - 1] maps frame k to frame k - 1
def propagate(edit_anchor, backward_flows):
    displacement = torch.zeros_like(backward_flows[0])
    frames = [edit_anchor]

    for pairwise in backward_flows:
        # D(k -> 0) = F(k -> k-1) + D(k-1 -> 0) sampled at F
        displacement = pairwise + warp(displacement, pairwise)
        frames.append(warp(edit_anchor, displacement))

    return frames

I apply the same displacement to the anchor's alpha mask, then intersect it with SAM 2's mask for the target frame. Every frame is therefore a clean sample of the same generated object, moving along correspondences from the real video. The generative morphing disappears because there is no second generated appearance to morph towards.

Optical flow still has a physical limit. When part of the object becomes hidden or a new surface is revealed, the anchor contains no trustworthy pixel for that region. Blindly warping through an occlusion drags colours across boundaries and produces a different kind of artefact.

I detect those regions with a forward-backward consistency check. A reliable pixel should travel from frame A to frame B and return close to its starting point. If the round trip has a large error, I mark that correspondence as invalid.

def flow_validity(forward, backward, tau=0.5):
    # Travel from A to B and back to A
    round_trip = forward + warp(backward, forward)
    error = (round_trip ** 2).sum(dim=1)
    motion = (forward ** 2).sum(dim=1)
    motion += (warp(backward, forward) ** 2).sum(dim=1)

    threshold = 0.01 * motion + tau
    return error < threshold
    # True means this correspondence is safe to use

The validity map becomes another constraint on the composite. For small unreliable regions, I feather the propagated alpha back into the original frame. If the invalid area grows beyond a threshold, the pipeline creates a deliberate new anchor, using the last trusted edit as a visual reference, and begins a new flow window. Regeneration is now a recovery path for genuinely unseen content, not a scheduled event every ten frames.

This changed how I thought about the whole system. Gemini is now responsible for one semantic appearance decision. RAFT carries that decision through motion. SAM 2 remains the spatial guardrail that says where the edit is allowed to exist, and the consistency check decides where motion can be trusted.

RIFE left the object propagation path entirely. The production pipeline no longer tries to smooth over several independent generations. It preserves one approved edit for as long as the video provides valid correspondences, then re-anchors only when the scene reveals information that the original anchor never saw.

try it out