Backend (NestJS + ML) Developer

Job Type: Full-time, Long-term

English level: B2+

About the Project

We’re building a GCP-based catalog of AI-generated images and videos for chat characters.

The system reuses previously generated media via vector search + reranking, and only generates new content when needed. Each result is post-analyzed (captions, detectors, embeddings) and stored with rich metadata tied to a specific character.

Responsibilities

Backend (NestJS + Prisma + Postgres/pgvector)

• Build APIs:
  • ComfyUI webhooks (generation_complete)
  • Recommendations, feedback/moderation, catalog states
  • Suggestions/popular/ask, gifts (wallet, purchase, fulfillment)
  • Precompute cancel logic
• Implement vector search (pgvector top-k) + MMR reranker (quality, novelty, diversity).
• Manage job queues & priorities: video_user, image_user, video_precompute (with cancellation).
• Design and maintain data models & migrations for media, tags, embeddings, feedback, jobs, gifts, wallets/transactions.

ML Integration

• Work with vLLM (Qwen2-VL 7B Instruct, JSON-only captions).
• Integrate OpenCLIP (text/image embeddings), YOLO, GroundingDINO, MMPose, PaddleOCR, OpenNSFW2.
• Support Whisper (optional).
• Build media analysis pipelines: keyframes, shots, OCR, pose, NSFW level, embeddings, perceptual hashes.
• Implement “human-in-the-loop” quality gate: like/dislike (+reasons), watch ratio for video; only approved outputs get indexed.

Must-Have Qualifications

• 4+ years of strong experience with NestJS and Postgres.
• Solid SQL schema design and migrations.
• Hands-on experience with pgvector (or Milvus/Qdrant) and cosine similarity search.
• Practical experience with CLIP embeddings and reranking (MMR, novelty, repetition penalties).
• Proven experience in ML/vision service integration (preferably vLLM) with strict JSON outputs.

Nice to Have

• Experience with ComfyUI API (including cancellation).
• Familiarity with Kueue/Volcano, HPA/KEDA.