Group Chat + Multimedia Integration Plan

Status: Planning | Current chat only supports text messages, need to extend for media.

Current State

NestJS Chat (apps/nestjs/): Mobile App:
  • Message types defined: text, image, bento, poke, schedule, gift
  • Cloudflare Stream integration exists for video
  • Hooks: useVickyChat, useChatStream, useConvexGroupChat

Architecture Overview

Required Work

Phase 1: Data Model Extension

1.1 Extend ChatMessage Entity Current:
type ChatMessage = {
  content: string  // text only
}
New:
type ChatMessage = {
  id: string
  roomId: string
  senderId: string
  senderKind: 'human' | 'ai'
  senderName: string
  content: string
  attachments: Attachment[]
  createdAt: Date
}

type Attachment = {
  id: string
  type: 'image' | 'voice' | 'video'
  url: string
  thumbnailUrl?: string  // for video
  duration?: number      // for voice/video
  mimeType: string
  size: number
}
1.2 Create Attachment DTOs
// Image attachment
class ImageAttachmentDto {
  id: string
  url: string
  width: number
  height: number
  mimeType: string
}

// Voice attachment
class VoiceAttachmentDto {
  id: string
  url: string
  duration: number  // seconds
  mimeType: string
  waveformUrl?: string
}

// Video attachment
class VideoAttachmentDto {
  id: string
  url: string
  thumbnailUrl: string
  duration: number
  mimeType: string
}

Phase 2: Media Storage API

2.1 R2 Setup for Images & Voice Configure R2 bucket for:
  • User-uploaded images (JPEG, PNG, GIF, WebP)
  • Voice messages (MP3, WebM, Opus)
// apps/nestjs/src/storage/r2.service.ts
export class R2Service {
  async uploadImage(roomId: string, userId: string, file: Buffer): Promise<string>
  async uploadVoice(roomId: string, userId: string, file: Buffer): Promise<string>
  async getSignedUploadUrl(roomId: string, userId: string, contentType: string): Promise<string>
  async getSignedDownloadUrl(key: string): Promise<string>
}
2.2 Cloudflare Stream for Video Video uploads via direct upload URL:
// apps/nestjs/src/storage/stream.service.ts
export class StreamService {
  async createDirectUploadUrl(): Promise<{ uid: string; uploadUrl: string }>
  async getVideoUrl(uid: string): Promise<string>
  async getThumbnailUrl(uid: string): Promise<string>
}
2.3 Media Endpoints
EndpointMethodDescription
/v1/media/upload-urlPOSTGet signed R2 upload URL
/v1/media/voice/upload-urlPOSTGet signed R2 upload URL for voice
/v1/media/video/direct-uploadPOSTGet Cloudflare Stream upload URL

Phase 3: WebSocket Extension

3.1 Message Events with Attachments Current:
// Client sends
{ type: 'message.send', content: 'Hello', requestId: '123' }

// Server broadcasts
{ type: 'message.created', message: { id, content, ... } }
New:
// Client sends
{
  type: 'message.send',
  content: 'Check out this!',
  attachments: [{ type: 'image', uploadId: 'upload_123' }],
  requestId: '123'
}

// Server broadcasts
{
  type: 'message.created',
  message: {
    id,
    content: 'Check out this!',
    attachments: [{ id, type: 'image', url: 'https://...', ... }]
  }
}
3.2 Upload Progress Events
// Server -> Client
{ type: 'upload.progress', uploadId: 'upload_123', progress: 50 }
{ type: 'upload.complete', uploadId: 'upload_123', attachment: {...} }
{ type: 'upload.error', uploadId: 'upload_123', error: 'File too large' }

Phase 4: Mobile Integration

4.1 Update Hooks
// useConvexGroupChat - add attachment support
sendMessage(content: string, attachments?: AttachmentUpload[])

// useVickyChat - add image attachment support
sendMessage(content: string, imageUrl?: string)
4.2 Media Picker
  • Image picker: expo-image-picker
  • Voice recorder: expo-av or react-native-audio-recorder-player
  • Video picker: expo-image-picker with video filter
4.3 Upload Flow
  1. User selects media
  2. Get signed upload URL from backend
  3. Upload directly to R2/Cloudflare Stream
  4. Include attachment URLs in message

Phase 5: Vicky AI Integration

5.1 Image Understanding Vicky should be able to see images:
const vickyAgent = new Agent({
  instructions: `
    You can see images that users share.
    Describe what you see when relevant.
  `,
  model: anthropic(process.env.MASTRA_MODEL, {
    multimodal: true  // Enable vision
  })
})
5.2 Multimodal Streaming Handle mixed text + image responses from Vicky.

File Structure

apps/nestjs/src/
├── chat/
│   ├── dto/
│   │   ├── chat.dto.ts          # Extended with attachments
│   │   └── attachment.dto.ts     # New: Attachment DTOs
│   ├── entities/
│   │   └── chat.entities.ts     # Extended ChatMessage
│   ├── realtime/
│   │   └── chat-realtime.gateway.ts  # Handle attachments
│   └── services/
│       └── chat.service.ts       # Save attachments
├── media/
│   ├── media.module.ts
│   ├── media.controller.ts       # Upload URL endpoints
│   ├── r2.service.ts            # R2 operations
│   └── stream.service.ts         # Cloudflare Stream
└── storage/
    └── storage.service.ts       # Unified storage interface

Environment Variables

# R2
R2_ACCOUNT_ID=
R2_ACCESS_KEY_ID=
R2_SECRET_ACCESS_KEY=
R2_BUCKET_NAME=

# Cloudflare Stream
CLOUDFLARE_ACCOUNT_ID=
CLOUDFLARE_API_TOKEN=

Open Questions

  1. Storage costs: R2 + Cloudflare Stream pricing for media storage
  2. Thumbnail generation: Auto-generate thumbnails for images/videos?
  3. Voice transcription: Transcribe voice messages for Vicky?
  4. Max file sizes:
    • Images: 10MB?
    • Voice: 5min max?
    • Video: 2GB (Stream limit)?

Testing Plan

REST API

# Create room
curl -X POST /v1/chat/rooms ...

# Get signed upload URL
curl -X POST /v1/media/upload-url \
  -H "x-user-id: user_alice" \
  -d '{ "roomId": "room_123", "contentType": "image/jpeg" }'

# Upload message with attachment
curl -X POST /v1/chat/rooms/:roomId/messages \
  -d '{ "content": "Look at this!", "attachments": [{ "id": "att_1", "type": "image", "url": "https://..." }] }'

WebSocket

// Send message with attachment
socket.emit('message.send', {
  content: 'Check out this photo',
  attachments: [{ id: 'att_1', type: 'image', url: 'https://r2.nous.../img.jpg' }],
  requestId: '123'
})

// Receive with attachment
socket.on('message.created', (data) => {
  console.log(data.message.attachments)
})

Last Updated: 2026-05-14