Group Chat + Multimedia Integration Plan
Status: Planning | Current chat only supports text messages, need to extend for media.
Current State
NestJS Chat (apps/nestjs/):
Mobile App:
- Message types defined: text, image, bento, poke, schedule, gift
- Cloudflare Stream integration exists for video
- Hooks:
useVickyChat, useChatStream, useConvexGroupChat
Architecture Overview
Required Work
Phase 1: Data Model Extension
1.1 Extend ChatMessage Entity
Current:
type ChatMessage = {
content: string // text only
}
New:
type ChatMessage = {
id: string
roomId: string
senderId: string
senderKind: 'human' | 'ai'
senderName: string
content: string
attachments: Attachment[]
createdAt: Date
}
type Attachment = {
id: string
type: 'image' | 'voice' | 'video'
url: string
thumbnailUrl?: string // for video
duration?: number // for voice/video
mimeType: string
size: number
}
1.2 Create Attachment DTOs
// Image attachment
class ImageAttachmentDto {
id: string
url: string
width: number
height: number
mimeType: string
}
// Voice attachment
class VoiceAttachmentDto {
id: string
url: string
duration: number // seconds
mimeType: string
waveformUrl?: string
}
// Video attachment
class VideoAttachmentDto {
id: string
url: string
thumbnailUrl: string
duration: number
mimeType: string
}
2.1 R2 Setup for Images & Voice
Configure R2 bucket for:
- User-uploaded images (JPEG, PNG, GIF, WebP)
- Voice messages (MP3, WebM, Opus)
// apps/nestjs/src/storage/r2.service.ts
export class R2Service {
async uploadImage(roomId: string, userId: string, file: Buffer): Promise<string>
async uploadVoice(roomId: string, userId: string, file: Buffer): Promise<string>
async getSignedUploadUrl(roomId: string, userId: string, contentType: string): Promise<string>
async getSignedDownloadUrl(key: string): Promise<string>
}
2.2 Cloudflare Stream for Video
Video uploads via direct upload URL:
// apps/nestjs/src/storage/stream.service.ts
export class StreamService {
async createDirectUploadUrl(): Promise<{ uid: string; uploadUrl: string }>
async getVideoUrl(uid: string): Promise<string>
async getThumbnailUrl(uid: string): Promise<string>
}
2.3 Media Endpoints
| Endpoint | Method | Description |
|---|
/v1/media/upload-url | POST | Get signed R2 upload URL |
/v1/media/voice/upload-url | POST | Get signed R2 upload URL for voice |
/v1/media/video/direct-upload | POST | Get Cloudflare Stream upload URL |
Phase 3: WebSocket Extension
3.1 Message Events with Attachments
Current:
// Client sends
{ type: 'message.send', content: 'Hello', requestId: '123' }
// Server broadcasts
{ type: 'message.created', message: { id, content, ... } }
New:
// Client sends
{
type: 'message.send',
content: 'Check out this!',
attachments: [{ type: 'image', uploadId: 'upload_123' }],
requestId: '123'
}
// Server broadcasts
{
type: 'message.created',
message: {
id,
content: 'Check out this!',
attachments: [{ id, type: 'image', url: 'https://...', ... }]
}
}
3.2 Upload Progress Events
// Server -> Client
{ type: 'upload.progress', uploadId: 'upload_123', progress: 50 }
{ type: 'upload.complete', uploadId: 'upload_123', attachment: {...} }
{ type: 'upload.error', uploadId: 'upload_123', error: 'File too large' }
Phase 4: Mobile Integration
4.1 Update Hooks
// useConvexGroupChat - add attachment support
sendMessage(content: string, attachments?: AttachmentUpload[])
// useVickyChat - add image attachment support
sendMessage(content: string, imageUrl?: string)
4.2 Media Picker
- Image picker:
expo-image-picker
- Voice recorder:
expo-av or react-native-audio-recorder-player
- Video picker:
expo-image-picker with video filter
4.3 Upload Flow
- User selects media
- Get signed upload URL from backend
- Upload directly to R2/Cloudflare Stream
- Include attachment URLs in message
Phase 5: Vicky AI Integration
5.1 Image Understanding
Vicky should be able to see images:
const vickyAgent = new Agent({
instructions: `
You can see images that users share.
Describe what you see when relevant.
`,
model: anthropic(process.env.MASTRA_MODEL, {
multimodal: true // Enable vision
})
})
5.2 Multimodal Streaming
Handle mixed text + image responses from Vicky.
File Structure
apps/nestjs/src/
├── chat/
│ ├── dto/
│ │ ├── chat.dto.ts # Extended with attachments
│ │ └── attachment.dto.ts # New: Attachment DTOs
│ ├── entities/
│ │ └── chat.entities.ts # Extended ChatMessage
│ ├── realtime/
│ │ └── chat-realtime.gateway.ts # Handle attachments
│ └── services/
│ └── chat.service.ts # Save attachments
├── media/
│ ├── media.module.ts
│ ├── media.controller.ts # Upload URL endpoints
│ ├── r2.service.ts # R2 operations
│ └── stream.service.ts # Cloudflare Stream
└── storage/
└── storage.service.ts # Unified storage interface
Environment Variables
# R2
R2_ACCOUNT_ID=
R2_ACCESS_KEY_ID=
R2_SECRET_ACCESS_KEY=
R2_BUCKET_NAME=
# Cloudflare Stream
CLOUDFLARE_ACCOUNT_ID=
CLOUDFLARE_API_TOKEN=
Open Questions
- Storage costs: R2 + Cloudflare Stream pricing for media storage
- Thumbnail generation: Auto-generate thumbnails for images/videos?
- Voice transcription: Transcribe voice messages for Vicky?
- Max file sizes:
- Images: 10MB?
- Voice: 5min max?
- Video: 2GB (Stream limit)?
Testing Plan
REST API
# Create room
curl -X POST /v1/chat/rooms ...
# Get signed upload URL
curl -X POST /v1/media/upload-url \
-H "x-user-id: user_alice" \
-d '{ "roomId": "room_123", "contentType": "image/jpeg" }'
# Upload message with attachment
curl -X POST /v1/chat/rooms/:roomId/messages \
-d '{ "content": "Look at this!", "attachments": [{ "id": "att_1", "type": "image", "url": "https://..." }] }'
WebSocket
// Send message with attachment
socket.emit('message.send', {
content: 'Check out this photo',
attachments: [{ id: 'att_1', type: 'image', url: 'https://r2.nous.../img.jpg' }],
requestId: '123'
})
// Receive with attachment
socket.on('message.created', (data) => {
console.log(data.message.attachments)
})
Last Updated: 2026-05-14