Desigining Chat service

System requirement?

1. Is the service peer to peer or group chat allowed
2. Is the service using end-to-end encrytpion (Watsapp) vs stroring chat unencrypted in the servers (FB ,default)
3. Media sharing allowed. Most likely you need to use a blob storage for this as it is cheaper than using DB. Think amazon S3 buckets
4. Message ordering in the chat should be consistent across all participant? timestamping can be used for ordering the chat

Key Aspect

1. You need to make sure your client (mobile app) is connected to one of the fixed backend server. This is deviation from the stateless nature of HTTP in which everytime client sends a request any random server will generate and send response. This is usually achieve using Websocket (which is bidirectional connection). This gives facility for server to push content to the client. Another alternative is client Polling perioidically but polling is expesive. You can connect say 10K client to a backend server. Use sharding to evenout the load
2. Use memcache or reddis to find out which client is served by which machine (node). if server X is responsible for a user A and A sends message to user B. B is handled by server Y. now there should be some fast way to query which server B is connected to. Something like below is good schema: UserID;ServerID(IP+PORT);TimeStampOFLastHeartBeat
3. UserConnection table schema UID1:UID2;conversation_id;encryptionKey (AES)
4. Can you use cassandra to store chats (Why? Its a AP system not a CP system so you can have in-consistent view at times). Better use CP system like HBASE. Why chats are usually immutable so there is no need of trasactional support (Sql). No-Sql DB like HBASE is more optimized for heavy write cases “conversation_id;text/media_url;timestamp”

Chat service design by Tushar Roy