案例
- News Feeds
- Stats Server
- Web Crawler
- Amazon Product Page
News feed（信息流）
Define feed
Organize
- aggregate（分類）
- dedup（去重）
- sort（排序）
Level1.0
Database Schema:
- User
- Friendship
- News
GetNewsfeed:
- merge news
- Newsfeed vs News
Why bad?
100+ friends
1Query-->Get friends list
1Query-->
SELECT news
WHERE timestamp>xxx
AND sourceid IN friend list
LIMIT 1000
IN is slow
Either Sequential scan or 100+ index queries
Level 2.0
Pull vs Push
Pull:Get news from each friend,merge them together.(NewsFeed generated when user request)
Push:NewsFeed generated when news generated.(we have another table to store newsfeed,may cause duplicate news)
Push:
1Query to select latest 1000 newsfeed.
100+ insert queries(Async)
Disadvantage:News Delay.
Level 3.0
Popular star(Justin Bieber)
Flowers 13M+
Async Push may cause over 30 minutes(13M+ insertions,delay too long)
Push+Pull
for popular star,don't push news to flowers
for every newfeed reqiest,merge non-popular user newfeed(push) and popular users newsfeed(pull)
Level 4.0
Push disadvantage
- Realtime
- Storage(Duplicate)
- Edit
Go back to PULL:
- Cache users' latest (14days) news
- Broadcast multiple request to multiple servers(Shard by userld).
- Merge & sort newsfeed
- Cache newsfeeds for this user with timestamp
Click Stats Server
How are click stats stored
A poor candidate will suggest write-back to a data store on every click
A good candidate will suggest some form of aggregation tier that accepts clickstream data,aggregates it,and writes back a persistent data store periodically
A great candidate will suggest alow-latecy messaging system to bugger the click data and transfer it to the aggregation tier.
If daily,storing in hdfs and running map/reduce jobs to compute stats is a reasonable approach
If near real-time,the aggregation logic should compute stats
PS：要如何統計鼠標點擊的次數以及相關區域呢？普通的程序員會將每次點擊的數據（log）直接存儲在數據庫一層。比較好的程序員會在前段與數據庫間加一個中間層，為點擊的數據流做一次聚合，每隔一段時間（1分鐘或10分鐘）做一次刷新，存儲到數據庫，大大減輕了后端的壓力。優秀的程序員綜合以上的兩種情況，對于數據量很大，實時性效果不高的情況下，可以通過分布式的批處理方式，將刷新聚合層的時間定位一天。對于時效性強的要適當縮短刷新時間。
Cache Requirement
- When a request comes look it up in the cache and if it hits then return the response from here and do not pass the request to the system.
- If the request is not found in the cache then pass it on to the system.
- Since cache can only store the last n requests,Insert the n+1th request in the cache and delete one of the older requests from the cache
- Design one cache such that all operations can be done in O(1)-lookup,delete and insert.
PS:如何設計cache（LRU設計相關）：
- 在層中緩存部分請求的處理方式，如果接收的請求在層中存在對應的處理方式，則無需把請求發送到后端系統
- 如果在層中找不到對應處理，則發送需求到后端
- 以定長隊列的形式緩存，緩存最近的n個需求，頭進尾出
- 將層中的匹配操作算法控制在O(1)范圍
Web Crawler
爬蟲
Amazon Product Page
The product page includes information such as
- product information
- user information
- recommended products(what do other customers buy after viewing this item,recommendations for you like this product,etc)
Reference
- http://highscalability.com
- The Log:What every software engineer should know about real-time data's unifying abstraction
- Job Interviews:How should I prepare system design questions for Goole/Facebook Interview?
- HOW TO ACE A SYSTEMS DESIGN INTERVIEW
- <Design Pattern>
- <Design_Oatterns_For_Dummies.pdf>
- http://www.hiredintech.com/app
