'''
Created on: Mar 09, 2013

@author: qwang

CCrawler means Common Cralwer, is a common framework for web page scrapy and analyze.

external dependences:
    1. rabbitmq, use rabbitmq for message deliver.
    2. redis, use redis for some cache work, like dns cache, domain encoding cache, etc.
    3. scrapy, use some scrapy features in CCrawler for convenience.
    4. mongodb, use mongodb for some storage work.
'''


# TODO remove scrapy and mongodb from dependency.

# TODO get it work first
