Airflow for Fun and Profit
Airflow Data Pipeline
This is a personal POC project for twitter streaming, predicting the sentiment of kewords searched in twitter stream, and discover the current trending tags and topics.
Actually, the indent of the project is to get my hands dirty with verious tools. This projects integrates Kafka for message queuing, Spark Streaming for stream processing, ElasticSearch as a search engine and Kibana as a frontend and visualization tool, also Spacy for text preprocessing, Scikit-learn for features representation, Textblob for sentiment prediction.