At Twitter, we built a data pipeline that processes tweet data with backend services written in Finagle and Akka, and eventually populates dashboards created using React JS and D3.js. D3.js was used to create the data visualizations. The application code was deployed on GCP Kubernetes Engine. 

We integrated Finagle and Akka to create a seamless backend. This involved leveraging Akka’s Actor model to handle high concurrency, as well as its features for backpressure and fault tolerance. Finagle’s load balancing features were used to distribute requests. The system was scaled horizontally  

We used MySQL to store the processed. We followed best practices for storing 100s of thousands of rows of tweet data in MySQL. That included indexing, partitioning, normalization, archiving, performance tuning, and monitoring. The data had to follow the rules of data governance which included data security, data quality, and compliance.