Data extraction from various databases using python. We've used pandas library in python for data extraction from databases and building pandas data frames from the extracted data.
Extracted data is written to csv and parquet files
Data Extraction using Python
Processing the Extracted data using PySpark
Extracted parquet file is processed using SparkSQL from PySpark library. Spark data frames are built by querying the parquet files using SparkSQL.
Building API for Front-end calls
These spark data frames are used by Flask for building RestAPI. Spark data frames are converted to json using Flask jsonify. We'll build both GET and POST methods using Flask depending upon the requirements
Visualizations built in Angular8 using D3.js framework
API will be called by the Front-end for getting data. This data will be used by Front-end for building visualizations like bar charts, line charts, stacked bar charts, pie charts.
Data filters will also be applied to these visualizations.
Data filters will also be applied to these visualizations.