Model evaluation and deployment orchestrator:
Objective:
Evluation pipeline of a deep learning model usually has a lot of reusable components. For example, for any object detection model, the way you compute metrics would remain the same irrespective of the model you use or dataset you are evaluating the model on. Having a plug and run type of evaluation framework would save Machine Learning Scientists anf Engineers a ton of time by automating the design, orchestration and logging of the results. In this project, I developed an orchestration library using a Directed Acyclic Graph (DAG) data structure. This library allowed the user to design pipelines using configuration files like yaml and automating the execution and monitoring of the pipeline. The use of the library was extended beyond single model evaulation for use cases like multi-model multi-dataset performance summary and automated CI/CD of evaluated models.
Contributions:
- Developed a python library for design and orchestration of data pipelines using DAG data structure.
- Adding state saver and logging facilities to automatically keep track of pipeline runs and restart failed data pipeline runs.
- Added generic code blocks and adaptors for dataset loader, summary generator, model evaulation and CI/CD.
Result:
- Reduced deep learning model evaluation and CI/CD pipeline design time by 90%.
- Facilitated comparison of evaluation metrics of multiple models on multiple datasets.
- Facilitated automated unit testing and CI/CD of models to production environment.