FILTERAI infra
Jev-based LLM DataOps Automation Pipeline
Rena Gao@weiweigao2222
This project introduces an automated pipeline for LLM data operations, encompassing data upload, selection, evaluation, model training, and performance assessment, designed to reduce wasted time in these processes.
A lot of people will waste time in both pre training and post training of LLM in general domain and domain specific training during data pipeline, and performance evaluation, even for label consistency checks. Now, I’ve created a RSI based Jev @typesafeai @CompleteSkeptic empowered automation pipeline including large scale raw data upload(both text modality and multimodality) data selection, data evaluation, model auto-self training in large scale GPU settings and model performance evaluation in general/ domain specific tasks. Open sources pipeline code link for this pipeline, already successfully operated in large scale tpu settings: github.com/RenaGao/jev-dataops
#Jev #LLM #dataefficiecny
Related builds
