Conference proceeding
POSUM: A Portfolio Scheduler for MapReduce Workloads
Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018, pp.351-357
22 Jan 2019
Abstract
MapReduce ecosystems are (still) widely popular for big data processing in data centers. To address the diverse non-functional requirements arising from many and increasingly more sophisticated users, the community has developed many scheduling policies for MapReduce workloads. Although some individual policies can dynamically optimize for single and stable performance objectives, such as minimizing runtime or cost, or meeting deadlines for realtime-jobs, it seems unlikely that individual policies will remain competitive for increasingly more dynamic workloads and objectives. In contrast, in this work we investigate the ability to dynamically balance performance and cost of a portfolio scheduler for MapReduce workloads. To this end, we design and implement a portfolio scheduling technique, that is, a system capable of adapting to the current workload characteristics and target objectives by periodically evaluating its set of potential policies, and of switching to »the best» policy that targets the current system state. We implement and evaluate our system with real-world experiments on a workload containing a mixture of real-time and batch jobs, with the purpose of minimizing deadline violations, while keeping batch job slowdown in check. Our results show that POSUM is a promising alternative: it can out-perform the individual policies of its portfolio for the combined optimization goal, even without precise predictions.
Metrics
5 Record Views
Details
- Title
- POSUM: A Portfolio Scheduler for MapReduce Workloads
- Creators
- Maria A VoineaAlexanu UtaAlexanu IosupYang SongBing LiuKisung LeeNaoki AbeCalton PuMu QiaoNesreen AhmedDonald KossmannJeffrey SaltzJiliang TangJingrui HeHuan LiuXiaohua Hu
- Publication Details
- Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018, pp.351-357
- Conference
- 2018 IEEE International Conference on Big Data, Big Data 2018
- Publisher
- Institute of Electrical and Electronics Engineers Inc
- Number of pages
- 1
- Resource Type
- Conference proceeding
- Language
- English
- Academic Unit
- Information Science (Informatics)
- Identifiers
- 991019189084004721