Advancing AI Models for Enhanced Efficiency and Accessibility
This comprehensive research agenda aims to address the critical challenges posed by current AI scaling laws, paving the way for more efficient, accessible, and environmentally sustainable AI technologies. By focusing on these key areas, we strive to contribute to the development of a robust and competitive AI ecosystem in South Korea and beyond.
Background and Rationale
The rapid evolution of artificial intelligence (AI) models has brought unprecedented capabilities but also significant challenges. As models grow in complexity, they face increasing limitations in terms of computational resources, costs, and environmental impact. The Neural Scaling Law indicates that these challenges will escalate exponentially as AI models continue to expand.
The development of large-scale AI systems has become concentrated among major technology corporations, creating substantial barriers to entry due to the immense computational and capital requirements1. This trend has led to a visible monopolization of the AI value chain, with global tech giants dominating both cloud infrastructure and AI platforms.
In South Korea, concerns have been raised about the potential subordination of domestic AI and platform industries to international tech giants, particularly following the widespread adoption of GPT-series models. The Korea Fair Trade Commission (KFTC) has highlighted structural factors that could stifle fair competition and harm consumers in the rapidly evolving generative AI market.
To address these challenges and foster innovation, research must focus on advancing AI models through efficient training data construction, lightweight learning models, and improved neural architectures.
Research Objectives
Our primary goal is to develop novel learning methodologies and model architectures that transcend the limitations imposed by the neural scaling law, optimizing both performance and efficiency in AI model training and operation.
Key objectives include:
- 1. Inference Cost Optimization: Enhance existing trained models through lightweight architectures and knowledge distillation techniques.
- 2. Training Cost Optimization: Implement zero-shot inference capabilities, distributed training paradigms, and selective data learning strategies to improve model scalability and accelerate learning processes.
- 3. Innovative Model Development: Design new architectures and large-scale multimodal models with linear computational complexity.
- 4. High-Quality Dataset Construction: Develop compact, high-fidelity ground truth datasets for efficient training across various multimodal domains.
Research and Development Phases
Phase 1: Foundation Building and Initial Algorithm Development
- Optimize transformer models using heterogeneous bit quantization, prompt distillation, and multi-criteria token merging techniques.
- Develop zero-shot inference modules, real-time diffusion generation models, and distributed training frameworks to enhance model scalability and learning speed.
- Create large-scale multimodal models with linear computational complexity, leveraging graph signal processing and object-centric tokenization algorithms.
- Implement dataset distillation algorithms for efficient training of multimodal baseline models.
Phase 2: Advanced Model Implementation and Optimization
- Develop state-space model-based, training-free merging techniques to enhance data access and computational efficiency.
- Implement mega-model scaling, robust zero-shot inference, and hybrid distributed training methodologies.
- Design efficient cyclic and large-scale multimodal models utilizing state-space models and shear fusion tokenization techniques.
- Create advanced dataset pruning and distillation algorithms, alongside 3D information-based data generation systems.
