duoan/TorchCode

The issue title 'FSDP training loop' without further body content suggests a request or discussion point regarding the implementation of Fully Sharded Data Parallel (FSDP) within TorchCode. FSDP is a critical advanced distributed training technique for large models. Its inclusion or discussion indicates a demand from users for learning and practicing state-of-the-art model scaling methods. For a platform focused on 'implementing from scratch,' integrating FSDP would significantly elevate its relevance for advanced PyTorch practitioners, addressing the complexities of training large-scale models efficiently. This points to a strategic opportunity to expand the curriculum into high-performance computing for deep learning.