Solving complex tasks using reinforcement learning (RL) from sparse rewards and demonstrations is appealing since one can avoid reward shaping. However, past approaches often require heavy curriculum engineering, extensive training time, and/or infeasible amounts of demonstrations. In this paper, we investigate how scalable RL algorithms, high-throughput GPU simulations, and demonstration-based auto-curricula can address these challenges. We show that our method, FlashRFCL, can solve a variety of high-dimensional robotic tasks faster in addition to relying on just a single demonstration each. We perform extensive ablation studies on the design choices of FlashRFCL, highlighting how each component impacts training speed and performance compared to prior work.