This was done so that the IRL robot manipulation tasks could be done fast enough. In the future, we may always need small models mixed with large models for some tasks (e.g., for slow long term planning and fast short term planning), though compute does have a tendency to improve exponentially...