Add Multi-tiered Checkpointing Support to Pathways

[SujeethJinesh]

Description

Adds Multi-tiered checkpointing support to MaxText. This will also work with Elastic Training Pause & Resume.

This feature is expected to roughly increase the goodput by ~6.5% and is only dependent on Orbax.

Note: Orbax hasn't cut a version with MTC support and bug fixes yet, so there will be a follow on change to update requirements.txt.

Tests

Tested internally against candidate Orbax.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

[codecov]

Codecov Report

❌ Patch coverage is 41.17647% with 10 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/maxtext/common/checkpointing.py	33.33%	5 Missing and 1 partial ⚠️
src/maxtext/utils/train_utils.py	20.00%	3 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!