Example notebook

[johnyquest7]

Could you please add a demo notebook ?
TIA

[lessw2020]

Hi @johnyquest7,
Yes, that's a good idea.
Probably a week or two out, but will add it to the todo list once I get into full ImageNet testing as I will have time while that testing is running.
Thanks for the feedback and will leave this thread open to track it.

[johnyquest7]

Thanks

[rsomani95]

@lessw2020 This is a really exciting repo, thank you for sharing your work.

There's a few points I'd love to get your insight on, and I think these would be a good addition to the notebook / README:

• How should num_batches_per_epoch be changed if training on multi-GPU setups?
• What are some good principles for thinking about picking an LR when using madgrad vs adamw?
• I noticed you have an LR Finder in the works too. Is it advisable to use your implementation rather than Learner.lr_find() if one is using fastai, or are both equivalent?

[rsomani95]

Thanks for the quick response and the insight!

Once you've got some of these benchmarks done, I'd love to conduct some tests on training object detectors on COCO. For some context - I've a pretrained mobilenetv3 that I'll be training on COCO from scratch with an FPN + RetinaNet arch, and am looking to explore the best options for hparams.

I used your previous Ranger implementation with much success for transfer learning, but I'm a bit more cautious to when it comes to training from scratch (both because of long runtimes and lesser experience / intuition).
That being said, I see that most papers / pretrained models (detectron2, mmdet) stick to the paper's hparams, and in turn a lot of papers' hparams are simply copied over from previous papers, which makes sense in a research setting to keep things constant, but not as much in a practical setting. I think there's definitely lots of unexplored territory, and based on your experiments on ImageNet, I think we can be optimistic about seeing training gains there as well.

---

I've noticed that a lot of papers use SGD w/ 1e-4 or Adam w/ ~3e-3 / 1e-4 for ImageNet training, but on the object detection side of things, a popular choice seems to be SGD w/ 1e-2. However, those params are always the settings for a specific image size with a batch size of 2 and an 8xV100 setup... which is far beyond my reach haha. In torchvision's official scripts, they have a formula for changing these based on hardware, in case this is a helpful reference:

https://github.com/pytorch/vision/blob/389109a8126a7c4affea86c37de53bb485c1c633/references/detection/train.py#L8-L9

Once your LR finder is ready (I see here that it's not recommended to use yet), would be great if a simple demo of that is in the notebook as well! That'd be great to have hardware specific intuition.

Thanks again!

[rsomani95]

@lessw2020 curious if you had a chance to look at it.

I think we should pass in num_batches_per_epoch ~= len(train_dl) // NUM_GPUs to ensure that self.total_iterations is computed correctly?

Currently, I'm running on 3 GPUs and total_iterations is computed as about thrice the normal amount

[lessw2020]

Hi @rsomani95,
Sorry for the delay - I still have not had a chance to address multi-gpu properly as currently finishing the Ranger21 arxiv paper. (hope to have it ready in the next week, it's nearly complete).
That said, your intuition here seems to be correct of simply dividing by num gpus should address the iterations aspect.
I've got access to a 4 GPU setup so after the paper I can do a more thorough job of setting Ranger21 up for it in a more automated fashion.
Just for reference, are you using DistributedDataParallel (DDP) for your training (i.e. instead of DP?).

[rsomani95]

No worries @lessw2020 :)
That's correct, I'm using DDP

[triciardvn]

How to use Ranger optimizer with YOLOV8?

This is the error for my code since I want to try the "Ranger" optimizer instead of the existing available ones using YOLOV8.

NotImplementedError: Optimizer 'Ranger' not found in list of available optimizers [Adam, AdamW, NAdam, RAdam, RMSProp, SGD, auto].To request support for addition optimizers please visit https://github.com/ultralytics/ultralytics.

How can I integrate this new optimizer as I want to do an object detection for etrikes and ebikes (I want to start training it). I already tried this code:

!pip install git+https://github.com/lessw2020/Ranger21.git

from ultralytics import YOLO
import os

# Load your model
model = YOLO('yolov8n.pt')

# Define hyperparameters
hyperparameters = {
    'lr': 0.001,
    'weight_decay': 0.0005,
    'momentum': 0.937
}

# Get the number of training images
num_train_images = len(os.listdir('/content/drive/MyDrive/e/train/images'))
batch_size = 16
num_batches_per_epoch = num_train_images // batch_size
num_epochs = 10

# Setup training
model.train(data='/content/drive/MyDrive/e/data.yaml', epochs=num_epochs, imgsz=640, batch=batch_size, optimizer='Ranger')

Please let me know what to do ASAP. Thank you.

Note: I need to use the ranger optimizer as I need it for my thesis.