torch.cuda.OutOfMemoryError: CUDA out of memory on customize dataset

[reality3024]

Thank you for sharing this project. I'm currently trying to run your code with my own dataset, but I keep encountering out-of-memory (OOM) issues when using the example config settings you provided — even though my dataset is relatively small (just over 5,000 images).

I’m running on a single RTX 5090 GPU with 32GB of memory. I was wondering if you could share the actual training parameters, memory usage, and hardware configuration you used when training on datasets like Office-31 or VisDA-2017?

[enesdoruk]

Hi, @reality3024

We tested on official datasetss with default hyperparameters. we have 32 GB A4000 GPU and we did not get any problems about memory. Did you try lowe batch size

[reality3024]

@enesdoruk Thanks for reply, I'm using the config you provide in the source code which is like as follow:

--train_batch_size 64
--eval_batch_size 4

And actually when I lower the train_batch_size to 32 it works properly, but what I really concern is when we lower the train_batch_size it can actually effect the performance of the model, which means I cannot completely reproduce your result on your model.

So I tried setting --gradient_accumulation_steps=2 to avoid performance effect, it works too but consume 27GB of my GPU memory, Which means it's possible with --gradient_accumulation_steps=1 it'll use up over 32GB, making impossible to run my dataset with default config from yours. I'm wondering if I'm using the same configs with yours which set train_batch_size 64?

[enesdoruk]

@reality3024, Sorry for late answer. Did you get different results? If you mention about your custom dataset, i can not directly give answer, you should arrange the parameters. and also, your custom dataset properties may be not fit our model.