Memory requirements for a new model

[DanrunFR]

Hello,
I am trying to train a new ACE model on a 3.5m word corpus. I want to use the same embedding configuration of the CoNLL 2003 model. I have tried a few times, but I kept running into memory problems. I had about 250G CPU memory but it didn't even get through 3 embedding models. I was wondering how much memory would you recommend using to train my model ?

[DanrunFR]

And also how much time would the training take ? I'm asking because I'm using our lab's computation facility, and I need to reserve the resources I need.

Thank you in advance for your response.

[wangxinyu0922]

Hi, to run conll 2003 dataset, it takes about 40~60G CPU memory. It seems that a 3.5m word corpus is about 10 times larger than the conll 2003 dataset. So it takes about 400G cpu memory to run the code. I suggest you may remove the flair embeddings (especially the multilingual embeddings) to reduce the feature size.

[wangxinyu0922]

For training, it takes about 2~4 days to train conll 2003 dataset on a single Tesla V100 GPU. You may reduce the training epoch to reduce the training time.