U-Net in TorchSharp

[K1T00]

Currently I’m trying to port this U-Net example https://github.com/Mostafa-wael/U-Net-in-PyTorch to TorchSharp. My results look comparable to the PyTorch example but training in TorchSharp is waaay slower. At least 10x slower. Same model, same batch size, images etc.
Anyone tried something similar and had more success?

[asieradzk]

Did you profile?

[K1T00]

I uploaded the code here -> https://github.com/K1T00/TorchSharpUnet

[K1T00]

Moving the using var d = torch.NewDisposeScope(); into my train/validate functions did a great speedup. I guess my issue is solved now :)

[asieradzk]

This makes sense. I also dispose per epoch, gives the best memory profile.

[K1T00]

If d = torch.NewDisposeScope() ... Are you also specifically calling d.DisposeEverything() ? Or is that not necessary ?

[yueyinqiu]

When d is disposed, d.DisposeEverything will be invoked automatically. So you could create and dispose the DisposeScope inside your loop body (recommended), like:

using TorchSharp;

for (; ; )
{
    using var d = torch.NewDisposeScope();
    // train for an epoch/batch
}

Or you could create the scope in advance and call DisposeEverything at the end of each loop:

using TorchSharp;

using var d = torch.NewDisposeScope();
for (; ; )
{
    // train for an epoch/batch
    d.DisposeEverything();
}

And dispose scopes will hold the reference of the tensors, so it could cause performance problems if you're not using them correctly. For example:

using TorchSharp;

using var d = torch.NewDisposeScope();
for (; ; )
{
    // train for an epoch/batch
}

In this case, all the tensors created in this loop will be kept (unless you explicitly dispose them), because they are always referenced by d.

[K1T00]

Ah I see ... Thank you :)

[K1T00]

I have another question. Let’s say I have two datasets, with pytorch I could do:

trainSet1 = MyCustomDataset(…)
trainSet2 = MyCustomDataset(…)
trainSetCombined = torch.utils.data.ConcatDataset([trainSet1, trainSet2])
trainLoader = DataLoader(trainSetCombined, …)

How can I combine datasets with TorchSharp?

[yueyinqiu]

A possible approach could be:

using TorchSharp;
using static TorchSharp.torch;

using var tensor1 = torch.tensor(new double[,] {
    { 11, 12 },
    { 12, 13 },
    { 13, 14 }
});
using var dataset1 = torch.utils.data.TensorDataset(tensor1, tensor1, tensor1);

using var tensor2 = torch.tensor(new double[,] {
    { 22, 23 },
});
using var dataset2 = torch.utils.data.TensorDataset(tensor2, tensor2);

using var tensor3 = torch.tensor(new double[,] {
    { 31, 32 },
    { 32, 33 },
    { 33, 34 }
});
using var dataset3 = torch.utils.data.TensorDataset(tensor3);

var dataset = new ConcatDataset<IList<Tensor>>([dataset1, dataset2, dataset3]);
for (int i = 0; i < dataset.Count; i++)
{
    var list = dataset.GetTensor(i);
    Console.Write(list[0][0].ToDouble());
    if (list.Count > 1)
        Console.Write(list[1][0].ToDouble());
    if (list.Count > 2)
        Console.Write(list[2][0].ToDouble());
    Console.WriteLine();
}

// 111111
// 121212
// 131313
// 2222
// 31
// 32
// 33

class ConcatDataset<T> : torch.utils.data.Dataset<T>
{
    private readonly torch.utils.data.Dataset<T>[] _datasets;
    public IReadOnlyList<torch.utils.data.Dataset<T>> datasets => _datasets;

    private readonly long[] _cumulativeSizes;
    public IReadOnlyList<long> cumulative_sizes => _cumulativeSizes;

    private static IEnumerable<long> Cumsum(IEnumerable<torch.utils.data.Dataset<T>> datasets)
    {
        var s = 0L;
        foreach (var e in datasets)
        {
            s += e.Count;
            yield return s;
        }
    }

    public ConcatDataset(IEnumerable<torch.utils.data.Dataset<T>> datasets)
    {
        this._datasets = datasets.ToArray();
        if (this._datasets.Length is 0)
            throw new ArgumentException(
                "datasets should not be an empty iterable", nameof(datasets));

        // PyTorch also says 'ConcatDataset does not support IterableDataset'.
        // But it's not our torch.utils.data.IterableDataset in TorchSharp.
        this._cumulativeSizes = Cumsum(datasets).ToArray();
    }

    public override long Count => this._cumulativeSizes[^1];

    private static long bisectRight(long[] a, long x)
    {
        var lo = 0;
        var hi = a.Length;
        while (lo < hi)
        {
            var mid = (lo + hi) / 2;
            if (x < a[mid])
                hi = mid;
            else
                lo = mid + 1;
        }
        return lo;
    }

    public override T GetTensor(long index)
    {
        if (index < 0)
        {
            // Maybe we could use System.Index as the parameter instead of long.
            if (-index > this.Count)
            {
                throw new ArgumentException(
                    "absolute value of index should not exceed dataset length",
                    nameof(index));
            }
            index = this.Count + index;
        }

        var datasetIdx = bisectRight(this._cumulativeSizes, index);
        long sampleIdx;
        if (datasetIdx == 0)
            sampleIdx = index;
        else
            sampleIdx = index - this._cumulativeSizes[datasetIdx - 1];
        return this._datasets[datasetIdx].GetTensor(sampleIdx);
    }
}

But you may find that torch.utils.data.DataLoader does not accept our ConcatDataset. One solution currently is to replace all the torch.utils.data.Dataset<T> above with torch.utils.data.Dataset or torch.utils.data.IterableDataset, as you need.

I would create a pull request and have a discuss about this soon.

[yueyinqiu]

#1354