We are adding byte-sized batch support in the file reader and fragment reader for inline data. However, we should also apply the budget for blob reads when lance is asked to load the blob data.
We might need to apply the budget twice. For example, if the budget is 1MB then we load 1MB worth of description data (and other columns). Then we join in blob data and split those batches further. It should be cappable at 2x oversubscription.
We are adding byte-sized batch support in the file reader and fragment reader for inline data. However, we should also apply the budget for blob reads when lance is asked to load the blob data.
We might need to apply the budget twice. For example, if the budget is 1MB then we load 1MB worth of description data (and other columns). Then we join in blob data and split those batches further. It should be cappable at 2x oversubscription.