[Marocco2]

Fixes #533
Fixes #580

By using Java libraries, this PR will approx speed up download up to 10x compared to DART implementation.
This was tested on Pixel 8 Pro on a 1Gbps Wi-Fi 6 connection

Before

• 1.6.11: ~4MB/s

After

• First commit PR: ~32MB/s
• Latest commit PR: ~84MB/s

Known issues

• It could throw up some error messages when cancelling downloads or checking in background (fixed with 4f5201b )
• A comprehensive testing is highly recommended

Disclaimer

This PR is made with the assistance of Github Copilot. Commits history of this PR is shown for transparency but I prefer that should be squashed once PR is merged

[Marocco2]

On my fork there are unsigned builds ready to be tested.
I set this repo to draft so everybody could comment on and ask needed changes before merging this

[Marocco2]

Other notes

I'm using Obtainium also on my PC using WSA and on 1.6.11 the DART implementation is relatively fast. I suspect that the core issue is a CPU or a memory bottleneck of some sorts. Fortunately this implementation should benefit both speed and battery life on mobile devices

[master-bob]

Definitely cool that the download speed will be improved!

I have some general comments, but someone else will need to check the implementation.

I think the downloadTimeoutError string needs to be defined in the translation files under assets/translations/*. I didn't see it defined anywhere. There are some other strings used (in MainActivity.ky:startNativeDownload, but I presume they are internal error messages so translations are unneeded.

On commits, I have opinions.

• Regarding the whole squashing thing, in general I think it's best to leave things unsquashed if appropriate as it allows for easier changes in the future; particularly with individual feature implementations.
• I'd also personally tag all the commits with the issue or PR number (probably Fix: 10x download speed by using native (Java) libraries #3214) as it will link them all together and help anyone digging through the commits in the future understand that they are together; I think this is good practice as well.

Feel free to disregard the commit opinions. Everyone has their own ideas. ^_^

[deminearchiver]

I know this may sound like I'm overcomplicating things, but maybe it would've been better to use a production-ready downloader (like aria2 or libcurl, for example)?

Having a lot of platform-specific code in a Flutter app can be hard to maintain in the future.

Also, using a native library for downloads would not only give a substantial performance improvement (native multithreading, synchronous FFI calls, etc.), but also delegate the fetching/downloading logic to the library, helping avoid potential problems in some edge cases.

[Marocco2]

Definitely cool that the download speed will be improved!

I have some general comments, but someone else will need to check the implementation.

I think the downloadTimeoutError string needs to be defined in the translation files under assets/translations/*. I didn't see it defined anywhere. There are some other strings used (in MainActivity.ky:startNativeDownload, but I presume they are internal error messages so translations are unneeded.

On commits, I have opinions.

• Regarding the whole squashing thing, in general I think it's best to leave things unsquashed if appropriate as it allows for easier changes in the future; particularly with individual feature implementations.
• I'd also personally tag all the commits with the issue or PR number (probably Fix: 10x download speed by using native (Java) libraries #3214) as it will link them all together and help anyone digging through the commits in the future understand that they are together; I think this is good practice as well.

Feel free to disregard the commit opinions. Everyone has their own ideas. ^_^

I agree that error messages could be translated, I do not object about it. Currently I've pushed an update that refactored a lot of code inside MainActivity to get the same benefits when pulling updates in background, so check that out

I know this may sound like I'm overcomplicating things, but maybe it would've been better to use a production-ready downloader (like aria2 or libcurl, for example)?

Having a lot of platform-specific code in a Flutter app can be hard to maintain in the future.

Also, using a native library for downloads would not only give a substantial performance improvement (native multithreading, synchronous FFI calls, etc.), but also delegate the fetching/downloading logic to the library, helping avoid potential problems in some edge cases.

I do agree to an extent: the scope of this project (undisclosed, my bad) was to solve my specific issue with Obtainium downloading speed on my phone that was unreasonably slow. I've picked up the easiest and fastest way possible to achieve that goal.
Considering that Obtainium is Android-first, it wasn't unreasonable when I made this PR to get platform specific for this issue and get rid of DART library when my commits didn't work as intended.
I do not intend to change this PR to implement another library atm, but feel free to open another one if you got an implementation available

Thank you for your replies!

[Marocco2]

Progress update

So, I've unlocked more throughput performance on downloads and I've reached nearly 90% of my internet speed compared to only 50% with previous commits.
I've also rebased the PR with the latest HEAD.
Right now, main focus is on getting background updates to work properly and finally adding translations when needed

[Marocco2]

@ImranR98 Is there a way to faster debug background updates? Using "Run background check now" appears inconsistent when I got some apps that needs to be updated

[Marocco2]

Progress update

After exploring a couple of alternatives, due to how Flutter works underneath I've decided to refactor the PR to be a Flutter plugin properly. This is intended for background updated as it wouldn't work without tampering generated code or throw out Flutter entirely from this project.

Due to the nature of GitHub's PR, I've decided to add a packages folder to store the native_download plugin and not cross-referencing it to another repo.
I hope this won't neglect my contribution on this issue and I hope this PR is gonna be merged shortly

[deminearchiver]

I recommend considering to change the plugin from using Method Channels to JNI? It could give a considerable performance boost when sending data between native and app code. Also fully synchronous, supports async via Kotlin's suspend functions, etc.

[Marocco2]

I recommend considering to change the plugin from using Method Channels to JNI? It could give a considerable performance boost when sending data between native and app code. Also fully synchronous, supports async via Kotlin's suspend functions, etc.

I didn't find particular regressions doing Method Channels but I'll take a look if there is some performance left on the table

[Marocco2]

I recommend considering to change the plugin from using Method Channels to JNI? It could give a considerable performance boost when sending data between native and app code. Also fully synchronous, supports async via Kotlin's suspend functions, etc.

I tried to see a JNI version what it could look like but I don't think it's worth it as it would modify pubspec needlessly while still using Method Channels

[Marocco2]

@ImranR98 This is ready to be merged

[deminearchiver]

Is there any chance the native_download plugin could be migrated to the new Flutter Kotlin-based Gradle templates? From what I remember, a recent release has changed the plugin templates to generate Gradle DSL .gradle.kts files instead of .gradle. This change would align the plugin with the app's own platform-specific code.

[Marocco2]

Is there any chance the native_download plugin could be migrated to the new Flutter Kotlin-based Gradle templates? From what I remember, a recent release has changed the plugin templates to generate Gradle DSL .gradle.kts files instead of .gradle. This change would align the plugin with the app's own platform-specific code.

I'll check if I can do quickly

[deminearchiver]

I'll check if I can do quickly

@Marocco2, I'd recommend backing up the android directory, then running flutter create --template plugin --org <ORG> . in the plugin directory (fill in your values), and then manually porting derivations from the backup. That's how I did it for most of my plugins.

[Marocco2]

I'll check if I can do quickly

@Marocco2, I'd recommend backing up the android directory, then running flutter create --template plugin --org <ORG> . in the plugin directory (fill in your values), and then manually porting derivations from the backup. That's how I did it for most of my plugins.

Done 👌

[flexxxxer]

Hmmm... this PR may conflict with certificate pinning feature added not so long ago as opt-in feature. Will test later

[flexxxxer]

Problems I see:

• this implementation uses legacy HttpURLConnection instead of OkHttp
• (I suspect) this implementation currently broken with RuStore because there is no trust to Mintsifry CA for RuStore
• (I suspect) this implementation currently does not honor certificate pinning feature because I cant see it any mechanisms utilization to achieve certificate pinning in this implementation

I would prefer this implementation to be using OkHttp in the first place, cant say anything for Obtainium developer

[Marocco2]

Hmmm... this PR may conflict with certificate pinning feature added not so long ago as opt-in feature. Will test later

I've tried the "broken" 1.6.14 and I did encounter the same github.com bug when I've tried to download something from a repo, so the certificate pinning pipeline is present somewhat. I didn't try RuStore, but if you'll see an issue while testing, feel free to add code suggestions to this PR

I would prefer this implementation to be using OkHttp in the first place, cant say anything for Obtainium developer

AFAIK, OkHttp is a third-party library. I'm not comfortable to add extra packages without @ImranR98 input, so for now I think it's out of scope for this PR.
If it gets merged, you can open another one with OkHttp if the performance stays the same

[flexxxxer]

@Marocco2

Your implementation does not provide certificate pinning nor it inherit implementation currently presented in Obtainium. I obviously cloned your fork, switched branch to fix/download-speed and checked that your commits exists just in case if I am dumb. I tested your fork and RuStore does not work, nor certificate pinning works

[Marocco2]

@Marocco2

Your implementation does not provide certificate pinning nor it inherit implementation currently presented in Obtainium. I obviously cloned your fork, switched branch to fix/download-speed and checked that your commits exists just in case if I am dumb. I tested your fork and RuStore does not work, nor certificate pinning works

Alr, I'll see what I can do shortly

[Marocco2]

@flexxxxer Quick update: currently this PR has not been merged with latest base, so it shouldn't have the github certificate pinning commit. I'll push the rebased branch on github

[Marocco2]

@flexxxxer Now it should work with RuStore

[flexxxxer]

@Marocco2

I tested latest changes. Problems:

• apps from RuStore wont obtain without Certificate pinning enabled. See how it is fixed currently in Obtainium here - basically trust Mintsifry CA + all other CAs from Android Root CA store for RuStore when certificate pinning disabled and trust Mintsifry CA + Harica CA for RuStore when certificate pinning enabled
• cancelling app obtaining process does not work. I tested in emulator, multiple times and it still reproducible

[Marocco2]

@Marocco2

I tested latest changes. Problems:

• apps from RuStore wont obtain without Certificate pinning enabled. See how it is fixed currently in Obtainium here - basically trust Mintsifry CA + all other CAs from Android Root CA store for RuStore when certificate pinning disabled and trust Mintsifry CA + Harica CA for RuStore when certificate pinning enabled
• cancelling app obtaining process does not work. I tested in emulator, multiple times and it still reproducible

• I'll check again for RuStore

• On my end it works:
  

[ImranR98]

Not sure im comfortable adding a whole new custom library for faster downloads tbh. Are there no existing solutions (like the ones suggested earlier)?

Also if we're going to use native code there should be a way to hook into Android's own cert store, in which case it would make sense to extend the lib to be used for all our http needs.

Appreciate the effort on the PR. Just thinking out loud as I'm not sure it's a good idea.

[Marocco2]

Not sure im comfortable adding a whole new custom library for faster downloads tbh. Are there no existing solutions (like the ones suggested earlier)?

Unfortunately it's unfeasible due to how Flutter plugins such as workmanager works. I've tried a bunch of solutions before posting the PR but they aren't as functional as this (I too I think this PR isn't so pretty) but it is what it is.
Ideally there should be another repo to link this native plugin like the other ones and add it on pubspec. I've tried searching for third party ones but no one was suitable for this kind of problem.

Also if we're going to use native code there should be a way to hook into Android's own cert store, in which case it would make sense to extend the lib to be used for all our http needs.

I dunno if it's possible and I think it won't matter too much on both performance and security

Appreciate the effort on the PR. Just thinking out loud as I'm not sure it's a good idea.

In the end, IMHO, developing this PR on Flutter was very painful for me (the tooling aren't so smooth as a Kotlin app on Android Studio) even more so if these app binaries wouldn't see the light of the day beside Android.
Still, compiling this plugin is painless while building the app with gradle integration, so there is that.
I think it's worth it if it means faster app installation management and battery life that are crucial in this kind of apps

[flexxxxer]

@Marocco2

Obtainment cancellation does not work as I stated above, see video recording I attached:

Obtainment cancellation does not work

Screencast_20260910_122615.mp4

I pasted link to MirPay app on RuStore: link

Also there are issues with certificate pinning implementation. I didnt expect that you would implement it anyway, so thank you! Issues related to this code:

• RuStore has no certificate pinning. See your code: you trusting Mintsifry CA and other CAs for RuStore regardless if certificate pinning enabled or disabled. See condition logic in configureTls
• certificate pinning isnt per-domain with fallback to root domain. Instead you need to use Map + lookup logic as Obtainium currently does: lookup on domain for trusted CAs, fallback on try to lookup on root domain for trusted CAs, fallback on no certificate pinning (I am talking in particular for _certificatePins and _createCertPinning in Obtainium code. Compare it to your condition logic in pinnedCertificatesFor

[flexxxxer]

@ImranR98

maybe better to report this issue about bad obtainment speeds to upstream? I didnt dig into that on flutter issue tracker, maybe it already reported...

The whole situation is mess. Maybe one day someone will rewrite Obtainium with Kotlin + Compose...

[Marocco2]

@flexxxxer Thank you for the repro! I noticed you have tested the debug version, meanwhile I've checked only normal version. Could you reproduce it also in that flavor? It may be related on how it handles split-parallel downloads to maximize connection throughput

Regarding RuStore, I'll rewrite the logic as you wish. It may come later in the day but I'm not sure

[Marocco2]

@ImranR98

maybe better to report this issue about bad obtainment speeds to upstream? I didnt dig into that on flutter issue tracker, maybe it already reported...

The whole situation is mess. Maybe one day someone will rewrite Obtainium with Kotlin + Compose...

I would really really really like to drop off this PR if DART implementation can be fixed upstream or it hasn't set correctly and I didn't notice it. Download progress was a culprit for low download performance also in DART but it can't explain why 50% of throughput speed is missing

[flexxxxer]

@Marocco2

I noticed you have tested the debug version, meanwhile I've checked only normal version. Could you reproduce it also in that flavor? It may be related on how it handles split-parallel downloads to maximize connection throughput

Yeah sure I will test release version later. Thanks for note

[Marocco2]

@flexxxxer So, I've changed how native downloader behaves according to your specification, please review it if I've understood it correctly. I've got rid of duplicate code inside .kt for maintenance purposes: now a tlsPolicyForUrl will be exchanged to the native plugin via apps_provider.dart that will set up the root certificates when certificate is pinning on/off and RuStore exception.

Also, I think I found the cancellation bug: a missing final on the request.cancel. Gonna test it later

[flexxxxer]

Sorry, I was busy. Will check in the few days

[flexxxxer]

@Marocco2 seems now certificate pinning implemented correctly

I suggest to replace:

    final policyHost = certificatePinning
        ? (_certificatePinAssetNames.containsKey(host)
              ? host
              : _certificatePinAssetNames.containsKey(rootHost)
              ? rootHost
              : null)
        : null;

with:

    String? policyHost;
    if (certificatePinning && _certificatePins.containsKey(host)) {
      policyHost = host;
    } else if (certificatePinning && _certificatePins.containsKey(rootHost)) {
      policyHost = rootHost;
    }

First one harder to read I think

[flexxxxer]

Obtainment cancellation does not work as I stated above, see video recording I attached

I cannot reproduce that anymore. Seems fixed in flutter or in your code

[Marocco2]

@Marocco2 seems now certificate pinning implemented correctly

I suggest to replace:

    final policyHost = certificatePinning
        ? (_certificatePinAssetNames.containsKey(host)
              ? host
              : _certificatePinAssetNames.containsKey(rootHost)
              ? rootHost
              : null)
        : null;

with:

    String? policyHost;
    if (certificatePinning && _certificatePins.containsKey(host)) {
      policyHost = host;
    } else if (certificatePinning && _certificatePins.containsKey(rootHost)) {
      policyHost = rootHost;
    }

First one harder to read I think

FYI, you can write requested changes on your PR review so I can press "Accept changes" easily.

I'll merge them anyway, so don't worry