[tests] fix JcwGen targetSdkVersion and remove dead .apk test infra

Documentation/building/unix/instructions.md

@@ -118,9 +118,8 @@ All NUnit-based tests can be executed via the `make run-nunit-tests` target:
 
 	$ make run-nunit-tests
 
-All `.apk`-based unit tests can be executed via the `make run-apk-tests` target:
-
-	$ make run-apk-tests
+All `.apk`-based (on-device) unit tests are run with `dotnet test`; see
+[Running On-Device Tests](#running-on-device-tests) below.
 
 
 ## Running Individual NUnit Tests
@@ -153,79 +152,47 @@ or via
 
     msbuild Xamarin.Android.sln /t:RunNunitTests /p:TEST=Xamarin.Android.Build.Tests.Aapt2Tests.Aapt2Compile
 
-## Running Individual `.apk` Projects
-
-You can run selected apk test by passing PACKAGES variable to
-`make run-apk-tests`. For example:
+## Running On-Device Tests
 
-    make run-apk-tests PACKAGES="Xamarin.Forms_Performance_Integration;Xamarin.Android.Locale_Tests"
+The on-device test apps are ordinary `dotnet test` projects that run their
+NUnit tests inside an Android instrumentation. Install the app, then run the
+tests against the attached device or emulator:
 
-or with msbuild:
+    $ ./dotnet-local.sh build -t:Install -c Release \
+        tests/Mono.Android-Tests/Mono.Android-Tests/Mono.Android.NET-Tests.csproj
+    $ (cd tests/Mono.Android-Tests/Mono.Android-Tests && \
+        ../../../dotnet-local.sh test Mono.Android.NET-Tests.csproj --no-build -c Release \
+            --report-trx --results-directory ../../../bin/TestRelease/TestResults)
 
-    msbuild /t:RunApkTests tests/RunApkTests.targets /p:ApkTests='"Xamarin.Forms_Performance_Integration;Xamarin.Android.Locale_Tests"'
+The same pattern works for
+`tests/CodeGen-Binding/Xamarin.Android.JcwGen-Tests/Xamarin.Android.JcwGen-Tests.csproj`.
 
-Another possibility is to run them manually as described below.
+Results are written as `.trx` files into the `--results-directory` directory.
 
-See also the [`tests/RunApkTests.targets`](../../../tests/RunApkTests.targets) and
-[`build-tools/scripts/TestApks.targets`](../../../build-tools/scripts/TestApks.targets)
-files.
+### Running Specific On-Device Tests
 
-All `.apk`-based unit test projects provide the following targets:
+`dotnet test` filtering options apply, for example:
 
-  * `DeployTestApks`: Installs the associated `.apk` to an Android device.
-  * `UndeployTestApks`: Uninstalls the associated `.apk` from an Android device.
-  * `RunTestApks`: Executes the unit tests contained within a `.apk`.
-    This target must be executed *after* the `DeployTestApks` target.
+    $ ../../../dotnet-local.sh test Mono.Android.NET-Tests.csproj --no-build \
+        --filter-class "Xamarin.Android.RuntimeTests.JnienvTest"
 
-To run an individual `.apk`-based test project, a package must be built, using the
-`SignAndroidPackage` target, installed, and executed.
+### Running On-Device Tests with Include/Exclude
 
-## Running `.apk` Projects with Include/Exclude
-
-If an `.apk`-based unit test uses the NUnit `[Category]` custom attribute, then
-those tests can be explicitly included or excluded from execution by setting
-the `$(INCLUDECATEGORIES)` or `$(EXCLUDECATEGORIES)` make variables.
+If an on-device test uses the NUnit `[Category]` custom attribute, then those
+tests can be explicitly included or excluded from execution by setting the
+`$(IncludeCategories)` or `$(ExcludeCategories)` MSBuild properties when
+building the test app. These flow to the on-device instrumentation through
+`runtimeconfig.json`.
 
 For example, to exclude tests that use the internet (`InetAccess`) category:
 
-	$ make run-apk-tests EXCLUDECATEGORIES=InetAccess
-
-`$(INCLUDECATEGORIES)` functions in the same fashion.
+    $ ./dotnet-local.sh build -t:Install -c Release \
+        -p:ExcludeCategories=InetAccess \
+        tests/Mono.Android-Tests/Mono.Android-Tests/Mono.Android.NET-Tests.csproj
 
 To specify multiple categories, separate each category with a `:` character.
 
 
-### Running A Single Test Fixture
-
-A single NUnit *Test Fixture* -- a class with the `[TestFixture]`
-custom attribute -- may be executed instead of executing *all* test fixtures.
-
-The `RunTestApks` target accepts a `TestFixture` MSBuild property
-to specify the test fixture class to execute.
-
-If using `Xamarin.Android.NUnitLite` for projects outside the `dotnet/android`
-repository, such as NUnit tests for a custom app, the `RunTestApks` target
-will not exist. In such scenarios, the [`adb shell am`][adb-shell-am]
-`instrument` command can be used instead. It follows the format:
-
-[adb-shell-am]: https://developer.android.com/studio/command-line/adb.html#am
-
-	$ adb shell am instrument -e suite FIXTURE -w PACKAGE/INSTRUMENTATION
-
-Where:
-
-  * `FIXTURE` is the full *managed* class name of the NUnit test fixture to
-    execute.
-  * `PACKAGE` is the Android package name containing the NUnit tests
-  * `INSTRUMENTATION` is the *Java callable wrapper* class name to execute,
-    located within the Android package `PACKAGE`.
-
-For example:
-
-	$ adb shell am instrument -e suite Xamarin.Android.LocaleTests.SatelliteAssemblyTests \
-		-w "Xamarin.Android.Locale_Tests/xamarin.android.localetests.TestInstrumentation"
-
-
 # How do I build `Mono.Android.dll` for a given API Level?
 
 There are a few ways to do it:

Documentation/building/windows/instructions.md

@@ -124,9 +124,9 @@ and path for the proper version of `dotnet`.
 
 ## Running Unit Tests
 
-All `.apk`-based unit tests can be executed via
+All NUnit-based unit tests can be executed via
 
-    dotnet-local.cmd build Xamarin.Android.sln /t:RunApkTests
+    dotnet-local.cmd build Xamarin.Android.sln /t:RunNunitTests
 
 ### Listing Nunit Tests
 
@@ -142,34 +142,38 @@ You can run then a single (or a group) of tests using the `$(TEST)` MSBuild prop
 
     dotnet-local.cmd build Xamarin.Android.sln /t:RunNunitTests /p:TEST=Xamarin.Android.Build.Tests.Aapt2Tests.Aapt2Compile
 
-### Running Individual `.apk` Projects
+## Running On-Device Tests
 
-See also the [`tests/RunApkTests.targets`](../../tests/RunApkTests.targets) and
-[`build-tools/scripts/TestApks.targets`](../../build-tools/scripts/TestApks.targets)
-files.
+The on-device test apps are ordinary `dotnet test` projects that run their
+NUnit tests inside an Android instrumentation. Install the app, then run the
+tests against the attached device or emulator:
 
-All `.apk`-based unit test projects provide the following targets:
+    dotnet-local.cmd build -t:Install -c Release tests\Mono.Android-Tests\Mono.Android-Tests\Mono.Android.NET-Tests.csproj
+    pushd tests\Mono.Android-Tests\Mono.Android-Tests
+    ..\..\..\dotnet-local.cmd test Mono.Android.NET-Tests.csproj --no-build -c Release --report-trx --results-directory ..\..\..\bin\TestRelease\TestResults
+    popd
 
-* `DeployTestApks`: Installs the associated `.apk` to an Android device.
+The same pattern works for
+`tests\CodeGen-Binding\Xamarin.Android.JcwGen-Tests\Xamarin.Android.JcwGen-Tests.csproj`.
 
-* `UndeployTestApks`: Uninstalls the associated `.apk` from an Android device.
+Results are written as `.trx` files into the `--results-directory` directory.
 
-* `RunTestApks`: Executes the unit tests contained within a `.apk`.
-  This target must be executed *after* the `DeployTestApks` target.
+### Running Specific On-Device Tests
 
-To run an individual `.apk`-based test project, a package must be built, using the
-`SignAndroidPackage` target, installed, and executed.
+`dotnet test` filtering options apply, for example:
 
-### Running `.apk` Projects with Include/Exclude
+    ..\..\..\dotnet-local.cmd test Mono.Android.NET-Tests.csproj --no-build --filter-class "Xamarin.Android.RuntimeTests.JnienvTest"
 
-If an `.apk`-based unit test uses the NUnit `[Category]` custom attribute, then
-those tests can be explicitly included or excluded from execution by setting
-the `$(IncludeCategories)` or `$(ExcludeCategories)` MSBuild properties.
+### Running On-Device Tests with Include/Exclude
 
-For example, to exclude tests that use the internet (`InetAccess`) category:
+If an on-device test uses the NUnit `[Category]` custom attribute, then those
+tests can be explicitly included or excluded from execution by setting the
+`$(IncludeCategories)` or `$(ExcludeCategories)` MSBuild properties when
+building the test app. These flow to the on-device instrumentation through
+`runtimeconfig.json`.
 
-    dotnet-local.cmd build Xamarin.Android.sln /t:RunApkTests /p:ExcludeCategories=InetAccess
+For example, to exclude tests that use the internet (`InetAccess`) category:
 
-`$(IncludeCategories)` functions in the same fashion.
+    dotnet-local.cmd build -t:Install -c Release /p:ExcludeCategories=InetAccess tests\Mono.Android-Tests\Mono.Android-Tests\Mono.Android.NET-Tests.csproj
 
 To specify multiple categories, separate each category with a `:` character.

Documentation/guides/profiling.md

@@ -71,43 +71,6 @@ After launching the app, we get additional information:
 These are timings taken at different points during .NET for Android
 startup: across native and managed code.
 
-In case you have the XA build tree by hand, it is possible to process
-that information like:
-
-    > msbuild /v:d build-tools/timing/timing.csproj /t:LogcatTiming /p:PID=3011  # process id of your application
-    ...
-    Task "ProcessLogcatTiming"
-      Time:      0ms process start, application: '' PID: 3011
-      Time:     57ms Message: Runtime.init: Mono runtime init; elapsed: 0s:0::44834
-      Time:     65ms Message: Finished loading assemblies: preloaded 1 assemblies; wasted time: 0s:0::49
-      Time:     73ms Message: JNIEnv.Initialize start
-      Time:     78ms Message: JNIEnv.Initialize: Logger JIT/etc. time: elapsed 3 ms]
-      Time:    101ms Message: JNIEnv.Initialize: managed runtime init time: elapsed 22 ms]
-      Time:    113ms Message: TypeManager.cctor start: 1567669218343.18
-      Time:    116ms Message: TypeManager.cctor time: 1567669218357.74 [elapsed: 14.563 ms]
-      Time:    117ms Message: JNIEnv.Initialize: TypeManager init time: elapsed 15 ms]
-      Time:    156ms Message: JNIEnv.RegisterJniNatives ("Android.Runtime.UncaughtExceptionHandler, Mono.Android", 0xffe3c454) start: 1567669218396.28
-      Time:    163ms Message: JNIEnv.RegisterJniNatives total time: 1567669218404.38 [elapsed: 8.097 ms]
-      Time:    163ms Message: Runtime.register: end time; elapsed: 0s:8::800097
-      Time:    164ms Message: JNIEnv.RegisterJniNatives ("Java.Interop.TypeManager+JavaTypeManager, Mono.Android, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null", 0xffe3bcf4) start: 1567669218405.08
-      Time:    164ms Message: JNIEnv.RegisterJniNatives total time: 1567669218405.86 [elapsed: 0.787 ms]
-      Time:    164ms Message: Runtime.register: end time; elapsed: 0s:0::813062
-      Time:    166ms Message: JNIEnv.Initialize end: elapsed 105 ms
-      Time:    166ms Message: JNIEnv.Initialize end: elapsed 105 ms
-      Time:    166ms Message: Runtime.init: end native-to-managed transition; elapsed: 0s:100::17520
-      Time:    166ms Message: Runtime.init: end native-to-managed transition; elapsed: 0s:100::17520
-      Time:    166ms Message: Runtime.init: end, total time; elapsed: 0s:109::900001
-      Time:    173ms Message: JNIEnv.RegisterJniNatives ("xatemplateaot.MainActivity, xatemplateaot", 0xffe3cbe4) start: 1567669218414.33
-      Time:    173ms Message: JNIEnv.RegisterJniNatives total time: 1567669218414.89 [elapsed: 0.552 ms]
-      Time:    173ms Message: Runtime.register: end time; elapsed: 0s:0::594434
-      Time:    232ms Message: JNIEnv.RegisterJniNatives ("Android.Views.View+IOnClickListenerImplementor, Mono.Android", 0xffe3bb54) start: 1567669218473.37
-      Time:    232ms Message: JNIEnv.RegisterJniNatives total time: 1567669218473.59 [elapsed: 0.219 ms]
-      Time:    232ms Message: Runtime.register: end time; elapsed: 0s:0::241370
-       -- Performance summary --
-      Last timing message: 232ms
-    Done executing task "ProcessLogcatTiming".
-    ...
-
 An example of these log messages would be [this managed
 call][managed_timing] or [this native call][native_timing].
 

Documentation/project-docs/ApkSizeRegressionChecks.md

@@ -2,9 +2,8 @@
 
 We are checking the apk sizes for regression during CI builds.
 
-The apk size information is collected in 2 places, in APK
-instrumentation tests and MSBuild tests. It is then compared to
-reference `.apkdesc` files with `apkdiff` tool,
+The apk size information is collected by MSBuild tests. It is then
+compared to reference `.apkdesc` files with `apkdiff` tool,
 https://www.nuget.org/packages/apkdiff/. It compares
 the size differences against reference sizes and fails when
 they are larger than given thresholds. The test result file contains
@@ -23,20 +22,22 @@ and some files are built with different optimizations.
 
 The `BuildReleaseArm64` test is used to collect apk size data.
 
-The test builds simple Xamarin Android and simple Xamarin Forms
-on Xamarin Android apps. We build it targeting legacy and NET6
-framworks, so this get us 4 variations to check.
+The test builds a simple .NET for Android app and a simple .NET MAUI
+app, for each supported runtime, so this gets us several variations
+to check.
 
 The reference files are located
 in `src\Xamarin.Android.Build.Tasks\Tests\Xamarin.ProjectTools\Resources\Base`
 directory. During the test run, we save `.apkdesc` files, with
-current sizes. These files can be used a new reference. The 4 files
+current sizes. These files can be used a new reference. They
 are named like this:
 
-    .../Base/BuildReleaseArm64SimpleDotNet.apkdesc
-    .../Base/BuildReleaseArm64SimpleLegacy.apkdesc
-    .../Base/BuildReleaseArm64XFormsDotNet.apkdesc
-    .../Base/BuildReleaseArm64XFormsLegacy.apkdesc
+    .../Base/BuildReleaseArm64SimpleDotNet.CoreCLR.apkdesc
+    .../Base/BuildReleaseArm64SimpleDotNet.MonoVM.apkdesc
+    .../Base/BuildReleaseArm64SimpleDotNet.NativeAOT.apkdesc
+    .../Base/BuildReleaseArm64XFormsDotNet.CoreCLR.apkdesc
+    .../Base/BuildReleaseArm64XFormsDotNet.MonoVM.apkdesc
+    .../Base/BuildReleaseArm64XFormsDotNet.NativeAOT.apkdesc
 
 The new reference files can be obtained from the test results
 archive - artifact of the given CI build (preferred method).
@@ -46,26 +47,8 @@ or the `build-tools/scripts/UpdateApkSizeReference.sh` script
 if you are on MacOS or *nix.
 
 The thresholds for these checks are set
-in `src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.cs`
-in `BuildReleaseArm64` method.
-
-# APK instrumentation tests
-
-2 instrumentation tests are used to collect apk size data,
-`tests\Xamarin.Forms-Performance-Integration` and
-`samples\VSAndroidApp` test apps.
-
-The reference file are located in `tests/apk-sizes-reference` directory.
-
-    com.companyname.vsandroidapp-Signed-Release.apkdesc
-    Xamarin.Forms_Performance_Integration-Signed-Release.apkdesc
-    Xamarin.Forms_Performance_Integration-Signed-Release-Aot.apkdesc
-    Xamarin.Forms_Performance_Integration-Signed-Release-Bundle.apkdesc
-    Xamarin.Forms_Performance_Integration-Signed-Release-Profiled-Aot.apkdesc
-
-The thresholds for these checks are set
-in `build-tools/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks/ApkDiffCheckRegression.cs`
-in fields of `ApkDiffCheckRegression` class.
+in `src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest2.cs`
+in the `BuildReleaseArm64` method.
 
 # How to resolve regression
 

Makefile

@@ -70,10 +70,6 @@ endif # $(SKIP_NUNIT_TESTS) == ''
 run-ji-tests:
 	$(call MSBUILD_BINLOG,run-ji-tests,,Test) $(TEST_TARGETS) /t:RunJavaInteropTests
 
-ifneq ($(PACKAGES),)
-APK_TESTS_PROP = /p:ApkTests='"$(PACKAGES)"'
-endif
-
 list-nunit-tests:
 	$(MSBUILD) $(MSBUILD_FLAGS) $(TEST_TARGETS) /t:ListNUnitTests
 
@@ -87,11 +83,5 @@ prepare: install-dotnet
 install-dotnet:
 	CONFIGURATION=$(CONFIGURATION) bash ./eng/install-dotnet.sh
 
-APK_SIZES_REFERENCE_DIR=tests/apk-sizes-reference
-
-update-apk-sizes-reference:
-	-mkdir -p $(APK_SIZES_REFERENCE_DIR)
-	cp -v *values-$(CONFIGURATION).csv $(APK_SIZES_REFERENCE_DIR)/
-
 update-api-docs:
 		$(call DOTNET_BINLOG,update-api-docs) -t:UpdateExternalDocumentation src/Mono.Android/Mono.Android.csproj

Xamarin.Android-Tests.slnx

@@ -42,5 +42,4 @@
   <Folder Name="/tests/TestRunner.Core/">
     <Project Path="tests/TestRunner.Core/TestRunner.Core.NET.csproj" />
   </Folder>
-  <Project Path="tests/Xamarin.Forms-Performance-Integration/Xamarin.Forms.Performance.Integration.csproj" />
 </Solution>