NeXus `search_quantities` in NOMAD

[lukaspie]

Recently, we have started playing around with adding more and more search quantities in NOMAD (mostly in #525). In particular, this includes adding all NeXus attributes as NOMAD quantities (using the __<attribute_name> convention) and field aggregation statistics (using __mean, __var, etc.), with the idea of powering the search in the apps that we are developing.

However, we soon started running into some limitations. Most notably, the GUI becomes incredibly slow for groups that have lots of fields. A noteable example is the XPS example, where there are a lot of cycle and scan repititions in the default NXdata group (see image). Loading the data/data/ENTRY:0/data tab here takes up to 1 min due to a lag in the GUI. Similar situations are expected in descriptions of microstructures, with potentially millions of instances of the same concept.

So, we are faced with an issue: which search_quantities do we want to expose from the NeXus definitions in NOMAD. This issue is meant to summarize the current situation and suggested ideas and also to store any results coming from upcoming discussions on this, e.g. in the TF meetings.

• One suggestion (made by @sanbrock and @rettigl) is that we only make those concepts that are specifically mentioned in the application definition available for search. That includes all required or recommended elements, but also those that are optional. But explicitly not those that are just defined in the base class that are inherited in the application definition.
  Comments so far on this:

  1. This could be a good solution, at least for now. We must consider that the appdefs will likely need to get more comprehensive to be usable in NOMAD and will probably blow up a bit. For MPES, Laurenz and I already started to mention more terms explicitly in the application definition (including all the AXISNAME and DATA fields in NXdata), see MPES: new concepts from NIAC discussions, searchable fields nexus_definitions#329.
  2. It is a bit strange to make the contents of the appdef depend on how we want to use them in NOMAD. Of course, adding more optional elements (as done above) is not a problem, but it feels slightly backwards to build the appdef for an experimental technique based on the search and visualization capabilities of NOMAD.
  3. If a new use case for an appdef comes up, where more concepts from the base classes are to be used, you would need to blow up the appdef even more or make an extension for it (like NXxps extending NXmpes), to allow for more search_quantities that are relevant for this new use case. The second option raises a problem/question which I have been asking myself for a while: If an appdef extends another one (think NXxps extending NXmpes), is it possible to add more elements or can they only specialize what is already in the sup-appdef. If I have an NXfit class in NXxps, but not in NXmpes, is a file that implements NXxps with a fit even compliant with NXmpes anymore?

• There is an alternative proposed by @mkuehbach: we ship a "concept filtration configuration" with pynxtools that explicitly states for each application definition what the searchable quantities should be. This would be a yaml/json file that defines a selected set of elements you can search for. This may include all or some of the appdef concepts, but could also include some of those from the base classes.
  Comments so far on this:

  1. This is of course another abstraction layer that the user is not aware of and that makes it difficult to understand which elements are searchable for which appdef.
  2. Whenever the application definition or base classes change, we also need to update these configurations.
  3. Such configurations should probably be shipped with the main pynxtools, not with the reader plugins. This avoid conflicts that could arise for two plugins writing to the same appdef (pynxtools-mpes/xps both write to NXmpes) or a plugin that can touch many appdefs (like pynxtools-igor).
  4. We use a similar option already for the configuration of the multiformat reader. These are JSON files that map concepts defined in the vendor specific files to the concepts in the appdef. For this, we have a CLI function that generates a template for a given application definition. We could likely reuse the python code of that script to give the default search_quantities config, i.e. all concepts defined in the appdef. This would then need to be customized.
  5. Another option this approach could open up: we could use the same filter for exporting from NOMAD. That is, a pynxtools-adjacent tool takes an existent NeXus file/data archive in NOMAD and a filter map and exports a smaller NeXus file. Here we are adding to the macro issue in NOMAD that is about exporting from NOMAD, but more than just download that file .

We could also go for a combination of the two approaches: for specific appdefs (i.e., the ones designed by FAIRmat) you have a "concept filtration configuration", whereas for the other application definition you can only search for what is in the appplication definition itself (this would be the default).

Further comments:

• Regarding the aggretation statistics, in order to not increase the number of quantities, we could bring NeXus fields to NOMAD as subsections. This was indeed implemented in the first version of the NexusParser as there was no support for attributes). So, we could make a subsection for each number quantity like 'energy__field' . This subsection would contain all its stats. And all its attributes we could also bring here as Quantities.

Sorry for the wall of text, but I just wanted to summarize the current situation accurately. Looking for input here and in the upcoming TF meetings. @FAIRmat-NFDI/areab

[rettigl]

2. It is a bit strange to make the contents of the appdef depend on how we want to use them in NOMAD. Of course, adding more optional elements (as done above) is not a problem, but it feels slightly backwards to build the appdef for an experimental technique based on the search and visualization capabilities of NOMAD.

I don't think that this depends on NOMAD. If we consider "searching for an element" as an application, then having all searchable fields in the corresponding application definition is natural.

[rettigl]

3. If a new use case for an appdef comes up, where more concepts from the base classes are to be used, you would need to blow up the appdef even more or make an extension for it (like NXxps extending NXmpes), to allow for more search_quantities that are relevant for this new use case. The second option raises a problem/question which I have been asking myself for a while: If an appdef extends another one (think NXxps extending NXmpes), is it possible to add more elements or can they only specialize what is already in the sup-appdef. If I have an NXfit class in NXxps, but not in NXmpes, is a file that implements NXxps with a fit even compliant with NXmpes anymore?

That is a good point. I think the current mechanism in Nexus does not prevent adding arbitrary entries, so there is no relationship of the kind NXspx is_a NXmpes.

[rettigl]

1. This is of course another abstraction layer that the user is not aware of and that makes it difficult to understand which elements are searchable for which appdef.

Alone for that reason, I have some issues with this idea.

[rettigl]

• Regarding the aggretation statistics, in order to not increase the number of quantities, we could bring NeXus fields to NOMAD as subsections. This was indeed implemented in the first version of the NexusParser as there was no support for attributes). So, we could make a subsection for each number quantity like 'energy__field' . This subsection would contain all its stats. And all its attributes we could also bring here as Quantities.

I am not sure I understand what this would mean conceptually.

[lukaspie]

Problems:

1. in XPS, we have hundreds of scans that are all instantiated as NXdata/DATA, creating too many quantities in one subsection (crashing/slowing down the GUI)
2. in APM, we have two problems hitting the limit of elastic search (~10k search quantities, arbitrary limit in NOMAD that will not be increased due to space restrictions)
   a. too many concepts with instantiable data
   b. too many instances for some of the defined concepts (think 2000 different samples)
3. in the MPES app, we cannot search for specific concepts with variadic names (example was beamTYPE in the application definition, which can be used for beam_probe and beam_pump and you only want to search for the energy of beam_probe)

Discussion/Decisions:

• For 1.: try to clean up the GUI problems.
  • Solution for now is to show all fields by default, but rather you have a "more" section in the GUI to show more and more fields -> we agreed on this
  • Possibility: make a registry that creates grouping of these fields for easier navigation in the GUI (possibly alphabetical)
• For 2.:
  • You can increase the number of elastic search quantities in an OASIS for a specific use case
  • Parsing should still continue even if there are more instances than allowed search quantities. Currently the whole processing chain stops if there are too many instances than allowed search quantities. Instead, a warning should we thrown, but the data should be stored anyway in JSON.
• For 3.
  • Clarify the application definition(s) to clear up the ontological mismatch. In the example above, we implemented beam_probe and beam_pump as separate concepts in the application definition.
  • Combined/conditional search: today not supported, but could possibly be in the future
    • Example: give me the mean of a concept (like beamTYPE) if the instance name has a certain value (beam_probe)
    • Example: give me all the user_name where the user_affiliation is "FHI"
      • Question: do you want to type "FHI" in the search bar or do you want to do this in a specific widget
    • We would like to have these combinations in one new widget -> this will probably be pretty involved
• Search bar: should there be an autocompletion for searching, i.e. to simplify the syntax?
  • will not be implemented in the old GUI, but will be in the new UI
• We keep NeXus fields and attributes as NOMAD quantities. This hinges on the GUI's ability to handle this.

Action items for now:

• Clarify the application definition(s) to clear up the ontological mismatch (i.e., where a variadic name actually describes two or more separate concepts)
• Visualization of many concepts in one lane using a "more" section
• Change: make the processing chain continue even if there are too many instances than allowed search quantities, throw error instead
• Lauri is working on the scatter plot, check that it is working (i.e., a scatter plot shows up)
• Lauri is working on the search bar, check that it is working (i.e., valid queries are actually showing correctly)
• Support OR queries for the terms widget

Action items (postponed):

• Make a registry that creates grouping of these fields for easier navigation in the GUI (possibly alphabetical)
• Documentation: what should you do if there are more instances than allowed search quantities?
• For quantities with variadic names, make its instance name searchable (already achieved for sections)
• Terms widget: add a scroll bar if possible
• Create a new search widget that implements conditional search as outlined above (low priority)

For future discussion, if the action items don't solve our problems right now, we may have to think about ways of limiting the allowed search_quantities.

Suggestions for limiting the search_quantities:

• The only allowed search_quantities are those fields and attributes that
  • are explicitly mentioned in the application definitions under fixed and variadic names
    • Use case: single lab has a specific axis in NXdata that they always use. They don't want to make a new application definition, but still search for this axis. If we have a variadic NXdata/DATA in the application definition, this search is allowed as well..
    • Use case: if we have beamTYPE in the application definition, it is also allowed to search for beam_laser.
  • are mentioned in those base classes that are specialized in the application definition and that are using nameType=specified, i.e. not those that are variadic. An example what we don't want to make searchable are instances of NXdata/DATA for which the name is not explicitly given in the appdef.
    • Use case: this solves 1. above, limiting the XPS, we have hundreds of scans that are all instantiated as NXdata/DATA, creating too many
  • What about nested definitions, think NXmpes/INSTRUMENT, can all fields in base_classes mentioned in NXinstrument be searched?
• Consequently, we only parse those fields that fullfil these requirements into the Data section.
• For the aggregation statistics, we keep creating them as before, but now we will just have less fields, so the GUI issues will (hopefully) go away.

[lukaspie]

As this comes up again and again, together with AI, I went on a deep dive into the actual technical reasons that determine which NeXus quantities are and, more importantly, are not (e.g. APM's atom_probeID[*].ranging.peak_identification.ionID[*].name) searchable today, and also into solutions that could change to allow more of the schema to be searched.

For reference, code links point at nomad-FAIR@develop (79c9de03) and pynxtools@nexus-metainfo-phase4 (0f0bc824), line numbers will change as those branches are changed.

The problem

NOMAD Search Apps (MenuItemHistogram, Column, etc.) reference pynxtools quantities via search_quantity strings like data.<path>#<schema> (see e.g. app_v2.py's schema variable and its MenuItemHistogram usages). Currently, paths more than 3 subsection-levels deep from a top-level class (Entry, Apm, Xps, ...) are not possible (the app fails to render); not because the path is wrong, but because they are never registered as searchable in the first place. This is an intentional limit on the search quantities pynxtools generates, to increase the performance of the appworker. Raising the depth limit to 4 or 5 makes the NOMAD appworker take an unreasonable amount of time to start (and also apps are much slower).

Current limitations and costs

There are two independent mechanisms in nomad-FAIR that determine the question "is this quantity searchable":

1. Registration cost (schema/concept level). Once per worker process at startup, reload_quantities_dynamic() walks every plugin schema's class tree (via its nested get_all_quantities() function) to build the list of searchable quantities. For most plugins this walk is unrestricted. For pynxtools specifically it's capped: only a fixed list of top-level classes (selected_path, like Apm, Em, Xps, etc.) are walked at all, and only ever 3 subsections deep (max_level = 3). This cap exists because pynxtools's schema is far larger and more interconnected than a typical plugin's — walking it unrestricted is what makes the appworker hang on startup. This cost depends only on the shape and size of the schema itself, never on how much data has actually been parsed.

2. Indexing cost (data/instance level). Separately, at parse time, each entry's actual populated values get written into Elasticsearch as nested documents, one per populated dynamic quantity (via create_searchable_quantity(), into the SearchableQuantity structure). Elasticsearch caps nested documents per entry at 10,000 by default — this is the "~10k search quantities" limit mentioned above, and it depends on how many actual instances (e.g. samples, scans) one specific entry contains.

Everything below addresses cost #1 — the registration/startup problem that's actually blocking deep paths from being searchable at all. Cost #2 (too many instances in one entry) is a separate, still-open problem, tracked above as "make parsing continue with a warning instead of stopping."

Why some obvious ideas don't help

• Flattening the application definitions (e.g. NXapm/NXem) to be shallower doesn't reduce the total cost of schema registration. max_level counts subsection hops regardless of what's named at each level — flattening just moves the same concepts to shallower levels, so eventually you'd have to go back to max_level=2 for the same number of total concepts.
• Pre-registering only the paths referenced by App configs and/or application definitions" (outlined above): IMO we should not do this as it's very hard to know ahead of time what people will actually want to search for, and it is another layer that people need to understand.

Options that could help

A. Memorize the shared-subtree walk (low risk, incremental)

get_all_quantities() runs independently for each of the ~11 top-level classes (Entry, Apm, Xps, ...), re-walking shared subsections (Sample, Instrument, User, ...) from scratch every time, even though the underlying class objects are identical across all of them. Caching for each subsection what it yields at a given remaining depth and reusing it across the multiple independent walks should cut a large amount of redundant work without changing any output.

The walk only ever runs once per worker process at boot — called from nomad/app/main.py and (guarded) from nomad/datamodel/__init__.py. So a cache scoped to that one call introduces no staleness/invalidation concerns — it's a pure performance change, not a behavior change. Cheapest option to try first; worth measuring before deciding how far it gets us. This change would be solely on nomad-lab.

B. Precompute the search-quantity list at schema-generation time (more complicated)

reload_quantities_dynamic()'s output is (mostly) a deterministic function of the compiled schema — pynxtools already generates its schema classes ahead of time and checks the generated .py files into the repo (built via build_base_classes_package()/build_applications_package()). The same idea could extend to search quantities: generate a companion manifest listing every valid (path, class, dtype) triple once, at generation time, and have NOMAD load and filter that file at startup instead of re-walking the metainfo tree live.

However, which quantities are enabled still depends on which entry points a given Oasis turns on (nomad.yaml), not on the schema alone. A possible fix is to precompute the full set for all definitions and filter it down cheaply at startup, rather than trying to precompute per-deployment.

This would require a change on pynxtools to prepopulate this list, as well as a change on the nomad-lab package to be able to understand this manifest at the top.

Don't confuse this manifest with entry_type.mapping, the actual Elasticsearch index mapping, built by _create_mapping_recursive(). That mapping skips dynamic quantities entirely. It only ever reflects statically annotated fields (Option D), so it's already cheap to build and has nothing to do with max_level. What we'd precompute here is the separate self.quantities registry reload_quantities_dynamic() builds, which is only used for query validation and never sent to Elasticsearch as mapping at all. Both get built inside the same create_mapping() call, which is exactly why it's easy to mix them up.

C. Generalize lazy, on-demand resolution (most fundamental fix)

Instead of eagerly registering everything up to some depth at startup, resolve a quantity's type on the fly, the first time it's actually queried, by walking the compiled schema along the exact requested path. This would extend validate_quantity(), which today only knows two outcomes: found in the static registry, or fall back to parse_quantity_name()'s dynamic path — which needs an explicit #<type> hint, because there's no compiled schema available to consult at that point. NOMAD's metainfo framework already has the building blocks for step-by-step path resolution elsewhere in the codebase (used for other lookups); this would mainly need wiring into that fallback.

This turns registration cost from "proportional to total schema size" into "proportional to what's actually queried"; the natural fix for the mismatch causing this problem in the first place.

Bonus: it would likely also help the old, XML-compiled NeXus schema, which needs the explicit #<type> hint, see the
nexus_prefixspecial-casing in validate_quantity()). Lazy resolution doesn't need that eager walk, so the same fix could remove that requirement too.

That would definitely the most invasive in nomad-FAIR, but also the most impactful change. I am not sure that even works given that the whole approach is to register everything on startup.

D. Hand-pick specific deep quantities as real, static search fields

NOMAD has a separate, older mechanism, via create_mapping() and its recursive _create_mapping_recursive(): explicitly marking a quantity (and every subsection on the path to it) with a search annotation (a_elasticsearch, see NOMAD docs) makes it a real, permanently mapped Elasticsearch field — completely bypassing both the startup-walk limit and the per-entry 10k-nested-document cap.

You could annotate just the specific chain you need (atom_probeID → ranging → peak_identification → ionID → name), and nothing else gets pulled in (this would be helpful for @mkuehbach). Or you can set auto_include_subsections=True once, higher up, which descends into everything below that point without needing per-level annotations. This is how material_type is built, turned on once at the root. That's less manual work for a whole subtree, but probably not feasible for us: the deeply-nested subtree for e.g. Apm would already be relatively too big on its own right now.

After testing this out with the APM app (with data.atom_probeID[*].ranging.peak_identification.ionID[*].name, where each of these subsections/quantities was annotated with a_elasticsearch), this does not actually work. The the static a_elasticsearch walk descends from the EntryArchive class into each subsection by its declared type (results → Results, data → EntryData), but only for these specific classes, never the concrete class an entry actually gets at runtime. So even with a_elasticsearch annotations in Apm, these are still not registered.

E. Per-technique entry points (reduces scope, not the underlying limit)

Splitting the single nexus_base_classes/nexus_applications entry points into one per technique (an APM-only entry point, an XPS-only one, etc.) would mean an Oasis that only cares about one technique only ever registers that technique's classes. That shrinks the total walk for that deployment, making a higher max_level affordable there specifically. Potentially we could make max_level a configurable value for such Oasis. It doesn't help a deployment that needs everything (like NOMAD Central), and it doesn't touch the separate per-entry 10k cap at all — but it's planned anyway for other reasons, so we will do this at some point.

Suggested pathways

These aren't mutually exclusive. A reasonable sequence:

1. Right now, before any nomad-FAIR change: hand-pick D (static annotations) for the specific paths people actually need (like the APM ranging/peak-identification case in the APM app). This needs no core changes at all — it's just adding annotations in pynxtools's own generated schema — so it can go in immediately and give relief for known pain points while everything else below is still being worked out. this was tested, does not work
2. Prototype A (memorization) next. Cheapest nomad-FAIR change, safest, no design change, directly measurable. If it alone gets max_level to somewhere around 4 to 6 at acceptable startup and query times, that may be enough on its own.
3. If A isn't enough, pursue C (lazy resolution) as the structural fix — larger effort, touches core nomad-FAIR search-validation code, but removes the depth ceiling as a concept entirely rather than pushing it back a little, and likely also cleans up the old-schema #<type> requirement as a side effect.
4. E (per-technique entry points) as a separate, complementary effort — worth doing on its own for Oases that only need part of the schema, regardless of what happens with A/C.

Cost #2 (per-entry 10k nested-document cap) is a separate problem, already tracked as an open action item above, and none of the above fixes it — worth keeping as its own task rather than folding it in here.

Where this touches which repo

• nomad-FAIR: nomad/metainfo/elasticsearch_extension.py (reload_quantities_dynamic, get_all_quantities, create_mapping, _create_mapping_recursive) and nomad/search.py (validate_quantity) — A, C, D, and the per-entry cap all live here.
• pynxtools: the NXDL-to-metainfo generator and entry-point registration (nomad/metainfo/_package.py, nomad/apps/app_v2.py) — B and E live here.