feat/labs migration

[alanpeixinho]

Summary

Migrates all read paths for lab information from JSONB misc fields (builds.misc->>'lab',
tests.misc->>'runtime') to the new labs table via lab_id foreign key, with JSONB fallback for rows not yet backfilled.
Every SQL query and Python helper that previously extracted lab names by parsing JSONB now uses COALESCE(labs.name, misc_fallback) through a LEFT JOIN on the labs table. This ensures:

• New data (with lab_id populated by the ingester) reads from the structured FK
• Historical data (without lab_id) still works via the JSONB fallback
• After a full backfill, the JSONB fallbacks can be removed (all marked with TODO comments)

How to test

legacy database

• Use a database where lab_id is NULL on existing rows
• Verify all pages display lab names as before:
  • Tree details page (boots/tests tabs) — lab column in test history
  • Tree commits history page — build lab and test lab in filters and rows
  • Hardware details page — lab in test/build summaries, history, and filters
  • Build details page — lab column in test list
  • Issue details page — lab column in test list
  • Notifications/metrics endpoint — lab summary counts

Migrated database (test in a local database)

• Run the migrations available
• Run the ingester (which already fills lab_id)
  • Verify all pages display lab names as before:
  • Tree details page (boots/tests tabs) — lab column in test history
  • Tree commits history page — build lab and test lab in filters and rows
  • Hardware details page — lab in test/build summaries, history, and filters
  • Build details page — lab column in test list
  • Issue details page — lab column in test list
  • Notifications/metrics endpoint — lab summary counts

[MarceloRobert]

If the lab name is unique in the database, why not use that as a primary key? In that case you'd make a first query to check which labs are not in the database yet, a second query to update the table if needed, but you wouldn't need the third query to get the id of the newly added labs

[alanpeixinho]

Mostly to avoid string keys, since this is an internal key, we might benefit more from integer indices.

[MarceloRobert]

Makes sense. I am worried that we are adding one query here, one query there and soon we will have a bloat of unnecessary queries in the ingester, but it's fine for now

[alanpeixinho]

It is a valid concern. If we plain to move some operations to ingestion time, we certainly are going to end up increasing "ingestion complexity". But I believe that, as long as we keep the ingester fast enough, simplifying the analysis step should be the goal.

[MarceloRobert]

I think you forgot a TODO here

[alanpeixinho]

Done

[mentonin]

Why not assign directly to lab like before?

[alanpeixinho]

Mostly because lab is now the column name, but thinking again. I might change the column name for lab_id, this way we keep the lab name free to use, and follow more closely the the foreign key nomenclature.

[alanpeixinho]

Changed mind, to really avoid rename hacking on the ORM, it was easier to just add a proper sql query instead.

[mentonin]

Why does this not use the fallback?

[alanpeixinho]

Done

[mentonin]

Shouldn't we use db_constraint=True?

[mentonin]

ditto for Tests

[mentonin]

I see all other tables are using db_constraint=False, so there must be a good reason for this, but I do not see it documented

[alanpeixinho]

For the kci keys, we unfortunately have some missing keys in a few rows. But since lab has only internal keys, I believe we can safely enforce constraints on them. Good point.

[mentonin]

Copy pattern from previous chunk for consistence and readability

[alanpeixinho]

The point is, this is the original implementation (pre-refactor)

    misc = sanitize_dict(build.misc) or {}
    lab = misc.get("lab", UNKNOWN_STRING)
    if lab:
        build_lab_summary = builds_summary.labs.get(lab)
        if not build_lab_summary:
            build_lab_summary = StatusCount()
            builds_summary.labs[lab] = build_lab_summary
        setattr(
            builds_summary.labs[lab],
            status_key,
            getattr(builds_summary.labs[lab], status_key) + 1,
        )

The conditional for lab is never false, because we are assigning UNKNOWN_STRING for lab.
However, it seems that this function is no longer used, and might be a legacy function used before performance improvements on HardwareDetails endpoints.
I will confirm it, and if this is the case, remove it.
We might as well plan to perform some dead code elimination, and check for similar cases.

[MarceloRobert]

Haven't tested yet, but looks good so far

[MarceloRobert]

very nit: prefer connections['default']

[alanpeixinho]

Done

[felipebergamin]

here the code we talked about. Not sure if this conditional should be removed

[alanpeixinho]

This one is tricky.
Post-migration we no longer read build_misc, so we can't distinguish "no misc at all" from "misc present but no lab" — we only have the lab field. And it's odd to branch on how lab is missing.
I think we should standardize: any missing lab as UNKNOWN.

[alanpeixinho]

If we do find a reason to treat them differently, we should include an explicit rule on this.

[felipebergamin]

why t.origin here ?
If we have neither a laboratory nor a runtime, won't the WHERE clause you wrote will filter out the record?

[alanpeixinho]

Nice catch. Missed removing some test here.

[alanpeixinho]

Fixed

[tales-aparecida]

I was a bit concerned with the 3 queries, so I've asked gemini to try to fold them into a single query, please review

Suggested change

	def assign_lab_ids(builds_buf: list[Builds], tests_buf: list[Tests]) -> None:
	"""Resolve each instance's real lab name to a labs.id and set its lab_id FK.
	Select-first so we only INSERT genuinely new labs (avoids burning the id
	sequence on every flush). New labs are committed outside the fact-insert
	transaction (autocommit).
	"""
	objs = [*builds_buf, *tests_buf]
	names = {name for obj in objs if (name := obj._lab_name)}
	id_map: dict[str, int] = {}
	if names:
	with connections["default"].cursor() as cursor:
	cursor.execute(
	"SELECT id, name FROM labs WHERE name = ANY(%s)", [list(names)]
	)
	id_map = {name: lab_id for lab_id, name in cursor.fetchall()}
	missing = [name for name in names if name not in id_map]
	if missing:
	cursor.executemany(
	"INSERT INTO labs (name) VALUES (%s) ON CONFLICT (name) DO NOTHING",
	[(name,) for name in missing],
	)
	cursor.execute(
	"SELECT id, name FROM labs WHERE name = ANY(%s)", [missing]
	)
	id_map.update({name: lab_id for lab_id, name in cursor.fetchall()})
	for obj in objs:
	obj.lab_id = id_map.get(obj._lab_name) if obj._lab_name else None
	def assign_lab_ids(builds_buf: list[Builds], tests_buf: list[Tests]) -> None:
	"""Resolve each instance's real lab name to a labs.id and set its lab_id FK.
	Uses a single optimized CTE join to insert missing records and fetch all IDs
	in 1 round-trip without sequence waste or ORM overhead.
	"""
	objs = [*builds_buf, *tests_buf]
	names = list({obj._lab_name for obj in objs if getattr(obj, "_lab_name", None)})
	if not names:
	for obj in objs:
	obj.lab_id = None
	return
	query = """
	WITH input_names AS (
	-- 1. Unnest and deduplicate inputs in Postgres C-memory
	SELECT DISTINCT unnest(%s::text[]) AS name
	),
	inserted AS (
	-- 2. Anti-join via LEFT JOIN / IS NULL (Faster than NOT EXISTS)
	INSERT INTO labs (name)
	SELECT i.name
	FROM input_names i
	LEFT JOIN labs l ON l.name = i.name
	WHERE l.name IS NULL
	RETURNING id, name
	)
	-- 3. Combine newly inserted IDs with existing IDs
	SELECT id, name FROM inserted
	UNION ALL
	SELECT l.id, l.name
	FROM labs l
	JOIN input_names i ON l.name = i.name;
	"""
	with connections["default"].cursor() as cursor:
	cursor.execute(query, [names])
	# Unpacks (id, name) tuples into a {name: id} dictionary directly
	id_map = dict(cursor.fetchall())
	for obj in objs:
	lab_name = getattr(obj, "_lab_name", None)
	obj.lab_id = id_map.get(lab_name) if lab_name else None

[felipebergamin]

great optimization but I think it might burn the id sequence in some cases, right? which is the concern of the original approach.

[tales-aparecida]

Actually, it doesn't...

but it's great that you mentioned that! We still need need ON CONFLICT (name) DO NOTHING like we have in the original code to avoid the race condition.

Only when the race condition happens there will be ID burning on both the original and using the single query

[alanpeixinho]

This is tricky, because in the original solution, despite having tree queries, only the first simpler query would hit most of the time (since we expect to have the number of labs being significantly smaller than the number of builds or tests ).
With the unified query we are running 100% of the time a more complex query (I will take a look how this impact performance).

[alanpeixinho]

On the topic of race condition (thanks for catching that), I believe this can be solved with a lock in the solution with split queries.

[tales-aparecida]

I'm concerned with how much overhead the labs will bring to the kcidb ingestion. either put it under a feature flag, so we can toggle it off quickly in case it's catastrophic, or show some metrics comparing the time to consume a few hundred payloads in a local development, before and after this MR