Finding where the data really lives

• Explain why the declared data estate (diagrams, inventories, catalogues) differs from the real one, and where the gaps usually are.
• Triangulate inventories, interviews and query logs to find every source an agent's answers depend on, including shadow copies and spreadsheets.
• Produce a source map that records, per source, entities, fields, owner, system-of-record status, refresh cadence, quality, sensitivity, region and access path.
• Implement a small tool that builds a source map from an inventory plus access-history logs and flags shadow copies, unowned sources, stale copies and residency mismatches.
• Reconcile conflicting sources by choosing an authoritative source per attribute and getting the owner to sign the precedence rule.
• Plan access requests as critical-path items, with the right owner, purpose, scope and region for each.

Think of three maps of the same city. The planning office has the official street map: inventories, the configuration management database (CMDB), the data catalogue, the architecture diagram. The residents give directions: interviews with the people who do the work and who run the platforms. The traffic data shows where cars actually go: query logs, access history, lineage events, and what you see when you sit next to someone for an afternoon. Each is partly wrong. The official map misses the alleys, residents forget the routes they take without thinking, and traffic data only covers roads with sensors.

So you lay them over each other and look at where they disagree. A table in the logs but not in the inventory is a shadow copy or a forgotten pipeline. A system everyone mentions but nobody owns is an access request that will stall. A source shown as live whose last write was four days ago is a stale answer waiting to happen.

flowchart LR
    subgraph Declared
        I1[CMDB and EA repository]
        I2[Data catalogue]
        I3[Record of processing activities]
        I4[Vendor inventory]
    end
    subgraph Described
        D1[Business user interviews]
        D2[Platform owner interviews]
    end
    subgraph Observed
        O1[Query and access history]
        O2[Lineage events]
        O3[Workflow shadowing]
    end
    Declared --> T[Triangulate and diff]
    Described --> T
    Observed --> T
    T --> SM[Source map]
    SM --> AR[Access requests]
    SM --> RD[Readiness assessment]
    SM --> PII[Masking and residency design]
    SM --> LN[Lineage and catalogue]

The source map: what it is and what it is not

A source map is the engagement artefact that lists every data source the use case depends on, with enough detail to make decisions. It is not the customer's data catalogue, though it should end up in one. It is not the GDPR record of processing activities (RoPA), though it should agree with it. It is not an architecture diagram, though you will draw one from it. It is narrower than all three, because it is scoped to the questions your agent must answer, and more honest, because it is built from evidence.

A useful source map has one row per source object (table, view, API resource, document library, file) and these columns:

Scope it with the questions. Take the representative questions agreed with the sponsor, usually the same ones that seed your golden set (Golden datasets and the metrics that matter), and for each one write down the fields needed to answer it. The source map is complete when every one of those fields maps to a row with an owner and an access path. That gives you a stopping rule, which discovery otherwise lacks.

Discovery from inventories

Start with what the organisation already says it has. It is fast and it shows you which views of the estate are maintained. Typical inventories in a bank or insurer:

• CMDB and enterprise architecture repository. In a ServiceNow shop the CMDB lists applications, owners and support groups. It is usually good on applications and poor on the datasets inside them.
• Data catalogue. Informatica, Collibra, Unity Catalog, Microsoft Purview, DataHub or a similar tool. Good on technical metadata for the systems that were scanned; often weak on ownership and business meaning, and blind to anything that was never registered.
• Record of processing activities. Under GDPR Article 30, controllers keep records of their processing of personal data. The DPO's RoPA tells you which business functions process which categories of personal data, for what purpose, and where it is transferred. It is the closest thing to a sensitivity map the customer has.
• Vendor and SaaS inventory. The security or compliance team often keeps one for SOC 2 or ISO 27001. In Vanta, for example, the vendor inventory sits in the Vendors section of third-party risk management (as of September 2026). SaaS tools that hold customer data show up here even when IT does not run them.
• Integration and job schedules. ETL schedulers, integration platform flows and data-sharing agreements show which copies are made, how often and by whom.

Read inventories for what they leave out. If the catalogue has 4,000 tables from the warehouse and none from the SharePoint policy library, the policy documents are a discovery task, not a solved one.

Discovery from interviews

Interviews tell you what the inventories cannot: what the data means, which copy people trust, and what they do when the system is wrong. Interview two groups separately.

Platform and data owners (data platform lead, CRM platform owner, core banking operations, the DPO's office) know the official pipelines and the access rules. Ask them about refresh schedules, known defects, what is a copy of what, and who approves access. Expect them to describe the estate as designed.

Business users and their team leads know the estate as used. Do not ask "which systems do you use?" because you will get the official list. Ask about concrete, recent work:

• "Walk me through the last time a client asked you this. What did you open first, and then what?"
• "When the system's number looks wrong, where do you check it?"
• "What do you keep track of yourself because the system doesn't?"
• "What arrives in your inbox every week as an attachment?"
• "If this tool vanished tomorrow, what would you rebuild first?"
• "Who would you call if this figure was wrong?"

The last question finds the real owner more often than the CMDB does. Combine interviews with shadowing (Shadowing the work as it's really done): an hour watching someone answer client questions shows you the tabs, files and side channels they would never think to mention.

Discovery from query logs and access history

Logs are the closest thing to ground truth. Modern platforms record who read and wrote which objects, from which client, when. Mining them shows you copies nobody listed, tables that are heavily used but undocumented, pipelines that stopped running, and which tools (spreadsheets, BI tools, notebooks) pull which data.

On Snowflake, SNOWFLAKE.ACCOUNT_USAGE.ACCESS_HISTORY has one row per SQL statement with the user, the objects read directly (direct_objects_accessed), the underlying base objects (base_objects_accessed) and the objects written (objects_modified). It requires Enterprise Edition or higher, has up to about 3 hours' latency and keeps a year of data; QUERY_HISTORY has about 45 minutes' latency and is available on all editions. Access to both is granted through the GOVERNANCE_VIEWER database role or imported privileges on the SNOWFLAKE database (as of September 2026; see Account Usage). A discovery query looks like this:

-- Snowflake: read/write pairs for the last 30 days, one row per object read and object written.
-- Run by the customer's platform admin under a role with GOVERNANCE_VIEWER.
SELECT
    ah.query_start_time,
    ah.user_name,
    r.value:"objectName"::string AS object_read,
    w.value:"objectName"::string AS object_written
FROM snowflake.account_usage.access_history AS ah,
     LATERAL FLATTEN(input => ah.base_objects_accessed, outer => true) AS r,
     LATERAL FLATTEN(input => ah.objects_modified, outer => true) AS w
WHERE ah.query_start_time >= DATEADD('day', -30, CURRENT_TIMESTAMP());

On Databricks with Unity Catalog, system.access.table_lineage records read and write events with source_table_full_name, target_table_full_name, created_by, entity_type and event_time, kept on a rolling one-year window. system.query.history records executed_by, client_application, statement_type and start_time for queries on SQL warehouses and serverless compute. Two caveats: statement_text is returned as <REDACTED> unless you are an account admin or in the databricks_pii_access group, and the table only covers workspaces in the same region as the workspace you query from (as of September 2026; see lineage system tables and query history).

-- Databricks: which tables feed which, and which client tools read the most.
SELECT source_table_full_name, target_table_full_name, created_by, entity_type,
       MAX(event_time) AS last_seen, COUNT(*) AS events
FROM system.access.table_lineage
WHERE event_date >= date_sub(current_date(), 30)
GROUP BY source_table_full_name, target_table_full_name, created_by, entity_type;

SELECT client_application, executed_by, COUNT(*) AS statements
FROM system.query.history
WHERE start_time >= date_sub(current_date(), 30)
GROUP BY client_application, executed_by
ORDER BY statements DESC;

Know the limits. Logs cover one platform each; the mainframe, the CRM and file shares have their own logs or none. Lineage is recorded only where the platform can infer it. A table with no writer in 30 days may be refreshed quarterly. And query logs are sensitive data themselves (user names, sometimes customer values in query text), subject to residency like everything else. Ask the platform owner to run the queries, take results aggregated by role where you can, exclude statement text, and agree how long you keep the output.

The spreadsheet nobody admits to

Every enterprise runs partly on end-user computing: spreadsheets, Access databases, personal extracts, shared-drive CSVs and macros. Each exists because an official system did not meet a need: a pipeline tracker because the CRM does not model renewals the way RMs work, a limits sheet because the credit system's export is unusable. They often hold the most useful data in the building and the least governed. People rarely volunteer them, because admitting to one feels like admitting to a policy breach. Signals that one exists:

• Query logs showing a spreadsheet or BI client reading a sandbox or personal schema, especially many distinct users reading one object.
• CREATE TABLE AS SELECT from governed tables into a sandbox, by an analyst rather than a service account.
• Interview answers like "I check it against my sheet" or "the team lead sends it round on Mondays".
• Shared-drive or SharePoint files with names like Copy of, FINAL_v3 or a year in the name, opened daily.
• Numbers in interviews that do not match the system of record.

Treat discovery as an amnesty, not an audit. Say up front that you are mapping what people need, not reporting who broke a rule. Then decide what to do with each one: formalise it (move it into a governed table with an owner, because it holds data nobody else has), replace it (the agent answers the question it existed for, from the system of record), exclude it (the agent never reads it, and you say so in the source map), or escalate it (it holds Restricted data in an uncontrolled place, which is the data owner's or the DPO's problem to resolve, not yours to silently work around).

Data mapping: entities, fields, grain and keys

Discovery finds the sources. Mapping explains them. For each business entity your questions touch, record the physical objects that hold it, the fields you need, what each field means in business terms, the grain (one row per customer, per account, per account per day), and the keys you would join on.

Grain and keys are where agents go wrong. A text-to-SQL agent joining a per-customer table to a per-account-per-day table without knowing the grain will multiply balances. A CRM that keys customers by its own ID, with the core banking ID in an optional external-ID field, will drop every customer whose external ID is blank. Record match rates between keys as you map: "94% of active C360 customers have a matching CRM account" is a finding that changes the design. The deeper readiness work belongs to Is the data ready?; the point here is to capture the facts while you are talking to the people who know them.

Systems of record versus copies

A system of record (SoR) is the authoritative source for a piece of data: when two systems disagree, the SoR wins. Two refinements matter in practice.

First, authority is per attribute, not per system. Core banking owns balances and maturity dates. The CRM owns the RM assignment and relationship notes. The product office's rate card owns current rates. Master data management (MDM) tools such as Informatica MDM exist to make this explicit, with survivorship rules that pick the winning value per attribute when building a golden record. If the customer has MDM, its rules are a gift: they are the precedence decisions someone already negotiated.

Second, most of what you will query is a copy. The customer 360 is a nightly copy of core banking; group reporting copies it again; an analyst's sandbox copies it a third time. Each hop adds latency, may add transformations, and may lose controls: a masking policy on the original table does not follow a CREATE TABLE AS SELECT into a sandbox. Record the chain. Point the agent at the nearest governed copy with a named owner and a known refresh, and note why it does not read the SoR directly (usually: the mainframe is reachable only through the nightly extract).

Owners, stewards and custodians

Three roles, often confused:

• Data owner: accountable for the data, decides who may access it and for what. Usually a business role (Head of Core Banking Operations, Head of Retail Products).
• Data steward: maintains definitions, quality rules and documentation. Often a senior analyst in the owning team.
• Custodian: runs the platform the data lives on (the data platform team, the CRM platform team). Custodians grant access technically, but they should not approve it on the owner's behalf.

Platform teams often present themselves as owners because they hold the keys, until a question comes up ("can RMs see each other's clients?") that only the business owner can answer. Record owners by role, with the current person in a separate column, so the map survives a reorganisation. An in-scope source without an owner is a blocker: nobody can approve access, fix defects or sign lineage evidence. Raise it with the sponsor in week one.

Refresh cadence and freshness

Every source has a declared cadence ("daily at 02:00") and an observed one (when it was actually last written). Record both. Observed cadence comes from load or copy history, the lineage tables, MAX(updated_at) on the object itself, or the scheduler's run history. A daily job that has been failing silently for three days is common, and an agent that reports "maturing this week" from a three-day-old copy is wrong in a way users notice fast.

Freshness feeds three later decisions. Answers should carry an "as of" timestamp drawn from the source's last refresh (Answers with receipts: grounding and citations). Use cases that need fresher data than the cadence allows need events instead of batch (Streaming data for real-time agents). Freshness checks become quality rules with owners (Lineage: proving where an answer came from).

Quality signals at discovery time

You are not doing the full readiness assessment yet, but cheap profiling during discovery saves weeks later. For each candidate source, with the owner's permission and on masked data where needed, capture: row counts against what the owner expects, null rates on the fields you need, key uniqueness, orphan rates on joins, the date range covered, and a reconciliation count against the SoR ("C360 has 2.31 million active customers; core banking reports 2.33 million"). Write down known defects that people mention in interviews; they are rarely in any catalogue. Hand the detailed work to Is the data ready? and Lineage: proving where an answer came from.

Sensitivity classification and region

Classify every row of the map. Use the customer's scheme if it has one (commonly Public, Internal, Confidential, Restricted), and record personal data categories alongside it: identifiers such as national ID and tax ID, financial data, special categories under GDPR, payment data under RBI's 2018 direction on payment system data storage. Two rules make this tractable:

• Inheritance. A copy is at least as sensitive as the most sensitive thing it was built from, unless a documented transformation (aggregation, tokenisation) reduces it. A sandbox built from a Restricted table is Restricted.
• Region travels with the data. Record where each object is stored and processed. A copy of EU customer data in an India-region warehouse is a transfer, whatever the reason it was made.

Automated classifiers help. Informatica's Cloud Data Governance and Catalog and Collibra's catalogue both advertise automated classification of sensitive data such as PII, and Snowflake exposes classification results in ACCOUNT_USAGE.DATA_CLASSIFICATION_LATEST on Enterprise Edition and above (as of September 2026). Treat their output as candidates for the steward to confirm, not as the classification. Free-text fields such as CRM notes defeat column-level classifiers. Discovery and masking at depth belong to Finding and masking personal data, and residency at depth to Data residency and sovereignty.

Access paths

For each in-scope source, record how the agent would reach it. The options differ in latency, control and approval effort.

Record the network path and region too. A CRM API that is fast from Mumbai may sit behind a proxy that routes through another region. The access path column is where the source map turns into engineering tasks.

Catalogue-assisted discovery

If the customer has a catalogue, use it first. It turns days of interviews into an afternoon of reading, and it is where your source map should live after you leave.

Informatica. Informatica is now part of Salesforce (the acquisition completed on 18 November 2025) and its cloud platform is the Intelligent Data Management Cloud (IDMC). Within IDMC, Cloud Data Governance and Catalog combines catalogue, governance and data quality. Administrators configure catalog sources and scanners in Metadata Command Center to extract metadata, which you then browse, with lineage and classification, in Data Governance and Catalog. CLAIRE is the AI layer across IDMC: CLAIRE GPT is a standalone service with a discovery agent that searches IDMC services for relevant data assets, alongside data quality, data integration and product help agents. In 2026 Informatica has also positioned CLAIRE as a "headless" multi-agent layer that other platforms and agents can call (as of September 2026; see CLAIRE GPT on Trailhead and the Cloud Data Governance and Catalog page). Names and packaging have changed several times; check the customer's contract for what they actually licence.

Collibra, platform and open-source catalogues. Lineage: proving where an answer came from compares the options (Collibra, Unity Catalog, Microsoft Purview, DataHub, OpenMetadata) and owns how a catalogue is run. For discovery, Collibra's value is its governance layer: owners, stewards and approved definitions, the hardest part of the source map to get. Platform catalogues such as Unity Catalog and Snowflake's metadata views give technical metadata, tags and lineage only within their own platform.

Catalogues give you scanned systems, schemas, technical lineage, classification candidates and sometimes owners and glossary terms. They miss anything never scanned (file shares, spreadsheets, SaaS outside IT, the mainframe if no scanner was configured), ownership nobody updated after rollout, and business meaning never written down. A catalogue's AI assistant answers confidently from that same incomplete metadata. Treat catalogue output as the declared view in your triangulation, not as the answer.

Reconciling conflicting sources

You will find the same attribute in several places with different values. A customer's segment is "Emerging Corporate" in the CRM and "SME-2" in core banking. A deposit's maturity date in the warehouse differs from the RM's tracker by a week. Do not pick one quietly. Work through it:

1. Quantify. Sample matched records and measure the disagreement rate per attribute. "3% of segments differ" and "41% differ" are different problems.
2. Explain. Find the cause: different definitions (month-end segment versus live segment), timing (nightly copy versus live), transformations (code mapping tables), manual edits (the tracker records the renewal instruction, not the contract).
3. Decide per attribute. Choose the authoritative source for each attribute and the rule for when it is missing or late. Often the answer is "both, labelled": the contractual maturity date from core banking and the RM's renewal note from the tracker are different facts.
4. Get it signed. The data owner, not you, approves the precedence rule. Record it in the source map and, ideally, the catalogue's glossary.
5. Encode it. The curated view the agent reads implements the rule, so the agent never has to choose between conflicting values at answer time.

An agent given two conflicting sources and asked to reconcile them in the prompt will produce a different answer on different days. Reconciliation is a data decision made once, by an owner, in code.

Access requests and owners

Access requests are the slowest part of most engagements. At Northvale each one takes one to three weeks. Generate them from the source map, not from memory, and write them so the owner can approve without a meeting:

• What: the specific objects and fields, not "the C360 schema". Data minimisation is easier to approve.
• Why: the use case and the questions that need these fields.
• Who: the identity that will access it (a named service principal for the agent, on-behalf-of the user where possible), and who on your team needs access for development.
• Where: the environment (development with masked data, then production), the region and the network path.
• How long: an end date for development access, and review dates for production access.
• Controls: read-only, row-level policy, masking, logging.

Sequence them on the critical path. File development access to masked data first, because it unblocks the build. File production access early enough that the approval cycle finishes before the pilot. Track every request with its owner, date filed and status in the source map, and escalate stalled ones through the sponsor. A request sent to a custodian instead of the owner will be forwarded at best and silently parked at worst.

You are two days into the Northvale relationship-manager assistant. You have the data platform team's inventory, which lists what they believe exists, and a 30-day extract from Snowflake's access history that their admin produced with the query above, reduced to user, client application, objects read and objects written. The script below builds a source map from both, flags what needs attention, and prints Markdown you can paste into the engagement workspace.

Requirements: Python 3.11+, standard library only. Runs offline.

"""Build a source map from a system inventory plus warehouse query logs.

Standard library only; runs offline. Flags shadow copies, unowned sources,
stale copies, residency mismatches and spreadsheet feeds, then prints a
Markdown source map you can paste into the engagement workspace.
"""
from __future__ import annotations

from collections import defaultdict
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone

NOW = datetime(2026, 9, 14, 9, 0, tzinfo=timezone.utc)  # fixed so the output is reproducible
LOGGED_SYSTEMS = {"Snowflake (IN)"}  # the only system whose query logs we were given
SPREADSHEET_CLIENTS = {"Microsoft Excel", "Google Sheets"}
SENSITIVITY_RANK = {"Public": 0, "Internal": 1, "Confidential": 2, "Restricted": 3}
SEVERITY_RANK = {"high": 0, "medium": 1, "low": 2}
MAX_AGE = {  # declared cadence -> oldest acceptable refresh, with slack for late-running jobs
    "real-time": timedelta(hours=1),
    "daily": timedelta(hours=26),
    "weekly": timedelta(days=8),
}


@dataclass
class Source:
    name: str                  # fully qualified object name or document path
    system: str
    entity: str
    role: str                  # "system of record" | "copy" | "shadow copy" | "unknown"
    region: str                # where the data is stored: "IN" or "EU"
    sensitivity: str           # Public | Internal | Confidential | Restricted
    cadence: str               # declared refresh cadence, or "unknown"
    owner: str | None = None
    copy_of: tuple[str, ...] = ()
    access_path: str = "none agreed"
    documented: bool = True    # False if we only found it in the logs


@dataclass(frozen=True)
class QueryEvent:
    """One row derived from access history: who read and wrote which objects, from which client."""
    ts: datetime
    user: str
    client: str
    reads: tuple[str, ...]
    writes: tuple[str, ...] = ()


@dataclass(order=True)
class Finding:
    rank: int
    name: str
    severity: str = field(compare=False)
    message: str = field(compare=False)


@dataclass
class SourceMap:
    sources: list[Source]
    findings: list[Finding]
    readers: dict[str, set[str]]
    last_write: dict[str, datetime]


def ts(text: str) -> datetime:
    return datetime.fromisoformat(text).replace(tzinfo=timezone.utc)


INVENTORY: list[Source] = [
    Source("CBS.CIF.CUSTOMER", "Core banking (mainframe)", "Customer", "system of record",
           "IN", "Restricted", "real-time", "Head of Core Banking Ops", access_path="nightly extract only"),
    Source("CBS.DEP.TERM_DEPOSIT", "Core banking (mainframe)", "Term deposit", "system of record",
           "IN", "Restricted", "real-time", "Head of Core Banking Ops", access_path="nightly extract only"),
    Source("C360.CORE.CUSTOMER", "Snowflake (IN)", "Customer", "copy", "IN", "Restricted", "daily",
           "Data Platform Lead", ("CBS.CIF.CUSTOMER",), "curated view, read-only role"),
    Source("C360.CORE.TERM_DEPOSIT", "Snowflake (IN)", "Term deposit", "copy", "IN", "Restricted", "daily",
           "Data Platform Lead", ("CBS.DEP.TERM_DEPOSIT",), "curated view, read-only role"),
    Source("PRODUCTS.REF.RATE_CARD", "Snowflake (IN)", "Product rate", "system of record", "IN",
           "Internal", "daily", "Product Office", access_path="curated view, read-only role"),
    Source("SFDC.FSC.ACCOUNT_TEAM", "Salesforce FSC", "RM assignment", "system of record", "IN",
           "Confidential", "real-time", "CRM Platform Lead", access_path="REST API, on-behalf-of RM"),
    Source("eu_main.crm.customers", "Databricks (EU)", "Customer", "copy", "EU", "Restricted", "daily",
           None, ("EU core banking",), "Unity Catalog view (pending)"),
    Source("C360.GROUP_RPT.EU_CUSTOMERS", "Snowflake (IN)", "Customer", "copy", "IN", "Restricted", "weekly",
           "Group Finance Reporting", ("eu_main.crm.customers",)),
    Source("sharepoint:/policies/india", "SharePoint", "Policy document", "system of record", "IN",
           "Internal", "weekly", "Policy Office", access_path="Graph API, document ACLs"),
]

# Names starting with "@" are stages or external feeds, not tables.
QUERY_LOG: list[QueryEvent] = [
    QueryEvent(ts("2026-09-14T01:30"), "svc_etl", "Python connector", ("@CBS_EXTRACT",), ("C360.CORE.CUSTOMER",)),
    QueryEvent(ts("2026-09-11T01:40"), "svc_etl", "Python connector", ("@CBS_EXTRACT",), ("C360.CORE.TERM_DEPOSIT",)),
    QueryEvent(ts("2026-09-13T22:00"), "svc_ref", "Python connector", ("@RATE_FEED",), ("PRODUCTS.REF.RATE_CARD",)),
    QueryEvent(ts("2026-09-09T03:00"), "svc_grp_rpt", "Spark connector", ("@EU_SHARE",), ("C360.GROUP_RPT.EU_CUSTOMERS",)),
    # An analyst builds a pipeline tracker from governed tables in a sandbox schema...
    QueryEvent(ts("2026-09-13T18:05"), "a.mehta", "Snowsight",
               ("C360.CORE.CUSTOMER", "C360.CORE.TERM_DEPOSIT"), ("SANDBOX.RM.PIPELINE_TRACKER",)),
    # ...an old extract nobody remembers creating is still being read...
    QueryEvent(ts("2026-09-12T10:00"), "rm.k.iyer", "Microsoft Excel", ("SANDBOX.RM.SME_LIMITS_2024",)),
    QueryEvent(ts("2026-09-12T10:20"), "rm.s.das", "Microsoft Excel", ("SANDBOX.RM.SME_LIMITS_2024",)),
]
# ...and fourteen RMs pull the tracker into Excel every morning.
QUERY_LOG += [
    QueryEvent(ts("2026-09-14T04:00") + timedelta(minutes=i), f"rm.user{i:02d}", "Microsoft Excel",
               ("SANDBOX.RM.PIPELINE_TRACKER",))
    for i in range(14)
]


def build_source_map(inventory: list[Source], log: list[QueryEvent]) -> SourceMap:
    sources = {s.name: s for s in inventory}
    readers: dict[str, set[str]] = defaultdict(set)
    sheet_readers: dict[str, set[str]] = defaultdict(set)
    writers: dict[str, set[str]] = defaultdict(set)
    inputs: dict[str, set[str]] = defaultdict(set)
    last_write: dict[str, datetime] = {}

    for ev in log:
        tables_read = [r for r in ev.reads if not r.startswith("@")]
        for obj in tables_read:
            readers[obj].add(ev.user)
            if ev.client in SPREADSHEET_CLIENTS:
                sheet_readers[obj].add(ev.user)
        for obj in ev.writes:
            last_write[obj] = max(ev.ts, last_write.get(obj, ev.ts))
            writers[obj].add(ev.user)
            inputs[obj].update(tables_read)

    # Step 1: anything the logs show that the inventory does not.
    for obj in sorted((set(readers) | set(writers)) - set(sources)):
        upstream = tuple(sorted(inputs.get(obj, ())))
        known = [sources[u].sensitivity for u in upstream if u in sources]
        # A copy is at least as sensitive as its most sensitive input; unknown provenance defaults high.
        inherited = max(known, key=SENSITIVITY_RANK.__getitem__) if known else "Confidential"
        sources[obj] = Source(obj, "Snowflake (IN)", "unknown", "shadow copy" if upstream else "unknown",
                              "IN", inherited, "unknown", copy_of=upstream, documented=False)

    findings: list[Finding] = []

    def add(sev: str, name: str, msg: str) -> None:
        findings.append(Finding(SEVERITY_RANK[sev], name, sev, msg))

    # Step 2: rules. Each one is a question you would otherwise have to remember to ask.
    for s in sources.values():
        sensitive = SENSITIVITY_RANK[s.sensitivity] >= SENSITIVITY_RANK["Confidential"]
        if s.role == "shadow copy":
            built_by = ", ".join(sorted(writers[s.name]))
            add("high", s.name, f"Shadow copy of {', '.join(s.copy_of)}; inherits {s.sensitivity}; "
                                f"built by {built_by}; {len(readers[s.name])} distinct readers.")
        elif not s.documented:
            add("medium", s.name, f"Read by {len(readers[s.name])} users, not in the inventory and no writer "
                                  "seen in the log window: provenance unknown.")
        if s.owner is None and s.documented:
            add("high" if sensitive else "medium", s.name, "No named owner in the inventory.")
        if sheet_readers[s.name]:
            add("medium", s.name, f"Feeds spreadsheets: {len(sheet_readers[s.name])} users read it from Excel or Sheets.")
        for up in s.copy_of:
            src = sources.get(up)
            if src is None and s.documented:
                add("medium", s.name, f"Upstream '{up}' is not in the inventory: map it or record why not.")
            elif src is not None and src.region != s.region:
                add("high", s.name, f"Residency: {s.region} copy of {src.region} data ({up}). Confirm the basis with the DPO.")
        if s.role == "copy" and s.system in LOGGED_SYSTEMS:
            seen = last_write.get(s.name)
            if seen is None:
                add("medium", s.name, "Declared as a copy but no refresh seen in the log window.")
            elif NOW - seen > MAX_AGE.get(s.cadence, timedelta(0)):
                hours = int((NOW - seen).total_seconds() // 3600)
                add("medium", s.name, f"Stale: last refresh {hours} h ago against a declared '{s.cadence}' cadence.")
        if s.documented and s.access_path == "none agreed":
            add("low", s.name, "No access path agreed for the agent yet.")

    ordered = sorted(sources.values(), key=lambda s: (s.entity.lower(), s.name.lower()))
    return SourceMap(ordered, sorted(findings), readers, last_write)


def to_markdown(sm: SourceMap) -> str:
    lines = [f"# Source map: Northvale RM assistant (generated {NOW:%Y-%m-%d})", "",
             "| Object | System | Entity | Role | Owner | Cadence | Last refresh | Sensitivity | Region | Readers | Access path |",
             "|---|---|---|---|---|---|---|---|---|---|---|"]
    for s in sm.sources:
        seen = sm.last_write.get(s.name)
        last = f"{seen:%Y-%m-%d %H:%M}" if seen else "not logged"
        lines.append(f"| `{s.name}` | {s.system} | {s.entity} | {s.role} | {s.owner or '**none**'} | {s.cadence} "
                     f"| {last} | {s.sensitivity} | {s.region} | {len(sm.readers.get(s.name, ()))} | {s.access_path} |")
    lines += ["", "## Findings", ""]
    lines += [f"- **{f.severity.upper()}** `{f.name}`: {f.message}" for f in sm.findings]
    return "\n".join(lines)


if __name__ == "__main__":
    print(to_markdown(build_source_map(INVENTORY, QUERY_LOG)))

Running it prints an 11-row source map table (the nine inventoried sources plus the two sandbox objects found in the logs, both with owner none) followed by these findings:

- **HIGH** `C360.GROUP_RPT.EU_CUSTOMERS`: Residency: IN copy of EU data (eu_main.crm.customers). Confirm the basis with the DPO.
- **HIGH** `SANDBOX.RM.PIPELINE_TRACKER`: Shadow copy of C360.CORE.CUSTOMER, C360.CORE.TERM_DEPOSIT; inherits Restricted; built by a.mehta; 14 distinct readers.
- **HIGH** `eu_main.crm.customers`: No named owner in the inventory.
- **MEDIUM** `C360.CORE.TERM_DEPOSIT`: Stale: last refresh 79 h ago against a declared 'daily' cadence.
- **MEDIUM** `SANDBOX.RM.PIPELINE_TRACKER`: Feeds spreadsheets: 14 users read it from Excel or Sheets.
- **MEDIUM** `SANDBOX.RM.SME_LIMITS_2024`: Read by 2 users, not in the inventory and no writer seen in the log window: provenance unknown.
- **MEDIUM** `SANDBOX.RM.SME_LIMITS_2024`: Feeds spreadsheets: 2 users read it from Excel or Sheets.
- **MEDIUM** `eu_main.crm.customers`: Upstream 'EU core banking' is not in the inventory: map it or record why not.
- **LOW** `C360.GROUP_RPT.EU_CUSTOMERS`: No access path agreed for the agent yet.

Read the findings as an agenda for the next three meetings:

• The residency finding goes to the Dublin DPO first. A group-reporting job copies EU customers into the India warehouse weekly. Whether it has a legal basis is not your call. Keeping it off the agent's allow-list is.
• The pipeline tracker is the spreadsheet nobody admitted to. Fourteen RMs read it into Excel every morning. It is built from Restricted tables into an unowned sandbox, almost certainly without its inputs' masking policies. It is also evidence of a real need: RMs track renewal intentions that core banking does not hold. Decide with the data owner whether to formalise it or replace it.
• The stale term-deposit copy breaks the sponsor's first example question. "Which clients have a deposit maturing this month?" would be answered from data 79 hours old. Raise it with the platform lead and make answers carry an "as of" timestamp.
• The unowned EU table blocks the Dublin pilot. Nobody can approve access to eu_main.crm.customers. Ask the Dublin sponsor to name an owner this week.
• The SME limits extract needs provenance. Either it is out of date and should be retired, or it holds something the credit system does not.

The script is deliberately simple. Its value is that the rules are written down and repeatable: when next week's log extract arrives, you rerun it and diff the findings. In a real engagement you load the inventory from the catalogue export and the events from the access-history query.

Which discovery method answers which question

What to do with a shadow source

Stopping rule for discovery. You can stop when all of these hold:

• Every field needed by the agreed representative questions maps to a source row.
• Every in-scope source has an owner by role and a named person, both confirmed.
• Every in-scope source has a role (SoR or copy with upstream), declared and observed cadence, sensitivity and region.
• Every conflict between sources on an in-scope attribute has a signed precedence rule.
• Every in-scope source has an access path and a filed access request.
• The last two interviews surfaced no new sources.
• Out-of-scope and escalated sources are listed with reasons.

• Builds on: Parsing and chunking documents that fight back, Retrieval that respects who can see what, Tools with scoped, auditable access, Authorization and multi-tenancy.
• Used by: Is the data ready?, Agents over tables: text-to-SQL and beyond, Finding and masking personal data, Lineage: proving where an answer came from, Data residency and sovereignty, The audit pack, Agents inside CRM, ERP and ITSM, Feasibility and data access, fast.
• Related: Shadowing the work as it's really done, Finding the real requirement, Keeping the knowledge base fresh.

• Snowflake: Access History: how ACCESS_HISTORY records direct, base and modified objects per statement, with worked queries for read, write and column-lineage analysis.
• Snowflake: Account Usage: latency, retention, edition requirements and database roles for every metadata view you would use in discovery.
• Databricks: Lineage system tables reference: schema and retention of system.access.table_lineage and the column lineage table, and their capture limits.
• Databricks: Query history system table reference: the system.query.history schema, client application field, statement-text redaction and regional scope.
• Informatica: Cloud Data Governance and Catalog: the IDMC catalogue and governance service, with classification and lineage capabilities.
• Trailhead: Work with CLAIRE GPT: what CLAIRE GPT and its discovery, data quality and integration agents do inside IDMC, with example prompts.
• Collibra: Data Catalog: catalogue, classification, ownership and data contract features from a governance-first vendor.
• ICO: How do we document our processing activities?: UK regulator guidance on using information audits and data mapping to build records of processing; a useful model for the sensitivity side of a source map.