Evaluating Identity Detection Coverage Across MSP Tenants

An empty incident queue can mean a quiet customer. It can also mean the connector stopped sending logs yesterday. Across several customer tenants, those two conditions need to be easy to distinguish.

For an MSP evaluating identity coverage, I'd begin with the data arriving from each tenant and the authority available to the analyst. A consolidated dashboard is useful only when it preserves that customer-specific context all the way through detection and response.

Know which layer produced the result

In a Microsoft Entra workforce environment, sign-in and audit logs record activity. Entra ID Protection adds Microsoft's risk detections. Sentinel can collect telemetry, run additional analytics, and create incidents. A result from one layer doesn't imply that the other layers saw or acted on the same event.

Microsoft's risky-user report documentation identifies P2 as the prerequisite for full risky-user data and distinguishes report access from actions such as password resets. Test the analyst's actual role. Being able to open the portal doesn't establish that the account can read the required details or perform the agreed response.

Keep a record for each tenant

Use the immutable tenant ID and workspace resource ID as the basis of the customer record. Add the owner, delegated roles, expected log categories, connector configuration, retention, and permitted response actions. Display names make the record readable, but joins and actions need stable identifiers.

Retention belongs in onboarding. Microsoft's Entra retention table lists seven days for standard audit and sign-in logs on Free and 30 days on P1/P2. Risk reports have their own periods. Upgrading later doesn't recover records that already expired, so a longer analysis window needs an appropriate destination configured in advance.

For each expected category, show the latest source time and ingestion time. These answer different questions. Old source events arriving now may indicate a backlog; a recent source event that took a long time to arrive indicates delay. Both can affect a scheduled detection.

A tenant that has stopped delivering a required category should show a data-health failure even when no threat rule matches. Otherwise the least observable customer can look like the healthiest one.

Carry customer identity through the case

A simple isolation fixture is useful here. Give tenant A and tenant B an identity with the same local ID. Put failed sign-ins and a later success in A, then put a role assignment in B. A central query must not join them into a single attack sequence.

Repeat that check beyond the query. Suppression keys, deduplication, ticket creation, enrichment, and remediation all need the tenant identity. A correct alert can still become a response against the wrong customer if an automation step drops that field.

Azure Lighthouse supports delegated management of customer Sentinel workspaces while the underlying data remains in those workspaces. An export or central ticket creates another copy with its own access requirements. Trace those copies when deciding what the MSP's analysts and integrations can see.

For cross-workspace analytics, Microsoft places alerts in the workspace where the rule is defined. That makes incident ownership a design choice: who works the central case, how is the customer identified, and which identity may act in that customer's tenant?

Test revoked delegation as well. A failed response should remain visible as a failed response, with the target tenant and error available to the analyst. Closing the ticket because an automation job ran would hide the unresolved access problem.

Check correlation before testing the platform

The downloadable identity_hunt.py provides a small starting fixture. It creates 35 invented records in an in-memory SQLite database and tests eight cases involving failed sign-ins, a successful sign-in, and a subsequent role grant.

The recorded results contain two intended matches. The cases also exercise a grant outside the time window, an inclusive boundary, too few failures, ordinary administrative activity, cross-tenant records, duplicate delivery, and a missing successful sign-in. The worked hunt explains the SQL and each outcome.

The local fixture establishes what that query does with those records. To evaluate Entra and Sentinel, run an authorized lab case through the actual sources, ingestion, rule, and incident workflow. Measure the time from the source event to an actionable case, then check whether an analyst with the intended role can complete the response.

The missing-success fixture is particularly useful during that review. Its query result is empty because the necessary event is absent. The service needs a separate way to report missing telemetry; changing the correlation threshold won't repair the source.

Account for the work around the alert

Price a pilot around the workload: licensed users and workload identities, bytes ingested by source, retention, analytics, automation, and analyst handling time. A tenant count alone doesn't describe a busy directory or the work required to onboard it.

Measure duplicated incidents and incomplete enrichment as well as detection delay. If an analyst has to switch accounts repeatedly to establish which customer a case belongs to, that cost will recur every day. The access and handoff questions belong in the customer review scope before the service starts.

For each tenant, I'd want a known data-health state and a response exercise that reaches the right customer with the right authority. Keep unresolved sources and permissions visible on that tenant's record. A fleet-wide percentage is a poor substitute when the missing coverage belongs to the customer whose incident arrives next.