[noxee]

How do you handle look ups that would require cross tenant queries? e.g. each customer has multiple devices connected devices to our system and we want to know the health (is it online, is it responding, etc).

The one way I've thought of current is either having an aggregate table in a "central" database that would be used to collect these kind of statistics. It would be "real-time" but it would be near real-time depending on the frequency of updates. The downside is you have the over head of maintaining a separate data source.

The other option was to just have the software set up to query each tenant at a time and take the performance/time hit. That's not really the best experience and probably violates the idea of data isolation.

[xemdetia]

I'm having a trouble understanding what you are saying by cross-tenant queries in context. What you are describing is application monitoring which would be completely independent of the DB holding customer data, which is what they are describing in your parent's post and the OP. Application runtime/health status could trivially be handled in a separate stream.

[noxee]

It's not really the application runtime/health status, it's the status of individual assets for each tenant. So a scenario would be as technical support person I would like to look at what customer assets are currently off-line across all tenants.

One way to do that would be to query each individual tenant database and then combine the information. But this would void the idea of security/isolation. The other downside is that it's slow.

But I think I've been looking at this from the wrong point of view, there should be a central health check service for these assets that collects the status of the assets and then that would be queried by the support person.

[xemdetia]

Generally I see it the other direction as a technical support person- each tenant's host is responsible for reporting uptime to a central service (because it's SaaS) and the customer is aware of this (even if it's basic syslog forwarding). Because we are not reaching in and they are sending out their customer data is still isolated.

Any time you are reaching into the customer datasets that is where you have the possibility of breach via that access and would have to make sure all of those events are audited and that audit records are secure. If it is sending monitoring data to an aggregation service that can be well-defined in contents and much easier to demonstrate that no sensitive tenant information is shared.

It's much easier and saner to setup a server to receive and configure each tenant's host to send exactly what is needed and then analyze who didn't send their stuff at regular intervals for a more direct investigation and everyone can agree on what needs to be done based on that.

[noxee]

Yeah I agree with what you're saying, as I said I was looking at the problem somewhat naively.

When you talk about having each tenant's host do you mean a dedicated application server for each host? The current setup we have is that we have multiple tenant databases but only two application servers providing functionality to them, so there's a switching cost on the application servers because they can't maintain a connection pool to the databases (they also have to know the credentials for the databases.

It seems excessive if you have a 100 tenants to have your infrastructure for serving the application replicated 100 times but maybe we're just haven't quite built our system correctly yet.

[xemdetia]

In the model I am describing every tenant component would have an agent that was reporting the state of that tenant. If there was a shared component/pool that wouldn't be reporting separately, I'm really only describing the tenant databases/tenant-specific files. So if you had a tenant database, there should be an agent that just is reporting on that tenant database.

As for separation of the application frontend it really is a risk assessment and culture thing. If the team understands that making sure you never cross streams of customer data is a critical requirement which is one of the first viability questions in the design then sharing really isn't too bad. If your team struggles with this or has recurring issues to this end separate application servers make more sense. Generally if you have tenants with data exposing another customer's data is a quick way to have trust in a product evaporate, I know I have had this happen as a customer where I was able to show a bug where I was getting another customer's accounting information in a cloud app which caused me to terminate the contract.

If you guys are minding the soup of the application servers well enough there's nothing say you can't, there are many people that do it.