CloudBees Smart Tests is a multi-tenant SaaS. Data is sent from a customer’s site to CloudBees Smart Tests.
This document covers frequently asked questions related to information security (infosec).
Does CloudBees Smart Tests maintain a written security plan or other security governance?
Yes, we do. We can furnish details upon request.
Does CloudBees Smart Tests hold any security certifications?
CloudBees Smart Tests is SOC 2 Type 1 compliant as of August 2023.
We also utilize AWS as our cloud provider, who maintains SOC 2 Type 2, ISO 27001, and ISO 27017.
Is access to personal information limited only to those individuals who need access to the information to perform the services?
Yes. Access controls are specifically granted only to individuals whose job function requires it.
Where do you run your AI models?
We currently use OpenAI as our LLM provider. Their privacy policy and data protection practices are in line with ours. OpenAI’s models are used to summarize test failures, group failures together, and for some customers, to select tests that are relevant to code changes.
We also train and run various regression models on our own compute infrastructure, which currently operates on AWS.
Are there logging capabilities within CloudBees Smart Tests that capture all AI-related events?
CloudBees Smart Tests includes robust AI event logging and traceability capabilities. We are continuously innovating to incorporate the latest standards related to AI logging and telemetry.
Do we provide human oversight and validation of the results that AI models are accurate?
Yes. CloudBees Smart Tests employs a human-in-the-loop model for oversight and validation. Our engineers and QA teams review model outputs and recommendations before deployment to production.
Are there processes to minimize hallucination?
Yes. CloudBees Smart Tests implements multiple layers of safeguards to minimize hallucination and non-deterministic behavior.
Is there a summative description of all information used to develop, train, validate, test, or improve the AI System?
Yes. CloudBees maintains documentation describing the datasets, data sources, and usage context for AI model development and evaluation.
Links: - Data examples - Data Privacy & Protection
Are inputs into the AI model validated?
Yes. Inputs into CloudBees Smart Tests AI models undergo multi-stage validation before processing. We are continuously enhancing safeguards—following industry best practices—for AI models before public deployment.
Does the AI System provide an explanation of the prediction, recommendation, or decision generated by such AI System as an output?
Where relevant and applicable, test execution history, titles, and logs explain or inform recommendations generated. Additionally, we are continuously evaluating product feedback to provide more helpful tips and explanations to end users.
Do you maintain a DPIA covering your AI/LLM processing activities? Are there any internal mechanisms to monitor bias, transparency, or explainability of your AI models?
Given that our AI processing does not involve high risk to the rights and freedom of individuals and limited personal data, we do not maintain a DPIA as such. However, upon request, we can complete one. Yes, we have manual checks such as linters, code runs, and other algorithms.
You mention that OpenAI is used as a subprocesser. Please confirm whether SCCs are in place for transfers to the US and whether a TIA has been completed for OpenAI.
SCCs are included within OpenAI’s Data Processing Agreement, which CloudBees has agreed to.