AI Vendor Discussion Guide
AI Vendor Discussion Guide
Feel confident that you've checked all the boxes
🛫
Getting started
Type of Tech: What's the core technology of your
model? Is it open-source, wrapper around someone else’s, or entirely
proprietary?
Deployment: If you manage deployment, how does it work? Can we
deploy this on our private cloud, or migrate if/when we want to?
Infrastructure: What hardware and software requirements are needed for
deployment?
Skills: What team skills are needed internally for successful deployment
and maintenance? Do you offer training support and services?
🧠
Making it smart
Using Our Data: How can we integrate our
proprietary datasets to enhance your model?
Data Retrieval: Can you describe the model's
context window (how much it can intake at a time) and its maximum output
length? How do you pull in more data when needed?
Who’s Involved: Do third-party tools process any
of our data?
Knowledge Management: How can we integrate
multiple knowledge bases with your model? How can we optimize efficiency when
scaled?
🚣♀️
Performance
Handling Demand: How does your system cope
during high-demand situations? How does performance fare with extensive
datasets?
Dealing with Errors: How does the model handle
“hallucinations”, i.e. erroneous outputs? Is there a human review process in
place?
Quality Assurance: How do you measure your
model's performance? How can we monitor the quality and accuracy of the model's
outputs?
🫶
Brand safety
Bias Prevention: What mechanisms are implemented
to reduce bias and inappropriate outputs?
Toxicity Standards: Do you follow certain benchmarks for detecting
harmful content? How frequently are these revised?
Output Filters: Are outputs subjected to any
automated filters?
Content Restrictions: Can we implement guidelines to flag or block
specific types of content?
🗣️
Feedback
Model Transparency: What tools are available to
understand the model's decision-making process?
Insights: Are insights available in real time? How does the
communication flow work in the event of a security issue?
Reporting: What kind of reports can be generated
to gauge the efficiency, accuracy and performance of the model?
User Feedback Process: How do you manage and act upon user
feedback?
🧑⚖️
Compliance
IP Concerns: How do you ensure that generated
content doesn't infringe on third-party IP rights? Are all outputs “enterprise
safe”?
Data: Who retains ownership of the input data
and generated content?
Privacy Standards: Which data privacy standards,
like GDPR, do you adhere to?
Legal Track Record: Are there any past, ongoing,
or anticipated legal issues related to your model?
Data Sources: From where did you source your
training data? How did you limit bias during model training?
Third-party Evaluation: Has your model been evaluated by an external party? Do you have certifications related to bias, toxicity, or data security?
🔐
Security
User Authentication: What authentication methods
do you support for users?
Access Protocols: Who on your team can access the foundational
models and our data? How long is data retained?
Single Sign-On (SSO): Do you facilitate SSO
capabilities? Is SCIM supported (for example, OKTA)?
Prompt Security: How are measures implemented to counteract
malicious prompts or exploitation events?
Thanks E7!
David
David Cutler
617-331-7852
