How do I avoid the hype when evaluating AI tools?

Whether you’re looking for AI to streamline operations, improve community engagement, or predict investment outcomes, the right tools can be transformative. But choosing an AI vendor isn’t as simple as signing a contract and flipping a switch. Between buzzwords, inflated promises, and overly complex technical jargon, it’s easy to get overwhelmed - or worse, make the wrong choice.

Here’s the reality: not all AI tools & vendors are created equally. Some offer solutions tailored to your industry’s unique challenges, while others push generic tools that won’t move the needle. Some treat AI tools identically to SaaS or on-premise offerings, while others invest in continual AI monitoring. Evaluating AI vendors requires a clear framework.

Let's break down what to look for, the right questions to ask, and the red flags you need to watch out for to ensure you make the best decision for your organization.

Six Steps to Evaluating AI Tools & Vendors

Step 1: Define Your Goals Before Engaging Vendors

Before you even start talking to vendors, get crystal clear on what you want AI to achieve. This isn’t just about identifying pain points; it’s about defining measurable outcomes.

Being specific ensures you don’t get sidetracked by flashy features that sound exciting but don’t solve your actual problems. It also helps you quickly evaluate whether a vendor’s product aligns with your priorities.

Our Advice: Vendors love to pitch “solutions for everything.” Don’t fall for it. Stay laser-focused on how their product fits your goals.

Step 2: Look for Affordable Housing Expertise

AI solutions aren’t one-size-fits-all. Vendors with experience in affordable housing—or at least adjacent industries like market-rate housing, property management, or community development—are far more likely to understand your unique challenges. They’ll be familiar with compliance requirements, resident data sensitivities, and operational bottlenecks.

Questions to Ask:

• Have you worked with organizations in affordable housing or similar industries?

• Can you provide examples of how your tool has solved problems like ours?

Our Advice: Ask about case studies, customer success stories, or partnerships within the affordable housing sector. If a vendor doesn’t have this experience, proceed cautiously—they may underestimate the complexities of your work.

Step 3: Assess the Technology (Without Getting Lost in the Details)

AI is packed with technical terms, but you don’t need to be a data scientist to evaluate whether a vendor’s technology is a good fit. Focus on these key areas:

1. Ease of Use
   How intuitive is the tool for your team? If it requires extensive training or has a steep learning curve, adoption could be a struggle.

2. Integration
   Can the AI tool connect & utilize data with your existing systems, like property management software, accounting software, CRMs, or other tools? Integration issues can create major headaches down the line.

3. Scalability
   Does the solution have room to grow with your organization? What happens when your portfolio expands, or your needs evolve?

4. Transparency
   Vendors should be able to explain how the AI works and how it makes decisions, especially if the tool involves resident data or compliance processes. Avoid systems where decision-making processes are a black box.

Questions to Ask:

• How does your AI work, and how are decisions made?

• What integrations are supported? What does this support look like?

• How do you ensure your tool remains effective as our organization grows?

Step 4: Get Serious About Data and Governance

AI relies on data, and for affordable housing organizations, that data often includes sensitive information. A vendor’s approach to data handling and governance should be a top priority during your evaluation.

Here’s What to Look For:

• Data Ownership: Your organization should always retain ownership of the data processed by the AI. The vendor’s role is to use it responsibly - not claim it as their own.

• Privacy Protections: Ask whether the vendor complies with data protection laws like GDPR, HIPAA, or local housing regulations. Their practices should align with both legal requirements and your ethical standards.

• Governance Policies: Strong governance is about accountability. Vendors should have clear policies for monitoring AI algorithms, managing data access, ensuring version control, and creating audit trails. Role-based access should limit who can interact with sensitive information. Keep in mind: AI governance will look significantly different than your standard technology tools and platforms. AI algorithms need consistent monitoring, oversight, and training to avoid "AI drift" and the erosion of value - which requires a very different process than typical internal support.

• Bias Mitigation: AI tools should avoid reinforcing bias that may have been introduced in the initial creation & training for the AI model. The vendor should have procedures for auditing their algorithms regularly and retraining models to correct for bias.

Questions to Ask About Governance:

• Who owns the data processed by your tool?

• How do you handle data security and privacy?

• What governance frameworks do you follow, and how do you ensure accountability?

• What steps do you take to identify and mitigate bias in your AI models?

• How often are you adjusting the underlying model powering your AI tool?

• Does this align with our current technical vendor security, risk management, and governance policies?

Red Flag: If a vendor can’t explain their approach to governance or dodges questions about data security, walk away.

Our Advice: Good governance isn’t just about compliance—it’s about protecting your organization’s reputation and ensuring AI tools are used responsibly to benefit both your team and your residents.

Step 5: Evaluate Ongoing Education & Communication Practices

Even the best AI tools require a ramp-up period. The right vendor won’t just sell you software - they’ll partner with you to ensure success. Look for vendors that offer comprehensive training, ongoing support, and clear documentation.

Questions to Ask:

• What training resources do you provide?

• What does your support model look like (e.g., dedicated account managers, 24/7 help desk, robust support library, etc)?

• How do you ensure a smooth onboarding process?

• How do you communicate AI model or algorithm updates? How often do these updates occur?

Our Advice: Beware of vendors that disappear after the sale. If their support ends at contract signing or after implementation, you’ll be on your own when problems arise. Remember: vendor support for AI tools requires continuous monitoring, model training, and attention to remain effective.

Step 6: Demand Proof of ROI

AI investments should deliver measurable results, whether it’s saving time, cutting costs, or improving resident outcomes. And these results should align with what you're looking to achieve with any AI implementation. Any vendor worth your time should be able to demonstrate ROI through real-world examples or pilot programs.

Questions to Ask:

• What results have your other clients achieved?

• Can we run a small-scale pilot to evaluate impact before a full rollout?

• How long does it typically take to see ROI?

• What stalls or blocks progress toward ROI most often with your other clients?

Red Flag: If a vendors dodges questions about ROI or overpromises results with vague claims, walk away.

Watch for Other Red Flags

AI is a buzzy topic, and not every vendor is the real deal. Stay alert for these warning signs:

• Vague Explanations: If a vendor can’t explain how their tool works in plain language, they might not fully understand it themselves.

• Overpromising: Be wary of claims like “We can solve all your problems instantly.” AI is powerful, but it’s not magic.

• Lack of Transparency: Vendors who are cagey about pricing, data handling, or support aren’t worth your time.

The Bottom Line

Evaluating AI vendors is about more than finding a shiny tool with the most features. It’s about finding a partner who understands your goals, respects your data, and supports your team in achieving success. With the right vendor, AI can help your organization streamline operations, make better decisions, and ultimately serve residents more effectively.

The best vendors don’t just sell solutions—they collaborate with you to create impact. Ask the hard questions, assess their approach to governance, and trust your instincts. The right choice today will pay off for years to come.

Want additional advice on evaluating AI tools & vendors? The Strategic Edge can help. Book a 15 minute call with our advisors to chat about your objectives - we'll help pull together your business & technical requirements, ensure strategic alignment with your roadmap, and join you for the vendor evaluation process.