Podcast: My Interview on Rogue Startups - Evaluating AI Model Performance
Jan 22, 2025
I recently had the pleasure of speaking with Craig on the Rogue Startups podcast about how I'm using AI in both my academic role teaching engineering students and in running Student Voice AI. We covered a wide range of topics from systematic model evaluation to the future of AI and its impact on education and work.
Key Points from Our Discussion
- Teaching AI to engineering students: My approach focuses on critical thinking rather than just prompt engineering, helping students understand the capabilities and limitations of these tools
- Systematic model evaluation: How I test and compare different AI models (ChatGPT, Claude, Gemini) for specific use cases
- Local vs. cloud AI models: When and why I run AI models locally instead of using web-based services, particularly for Student Voice where data privacy is paramount
- Content generation strategies: The two primary approaches to AI-driven content creation and how I maintain quality in our outputs
- The future of AI and work: My perspective on how AI will transform careers and education
The Human Element Remains Critical
Throughout the conversation, I emphasised that while AI can replace tedious, low-value tasks, the human element remains essential. I advocate for a hybrid approach where AI handles the routine work while humans provide judgment, refinement, and personality.
It's going to remove the possibility of being mediocre... You find the thing you are really, really good at, bringing your special sauce, be that personality or technique or problem solving, and then you're using these tools to build, solve or create.
My Practical Rules of Thumb for AI Implementation
Some key recommendations I shared:
- Always test the newest models as they're released
- Keep prompts organised in plain text files for easy reuse and iteration
- Be systematic in your testing across different models
- Start with small automation tasks that remove tedium rather than trying to replace entire workflows
- Use local AI models when privacy concerns exist with sensitive data
Listen to the Full Episode
You can listen to the full episode on the Rogue Startups website or through your favourite podcast app.
If you're interested in discussing any of these topics further, feel free to connect with me on LinkedIn
This episode is part of a series where Craig is interviewing different professionals about how they're using AI in their businesses. If you have questions about my approach to AI in education or with Student Voice AI, don't hesitate to reach out.