Imagine for a moment you'd talked to Steve Jobs, pre-Apple, and asked him for his portfolio, prior experience, references, etc. Or imagine you'd talked to Lee Clow (the primary, but not the only, conceptor of the 1984 ad). Had he ever done anything like that? No.
So experience, portfolio, references -- sure, they're all OK -- but they miss something even bigger. How does the CD on your project THINK/problem solve/ideate. What questions does he or she ask? How incisive and intelligent are those questions? How quickly does the CD grok what you're doing and trying to do? How willing are they to challenge your beliefs or suggest alternatives? How well, how compellingly does this person communicate? What if it's John/Jane A Smith and the firm is named JAS Branding: now, there's a real winner! -- it's so bad, you know s/he couldn't possible have an original thought.
As the financial regulators like to remind us, past performance is no guarantee of future results. Otherwise, there'd never have been a New Coke.
You're hiring creativity and a brain or brains. You have to ask yourself: can this individual (or this individual leading a team for this firm or using other people) hit it out of the park.
Nothing else matters.
Feel free to contact me, 15 minutes gratis, if you have any questions.