This side-by-side, pitting Google against Siri, Alex, and Bixby, was interesting to watch.

The question set was limited, so it’s not really a definitive look. It’d be interesting to see an open source effort to create an extensive, defining set of questions to pose, one that could be passed along to the assistant team at any company with a voice assistant in the works.

If I was creating such a question set, I’d definitely push the boundaries, go for fully interactive conversations, with call-backs to previous questions, and questions that require the ability to plumb the depths of apps.

For example, how about questions that explore email and browsing habits, pull up orders from recent purchases, test the ability of the assistant to recall recent messaging conversations.

The tests could show how much “understanding” is going on, but also provide insight into privacy boundaries as well.

Just a thought.