top of page

Virtual Humans

Just like Virtual Reality the whole chatbot/virtual assistant/conversational-AI space comes loaded with terminology baggage. The recent rise of virtual influencers, digital twins, virtual beings and digital humans only serves to complicate things. I wrote my first chatbot back in the mid 90s, and soon had it hooked up to the wonderful (and now venerable) Sitepal service to turn it into an animated head-and-shoulders talking avatar. When I was in Active Worlds in the late 90s I was building virtual characters whose "brain" resided on the web, not in the world, and our work on virtual characters in Second Life was so good (well elegant) that we passed one formal covert Turing Test and numerous informal ones. Another of our bots also passed a group covert Turing Test within an online chat environment with a 100% success rate!

In 2017 Professor Maggi Savin-Baden and I published our book on Virtual Humans, and this diagram from the book encapsulates much of my thinking, in particular that the "brain" of a virtual human should be pretty much independent of its manifestation, be that as an avatar in one or more virtual worlds, as a 2D avatar on a web page or text-bot on an IM service, as a robot or android in the physical world - or a combination of all of them all at the same time!

VH Component Architecture.PNG

The AI Landscape

AI Barriers.PNG

I'm very wary when people start talking about AI - Algorithms Incorporated is all most people (well marketeers) seem to mean, and it certainly shouldn't be synonymous with Neural Networks type Machine Learning.

I find this diagram a useful one to situate what I mean by AI, and place what other people are talking about when they mention AI. In the public imagination "AI" is still that think right over on the right-hand-side, whereas most modern "AI" is buried in a little box down by Complex Algorithms. What interests me are the 3 challenges the diagram shows - how can we make an AI that seems human (with an emphasis on elegant not brute-force approaches), how can we make an AI that is general-purpose (way beyond my capabilities, but forecast for around 2070-2100), and can we ever make a virtual being that is sentient - which I take to mean has its own subjective reality, internal narrative and dialogue, sense of purpose and sense of self-awareness. Lots more about all of that in the book!

Conversational AI

New Abi - Mar 2012.JPG

In more practical terms, a lot of the virtual human work I do tends to resolve around conversational AI, aka Chatbots. There tend to be three broad approaches to conversational AI:

  • Rules-based, typified by Artificial Intelligence Markup Language and Chatscript

  • Machine-Learning based, typified by RASA and Watson

  • Grammar and Semantic based - well under-researched

There is no one best way, each project should use the most appropriate approach, and may well use them in combination.

To me one of the key ingredients of an advanced bot is how information is stored and accessed. In this I think that semantic networks of triples (or similar) are vital, so that they more closely mirror how the brain seems to organise information, and enable to the bot to "random walk" its memory network and come up with with long but cohesive statements and responses which far better mirror human speech and conversation - quite apart from it also being able to find original insights on the data.

Use Cases

These are the main areas which interest me in the use of virtual humans, and where I may be able to help you in your thinking.

DesktopVR

chatwithmom.PNG

Non-Player Characters

vLearning and vTraining experiences without anybody else in them seem very empty. At the very least having an in-world guide can significantly help with immersion and keep the learner "in context" if they have problems. Using virtual environments to explore soft-skills has a long history, and I've done projects around customer service, autistic spectrum and patient care. In fact the research shows that even just having virtual bystanders looking on can improve the efficacy of the training!

  • Make virtual environments seem more lived in and more real

  • Help students through a task through an in-world guide, coach, mentor, expert or motivator - all subtly different roles

  • Use van Velsens film inspired 3-tier model of NPCs (extras-minor roles-stars) to focus your NPC investment

  • Store as much of the NPC logic outside of the world for portability, re-use and ease of management

  • Think about NPC race, gender and orientation and how they affect the training, and the trainee's lived experience

  • Text-based input is problematic in VR, and speech-recognition still not quite there - so how best to implement NPCs in VR?

  • Use visual cues to reflect NPC inner states

  • Think through the pros and cons of using recorded voice or text-to-speech

DesktopVR

tweetstream.PNG

Synthetic Wraps

Not all Virtual Humans need to be particularly visible. We've done a number of projects where we create multiple agents that then respond to an evolving situation and then message about it in simulated social media. This can let you take a relatively dry scenario or exercise and suddenly have your every action being commented on - just as it might be in the real world.

  • Give participants a better idea of the real-world impact of their actions

  • Use social media as a potential additional open-source intelligence channel

  • By automating the "background noise" hand-crafted messages that reflect a golden-thread or Master Event List serial aren't so obvious to the players

  • Have agents reflect different levels of chattiness, and expertise

  • Give agents allegiances and sentiments, and have them change dynamically as the player actions evolve

  • Use templating, variables and synonym approaches to generate thousands of message variations for minimal human set-up effort.

DesktopVR

anger management - oblique.JPG

Personal and Employee Support

I've worked on a number of projects around employee and personal support bots - putting information directly into someone's hand in an engaging and private way. Our research has showed that people sometimes appreciate the ability to find out information or explore ideas before talking to their manager or a health professional, and properly implemented such a bot could become a something of a life-long life-coach and virtual mentor.

  • Leverage the concept of "anonymous intimacy" - where people find talking to a chatbot an intimate experience, but also thinks its anonymous and non-judgmental - so they tend to reveal more than when talking to a human or just filling out a form

  • Think carefully about the persona of the bot, and how it might need to reflect the persona of the user (either echoing it or deliberately countering it)

  • The information exchanged is likely to be highly personal and confidential, so security and privacy are paramount

  • Integrate the mental health/support aspects of the bot with support for day-to-day activities, so the bot can be pro-active, and talking to the bot doesn't become a chore.

DesktopVR

6Hats-bordered.png

Reflexive Agents

Whilst trying to create bot which knows everything is a big ask, why not have a bot which knows that you probably have the answer, and can work with you to get it out - after all isn't this how many counsellors and gurus work? Inspired by Edward de Bono's Six Hats, but now moving well beyond it, we've developed reflexive bots on two MOD projects - Future Tools and Intelligent Ship to help extract the wisdom from the users.

  • The Reflexive Bot starts by identifying the type of issue in order to select the appropriate set of agents

  • The Bot asks some clarification questions which can help to choose the most appropriate questions for the agents

  • The Bot then guides the user through talking to each agent in turn - in De Bono terms this would be the White Hat (facts), Red Hat (gut feel), Yellow Hat (pros), Black Hat (cons), Green Hat (new ideas), Blue Hat (process)

  • Agents can flag responses that don't match the agent perspective, and users can flag key insights

  • The whole experience could be opened up to a large cohort to garner varied opinions and ideas in a very structured way

  • Bots and Agents can reflect a whole range of lenses, such as PESTLE, ASCOPE, PMESII and the 7 Questions.

  • Gets user to think about the broader implications of a project, especially ethical and moral implications

DesktopVR

halo_emotions_sm.png

Virtual Characters in Virtual Worlds

A bit more esoteric this one, but I'm a firm believer that we'll only get really "intelligent" and "human" bots if they have an embodied existence within a "real world". For me embodiment as an avatar in a proper, messy, populated mostly by humans, virtual world is actually a better bet than trying to build robot androids in the physical world as you can focus on the "brain" functions and forget about all that complex mechatronics! 

I've had a test bot, called Halo, in Second Life (and other worlds) on and off for nearly two decades now,  and with each iteration I'm exploring new facets of that Virtual Human diagram at the top of the page. 

  • A proper virtual world presents a virtual character with most of the same challenges as a physical robot in a physical world

  • The robotic avatar (robotar) has access to the same information and capabilities as a human - it's a level playing field

  • By immediate sight a user can't tell what's a human avatar and what's a robotar - so a level playing field for things like the Turing test (which we've passed in a virtual world)

  • The robotar needs to build spatial awareness, and spatially aware conversation - how do you implement the cocktail party effect for a robotar?

  • Ideally the robotar learns thing in/about the environment, rather than having them programmed - for instance Halo learnt to fear snakes as they reduce her health in SL.

DesktopVR

VirtualBarry.PNG

Virtual Personas

One of our biggest projects in this areas was a 3 year project on Virtual Humans for Dstl and the MOD. As well as looking at bots for training and coaching we also developed the concept of the Virtual Persona - a bot modelled on a real employee which would make their knowledge, experiences and insight available if they were not available (e.g. on leave or active duty) or even if retired. We were quite surprised at how successful the project was, and it was an ideal way of exploring some of the ethical issues which this sort of technology starts to raise.

I talk more about this project in my CogX talk, and the RAF included a story inspired by our work on Virtual Personas in it's Stories From the Future publication.

  • This is really about corporate knowledge management, not bots

  • Extensive use of knowledge graphs and semantic triples for information storage

  • Knowledge modelled on the biologically inspired semantic, episodic and procedural categories - all stored as triples

  • Background data mined from public sources, but with a personal filter

  • Personal data added by the bot "boot-strapping" from empty, based purely on a set of knowledge classes and structured conversations with the Subject

  • Demonstrations of adding more data direct from personal electronic sources - emails, phone, calendar, tweets

  • It seems that the "Virtual Barry" is now better known in Dstl than the real Barry!

  • When merged with a personal assistant/virtual life coach the creation of  your Virtual Persona could become a by-product of your day-to-day activity. See the "digital traces" audio at the bottom of the page!

DesktopVR

DigitalAfterlifeBook.jpg

Digital Immortality

Of course the next step from the Virtual Persona is what happens when the Subject dies? Can the employer still use them? Does the family and next-of-kin have a right of access? Can the Subject actually leave their Virtual Persona as something with personhood to "continue" their life after their death - not just to comfort and care for their family but also to continue to run a business or pursue a dream? Do we want an immortal Mark Zuckerburg, Elon Musk or Donald Trump?

I explain some of my thinking in this video on the Taylor and Francis web site.

  • If we crack the Virtual Persona then Digital Immortality really just becomes a hosting plan (my 2015 TEDxBrum presentation)

  • We need to differentiate between active and passive forms of Digital Afterlife/Digital Immortality

  • How will the courts view a Digital Immortal?

  • Our ad-hoc surveys show people are keener to have an Immortal version of themselves than they are of their loved-ones (!)

  • How would even moderate Virtual Personas affect the grieving and closure process?

  • Read my chapter on the technical issues of Digital Immortality (and the other chapters) in Maggi and Victoria's Digital Afterlife book.

Can I Help?

Here are some of the services which I can provide to help you make the best of Virtual Humans:

  • Just have a coffee chat to understand what all this is about, and how Virtual Humans will fit in to the Metaverse

  • Run a workshop to help you and your colleagues better understand the potential for virtual humans across your organisation

  • Provide an analysis of the different approaches to chatbot development so you can better understand the different affordances and capabilities

  • Help you consider how a reflexive bot could help with gathering diverse views on topics from across your organisation

  • Provide a synthetic social media wrap to your next exercise

  • Do a deep dive on the potential for virtual mentors and virtual personas in your organisation

  • Support academic research on virtual humans by providing the commercial viewpoint - and hopefully some bright ideas and insights

  • Act as your customer friend in developing and implementing a project, helping to interpret the tech talk of the developer, and ensuring that the developer keeps your ultimate goals (and constraints) in mind

More Information

For more information about Virtual Humans check out:

  • My gallery of past projects 

  • My research papers, mostly collected at ResearchGate

  • My talk at CogX on my work with the MOD on Virtual Personas

  • My interview on Virtual Humans on the 1202 Podcast (hosted by the "real" Barry!)

  • My blog post on Digital Afterlife on the Aura end-of-life management website

  • My virtualhumans.ai web site which is where I tend to blog about virtual humans, link to presentations I give, and of course buy the book!

  • Caprica - still one of the best movie visualisations of virtual worlds and virtual beings - especially the bit of audio below

  • All the other information on my Resources page

zoe_constructCaprica
00:00 / 00:41
VH Cover Image_sm.jpg

What the Bots say....

Wix now has the ability to get an AI chatbot to write text. So here's what it gave me when I asked it to give me a paragraph on virtual humans for the website.

At our software company, we've created virtual humans that can interact with users in real time. These simulations replicate human behavior and can be used for a variety of purposes, such as interactive customer service, sales, and training. While they have many advantages, such as being available 24/7 and reducing labor costs, there are also some challenges to consider. For example, developing and implementing them can be time-consuming. Overall, they’re an exciting innovation with tremendous potential.

bottom of page