Good Morning Karen. Cool or Scary?

Last month I spoke at a telecommunications industry event. The briefer before me showed a video by the Hypervoice Consortium, titled Introducing Human Technology: Communications 2025. It consists of a voiceover by a 2025-era Siri-like assistant, speaking to her owner, "Karen." The assistant describes what's happening with Karen's household. 15 seconds into the video, the assistant says:

The report is due today. I've cleared your schedule so you can focus. Any attempt to override me will be politely rebuffed.

I was already feeling uncomfortable with the scenario, but that is the point at which I really started to squirm. I'll leave it to you to watch the rest of the video and report how you feel about it.

My general conclusion was that I'm wary of putting so much trust in a platform that is likely to be targeted by intruders, such that they can manipulate so many aspects of a person's life. What do you think?

By the way, the briefer before me noted that every vision of the future appears to involve solving the "low on milk problem."

Comments

Ron Trunk said…
How does that movie line go again? Oh I remember: "I'm sorry, Dave. I'm afraid I can't do that."
4:49 PM
Unknown said…
I see my children taking advantage of that. I'll be the old geezer saying that contraption away from me.
7:21 PM
Anonymous said…
Everything crucial needs a manual override. That's like one of the fundamental principles of sci-fi paranoia.
1:01 PM
dre said…
Yes, in Sheridan and Verplank's Ten Levels of Automation of Decision and Action Selection -- http://images.slideplayer.com/2/730405/slides/slide_8.jpg -- we would typically want automation to be at best level 6 and at worst level 4. Karen was working at level 7 automation, which I agree is too much -- especially given the context of her situation.
1:12 PM
Anton Chuvakin said…
Well, we actually recently started a mini-research project on that very topic: http://blogs.gartner.com/anton-chuvakin/2015/07/10/on-evil-ais-and-evil-people/

In scope are AI/smart machine/robot exploitation...
5:42 PM
dre said…
Anton: wow that work very much correlates to my provided references!
11:35 PM
Unknown said…
This is interesting - but I think many things, even physical barriers like bi-folding gates, can be subject to attacks in this day and age.
8:53 AM
Anton Chuvakin said…
@dre Thanks --- we are hard at work on this [despite some people making fun of us for looking too far in the future]. I will blog some notes from our work fairly soon
6:08 PM
Anton Chuvakin said…
@anonymous Manual override, BTW, is a non-starter for A LOT of "AI"/smart machine uses -- trading, even driving [like you are watching a movie in your car and the driving robot say "yo..now you do it; it is too hard for me!" FAIL]
6:10 PM
Anonymous said…
Personal information online is targeted by hackers all of the time...but this is REALLY personal information...you're just waking up, the coffee is brewing in the kitchen, your mother 's ETA is 5 minutes, it seems to know where everybody is at any given time...including your children! This is a cyber-predator's dream.
That's information that just shouldn't be available, let alone 24/7.
If you can GUARANTEE that nobody else can access the data, I'm sure that there are more than a few governments and financial institutions that will pay dearly for that technology.
1:15 PM
Post a Comment

Popular posts from this blog

Zeek in Action Videos

Image
This is a quick note to point blog readers to my Zeek in Action YouTube video series for the Zeek network security monitoring project .  Each video addresses a topic that I think might be of interest to people trying to understand their network using Zeek and adjacent tools and approaches, like Suricata, Wireshark, and so on.  I am especially pleased with Video 6 on monitoring wireless networks . It took me several weeks to research material for this video. I had to buy new hardware and experiment with a Linux distro that I had not used before -- Parrot .  Please like and subscribe, and let me know if there is a topic you think might make a good video.
Read more

MITRE ATT&CK Tactics Are Not Tactics

Image
Just what are "tactics"? Introduction MITRE ATT&CK  is a great resource, but something about it has bothered me since I first heard about it several years ago. It's a minor point, but I wanted to document it in case it confuses anyone else. The MITRE ATT&CK Design and Philosophy document from March 2020 says the following: At a high-level, ATT&CK is a behavioral model that consists of the following core components: • Tactics, denoting short-term, tactical adversary goals during an attack; • Techniques, describing the means by which adversaries achieve tactical goals; • Sub-techniques, describing more specific means by which adversaries achieve tactical goals at a lower level than techniques; and • Documented adversary usage of techniques, their procedures, and other metadata. My concern is with MITRE's definition of "tactics" as "short-term, tactical adversary goals during an attack," which is oddly recursive. The key word in the tacti
Read more

New Book! The Best of TaoSecurity Blog, Volume 4

Image
  I've completed the TaoSecurity Blog book series . The new book is  The Best of TaoSecurity Blog, Volume 4: Beyond the Blog with Articles, Testimony, and Scholarship .  It's available now for Kindle , and I'm working on the print edition.  I'm running a 50% off promo on Volumes 1-3 on Kindle through midnight 20 April. Take advantage before the prices go back up. I described the new title thus: Go beyond TaoSecurity Blog with this new volume from author Richard Bejtlich. In the first three volumes of the series, Mr. Bejtlich selected and republished the very best entries from 18 years of writing and over 18 million blog views, along with commentaries and additional material.  In this title, Mr. Bejtlich collects material that has not been published elsewhere, including articles that are no longer available or are stored in assorted digital or physical archives. Volume 4 offers early white papers that Mr. Bejtlich wrote as a network defender, either for technical or pol
Read more