Okay. I hope everyone can see my screen.
Okay, well, good morning. I'm Arjan van Hessen.
I will tell a little bit more about myself in the next slide.
But well, just to inform you, I'm working at University of Twente, University of Utrecht and also at Telecats.
It's a software company in Enschede and I'm living in Utrecht.
And my work for the University of Utrecht is, well, bounded by CLARIAH.
And, well, that's so it's not scientific, but it's more, well, an organizational stuff.
But in turn, I'm working on speech technology and more or less also now on the understanding of speech.
But that will be something of this meeting.
Okay. Yeah, here is an interview four years ago, I believe, for people who are blind or
have difficulty in good looking. So they are used or they are helped with modern speech
and language technology. So there was an interview with me for their personal radio or well podcast
or how do you call it?
And I got the recording and I thought,
Well, let's see how speech recognition is doing.
Well, you can see, and I hope that you could also see the English translation who was generated
automatically based on this recognition result.
And so it's a nearly 100% correct recognition.
There were two smaller errors, Telecats, I mean, the company is Telecats, so it's
one company, but it wasn't in the vocabulary at that time. So it is recognized as Telen
& Kets. Okay. And it is the museum. And the museum is a special museum in Nijmegen, dedicated
for people who are blind or can't see very well. But for the rest, it is perfect. And
If you say, well, what is missing in this transcription?
It are the dots and the commas and eventually a question mark.
So besides of it, it was already a perfect recognition result.
This was done with the Calde recognizer,
available for all researchers at the University of Nijmegen.
So you can go there, register yourself, upload the file,
and you get the results.
However, there's a new engine.
We will talk today, talk about it today,
and that is outperforming the Kaldi recognizer.
Okay.
So if we are looking at the,
well, the dissemination of the human knowledge
of the last 3000 years, I mean, so well,
what has that to do with speech recognition?
well, a lot. So eventually, we started, of course, with talking
to each other. Later, it became text. So it was from generation
to generation possible to give your knowledge to the next
generations. But now we are in the speech mode and text mode
again. So you can say, well, we started with the Druids, for
example, who had a very oral history. And later in the
medieval time, or well, before that, to be honest, it was the
written context, information was written down in books. And in
the 14th century, it became pressed. So it was the, well,
the in textual form. And in the late, no, the early 20th
century. Recognition or recording devices were available. And the
first well, oral history, you may say recordings, you can see
here is an it is an America in I believe 1916 or something like
that. And so you can say from oral to written version to
Nowadays, the podcast, the videos, and all that stuff.
And we are back in our oral environment, you may say.
And for example, you can see here the National Initiative
to Sound and Vision in Hilversum.
It has a museum.
It has working offices.
And underground, so 30 meters deep,
it is the archive where they have more or less,
at this time, 800,000 hours of audiovisual material.
However, they don't know what, I mean, from part,
they know what it is about, but from huge parts,
they know it's a broadcast from 1937,
but what it is about, what is said is unknown.
At the same time, we see at the universities
that there is an increasing use of the videos, especially from COVID, but also before.
We have more and more teachers who are giving a video presentation.
If it is a video for educational stuff, it has to be stored in the archives for seven
years, but we don't know what it is about.
I mean, it is an interview or it's a lecture from Professor
Janssen, and he is talking about biology,
but that's what we know.
So we are missing, we do have some metadata,
but it is missing partly.
But the transcriptions normally are not available.
And so that means that if we want
to know what the lectures are about,
we need to listen and also to look at it and that can be nice but I mean with 150 000 hours
it's not doable to give a good overview of what is available. Okay it's not the new that we are
coming in and well you can say in decade or in time from artificial intelligence
it is quite popular in the newspapers and we see each half year there are new developments
and the latest was a chatGPT and we will talk about that later but we may say that artificial
intelligence is leaving the laboratories and becoming part of our well history and
that means that we have a question what is artificial intelligence well you may say
something magic that gives you the opportunity to to use the software for all kinds of smart things
But well, if you were looking in the past, you say that in between the 50s, when it was started, the term was was notified and it was more or less rule based.
So look for chess players. And that was considered as artificial intelligence at that time.
And in the 80s, it was machine learning. It became popular in the laboratories.
And nowadays we have machine learning with deep neural networks and the deep neural networks are a kind of a copy from the way we are well thinking.
So you may say it is coming closer to the way people think.
And if you can, if you're successful in making software that is copying at least part of our behavior or part of our, yeah,
you may call it artificial intelligence. But you see it is a changing also vision from the humanity about artificial intelligence.
And if you say, well, what is AI?
Of course, we have the famous games.
It was first with chess in the 90s, I believe, Deep Blue.
And in 16, so for seven years ago,
it was Google who started with the game Go.
And they developed DeepMind.
And to train DeepMind, they used all the available games
that were recorded and things like that.
And they tried to train the computer
with the existing games.
OK, it became very good.
However, then there was a clever one at Google that said,
well, if we have two versions of DeepMind,
we can make them play against each other.
And then the winner will win.
And that game is considered as a good starting point for the next training and so on and so on.
So they started for three weeks, I believe, and it was continuous playing, updating the results and then playing again and etc.
And in the end, I mean, the DeepMind is so good that it is absolutely the world class winner.
However, I'm not talking about artificial intelligence as such, but it is more the language-dependent
artificial intelligence.
In 2011, I believe, it was Watson, or IBM was Watson, who started to join the game.
It's kind of a quiz instead of the European version where you have a question and you
you have to give the answer. The answer is given and you have to invent the question.
But that's it. And they trained the computer and then there was an official presentation.
In Ken was the world champion of that year. Brett was the guy who won the most money in
the year before and they played both against Watson. And in the end Watson absolutely overwhelmed
The other two, I mean, I believe that Ken had 12,000,
Brett had $14,000, and Watson has 40 or $50,000.
So it was absolutely an easy game.
How was the system developed?
They used, of course, all American,
it was only in American, text available.
So the Wikipedia, books, newspaper articles,
internet stuff, et cetera, et cetera,
that was trained, given to the computer.
The computer learned to argue about the results.
And then in the end, it beat it, and that's game.
OK.
And then in 2017, so five years, six years
after the previous one, we got the transformer models.
And if you ask me exactly what a transformer model is,
I'm not 100% sure.
I'm studying it at the moment, but it's also for me quite new.
But here you have some nice views.
You will get the presentation afterward.
And it is a neural network, of course, that learns context
and does meaning by tracking relationship
in sequential data like the words in the sentence.
Transformer models apply an evolving set
of mathematical techniques called attention or self
attention to detect subtle ways, even distant data elements
in the sphere of influence and dependent on each other.
OK, well, that's a more or less nice definition
of the transformer model.
And it was started in 2017 by a paper of Google.
And it became absolutely new.
And then Stanford researchers, they
called that foundation models instead of transformer models.
So well, it's a different terminology
for the same technique.
But it turns out that these transformer models are very,
very strong in the current, well, AI revolution.
OK, transformer are translating text.
So what can you do?
You can give it a Dutch text and say,
will give me the English version.
And speech, also possible.
So for people who are hearing impaired,
it can be very useful.
They can detect trends and anomalies
to prevent fraud and things like that.
So it is important for health care and banking and that stuff.
And here, it is a nice overview.
You have the data, the foundation,
or the transformer models.
And here you have a couple of, well, resulting tools.
And I suppose that the most of you will know this video,
But, well, we can see.
OK, well, it isn't I mean, you can find it on the Internet.
It is an astonishing conversation
where the computer is calling a hairdresser or hair shopper
and make an appointment for, well, her boss
who's not involved at the moment.
And especially this human-like and this humbling
and things like that is, well, it
makes it very natural sounding.
And it is, yeah.
OK.
Then we, more or less at the same time, 2017 or 18,
open AI started and well as the name is promising they develop AI and it will be open well partly
at least and is it an evolution or a revolution well that's a discussion but
they started well they make a very impact full entrance in December 2022 so last year
is the introduction of chatGPT and they well I know that most of you will know chatGPT.
And yesterday I asked the computer to explain quantum computing in simple terms so that
was just a question to me and then you get a nice overview in which well more or less
quantum computing is explained, at least to me.
And even our last week, I believe,
they updated their language model,
you may say, to GPT-4.
So chatGPT-3 and 3 and 1 half was stopping in 2021
because they used the data from before 2021.
And that means that all current activities, the war in Ukraine
or things like that, you couldn't ask at chatGPT.
And now they update it to chat version 4,
GPT4 version 4.
And if you have a paid account, then you
can already access it.
And I suppose it will be becoming available
for the free engines as well.
But well, it is, again, better.
And you can give images to it.
They can do a kind of humor.
And it is absolutely amazing what is possible with chatGPT.
Then in 2022, in September, end of September,
they silently introduced Whisper.
And that is a speech recognition engine.
More or less based also on the Transformer models.
So what is Whisper? It is an automatic speech recognition
engine trained on nearly 700,000 hours of multilingual and
multitask data.
And to give you an idea,
700,000 hours is more than you and I will use in
our life. So it's an enormous amount of data.
And well, they used 60% I believe was English and 40% were other languages.
And they showed that use of such a large and diverse data sets leads to an improved robustness
of accents, background noise and also technical languages.
Moreover, it enables transcription in multiple languages. So it is one model
And you can give it in Dutch version or a Chinese version or an Italian version and
it will give you the translation in or the transcription in that language.
And also you can translate it to English.
So if you hear an interesting Chinese conversation and you don't know what it is about, you can
give it to whisper, make the transcription and then the translation in English and you
can see what it is about.
And what is very nice of OpenIA is that this time it was a really open sourcing model.
So they developed seven models, I believe, or eight or nine models, and you can download
them from their site and use them.
OK, if we look to speech recognition in the past two decades, I mean longer, in the 70s
started slowly, slowly, and it was more or less based on the Fourier transform. Then in 2000,
nearly, yeah, 2095, 96, they were the first initiatives with the HMM. And then in 2010,
this is the paper of Microsoft at the InterSpeech conference in Florence.
they started with the deep neural networks and well in 2019 I believe it was the first time
that for correct recorded American English conversations it was on the level of the human
accuracy outperformed it a little bit but let's say it was more or less on the human accuracy and
And that's quite recent.
And it is, well, something I made myself.
But I believe that with the coming of the transformer
models, it will increase a little bit.
I mean, yeah, more than 100% is not possible.
But we will see that in the coming years now
and in the coming years, it will have reached
this human accuracy for other languages than American English and it will also increase.
So in the end it will outperform as humans in correct recognition and said that you have
to remember that it is for correct good recorded audio.
If you have a conversation where people talk to each other or there is a train passing
by or other background noises, it will be different. However, it is performing very
good. But if it is outperforming humans, it's not sure at the moment. But for nice, cool
recorded audio, it will absolutely beat us in the coming years.
OK, however, speech recognition is not perfect. But even if it is perfect, I mean, what is
going wrong? Well, it is still the case that we do use recorded words, our vocabulary,
and in the eticali, so the current old version of the ASR, we can recognize 260,000 different
words, but there are always words that are not in that list, so they cannot be recognized.
An example was the marsupilami. With the current version of Whisper, I do believe that marsupilami
will be recognized, but I have to test it. And then there is in the use of the language
model, you can say something that is between Zwelle and Zwolle and you mean Zwelle. I'm
going by train from Utrecht to Zwelle and we saw that Zwelle is not that popular and Zwolle is
very popular so the language model is replacing the recognized Zwelle by Zwolle because it's more
logical that people will say Zwolle and more or less the train station in Zwelle doesn't exist
anymore. So it makes sense but it is not close to the recognition and it will be a discussion
if you want that or not. But okay. And what we do see is that people do say other words.
They want to say, well, give an example as well. And if you listen to it, you absolutely hear
and it is just a mistake from people. Yeah. Okay. That can be the case. And with our human brains,
we can say no, but he means well because Zola has nothing to do in this conversation.
But that is something else that is the interpretation of the recognition and not the recognition itself.
Okay and the speech AI is used for well this is an example of some five years ago
with the old recognition engines and it is an overview of the, well, the, oh, yeah, it
is an overview of, well, what we believe is for researchers in a normal way. You do have
an interview. It can be an analog interview or in nowadays you will have an digital interview.
The analog need to be digitized and you need a storage.
OK, we have stored the data and then we need the transcription
and you can do it by yourself, listen to it and type it out.
Or you can use at least initially automatic speech
recognition and then you get the timed text
and the only thing you have to do
is the checking of if it was recognized well
and then make some errors and replace them
things like that and this of course was in my field always a popular item because well
we believe really that speech recognition will help you however with an error rate of one out of
10 words there's an error it sometimes was too much work to check and improve the
recognition results. Instead of doing it, just transcribe it yourself.
However, with the current version of Whisper, I absolutely believe that it is very useful to first
run the speech organizer and then check the results because it absolutely improved your
velocity and it speeds up so this is a little bit an old stuff and yeah okay and then you have of
course to add your metadata and then you have searchable transcribed audio visual document
and you can say well give me the interviews where they talk about hunger and then you get
all the interviews where hunger is pronounced.
OK, this is more or less 10 years ago.
I'm a little bit less gray.
And it was a question of the Dutch court.
And they say, well, it is at the moment
it's not allowed to do recordings in the courtroom.
However, we are interested to see if it is working fine
and to see what are the possibilities.
So they asked us and we developed it.
And it was quite a success.
There's a video.
If you look for Rechtspraakherkenning on YouTube,
you will get this video.
And it will explain a little bit, well, the performance
and how it was working.
A couple of years later, so five years ago,
the field came with the same question
and say, well, our research, our researchers,
our people are recording more and more interviews,
but once they record it, they need to give it
to the courtroom, visit transcription.
So can speech recognition help us
in speeding up this process?
And yeah.
Oh, no.
And hey.
OK, OK, so we did the recordings, we simulated and then we start a test where we compared
the classic way. That's the blue line with the new one and that's the red one. And what
you see here is that I mean, if you have an interview with someone and you need to make
some notes. It takes time and sometimes, well, you need some pauses and here, here, here
and here and here. You can see that the time is increased, but there is not, they are not
speaking. So that means they need some time to make their notes. And then they say, OK,
on. However this is decreasing the quality of the interview because once someone starts talking
you want them to continue talking and you don't want to interfere with well stop a moment and I
need some notes. So in the end you see that at the same time there are a lot of more words so 30 to
50% more words were spoken so we have more material in the same time and this was quite
convincing the field in using the speech recognition. We did a test for the NEOD,
the witness story, getuigenverhalen.nl. They have 600,000 hours of oral history about World War II
and well you want I mean no one is going to listen to 600,000 hours of interviews
but you want to interviews that talking about well some particular topics you
are interested in so you can search in the spoken content and we did the
project for the foreign ministry ministry foreign affairs and it was in
Croatia and Bosnia, so Croatian memories and Bosnian memories, and we recorded 700 interviews.
However, at that time, the speech recognition was not working for those languages.
And so this was hand transcribed, but the translation in English was done automatically.
Clarin and the national or the European infrastructure for language and speech technology
supported this and they said well can you start an OH portal and it's now a transcription portal
at the University of München and you do have an account you can go to there and upload your files
select the language and download the results.
This will probably in the coming months replaced by Whisper,
but at the moment it is the old version.
It is the Google version.
It is here the Dutch version.
Well, that are the languages supported at the moment.
And okay, here are some shots from projects we did.
This was the forced alignment.
So we got the text from the second room, so the Tweede Kamer.
They are forced to give a correct transcription.
They gave it to us.
And what we did was the forced alignment.
So the subtitles are automatically generated.
And then the Flemish government was quite enthusiastic
about our Dutch effort.
And they asked us to do the same for them.
We did it and this was the result was a collaboration with the Flemish universities and Dutch universities and what they also built was a speaker recognition engine.
So we know exactly who is speaking at this moment.
And here are the partners.
Here are the results.
However, it is a verbatim transcription, so it is 100% more or less 100% correct transcription.
And it turned out that it's too much for them.
So they want more summary.
And that's not possible at the moment.
And here you see, again, the 150,000 hours.
So that will be more at the moment, I suppose.
But a couple of years ago, it was 150,000 hours of surf.
And surf is also experimenting together with us
to see if they can recognize all their material.
It is more or less 50% English, 50% Dutch.
OK, then a project from the Utrecht University.
Can you use it with the patients for the care to report?
So you're going to your GP, and you have some questions.
And normally, they are typing a lot.
Can you replace the typing by speech recognition?
And well, you have more or less an overview.
And it turned out that it works well.
And they are starting now some real life showcases
where they will show that the recognition will
help the GP in their daily work.
And of course, at University of Twente,
we are focusing on robots interaction
between children and elderly people with robots.
And we are using the speech recognition
to understand what they are saying.
And at the same time, we're also developing some software
to see how they are saying it. So the emotion in the speech is as important as what they
are saying. So how and what.
So then we have people with difficulties. I mean, the brain doesn't always cooperate.
Here I will show you an example of someone who has Parkinson. I'm not sure if you can
hear it.
So that's I mean you can understand it,
but you have to listen carefully and.
Even the modern speech engines not always
recognize this kind of people as well
as well as we wanted.
Yeah, we couldn't hear the no, no, no, no, no, no.
Well, you will get the presentation and you can listen to it yourself.
Yeah, I'm I forgot to organize the audio as well.
And this was a project we did two years ago.
So in the COVID period.
And well, I can show you or see if you can hear it.
Well, you can see the difference between him and also his helper.
I mean, if he speaks, the recognition is easy,
but he's impossible to recognize.
And what we did, we developed a special
speech recognition engine for him
to help him using the engine for work,
for school, for traveling,
and to tell something about himself.
It worked, but I mean, it can be much, much better.
And this will be, well, I will skip this one, but this is in English Parliament and you
hear so much noise in the background that it is okay, you can hear it, but that is very
difficult even for an modern engine.
Okay, as said before, emotion is important.
We know we are well at the level more or less that with good
recorded audio we are at the level of the human recognition.
However, we need some emotion inside and the question is, can
we detect the emotion in the conversation? Because if you
ask me, do you like football? I can say yes, and then that isn't
convincing. Yes, or I can say yeah.
And it is a yes, but I mean no.
So can you use that kind of emotion detection inside modern conversations?
And then it's the question, which emotion? I mean, we have the big five, sadness, anger, fear, joy and neutral.
I mean, they are more or less human.
And that means that all the people in the world are using, are having these five emotions.
However, sarcasm or other more subtle emotions are cultural dependent.
And that means that it depends on the speaker and also on the listener how the emotion is preserved.
So that will be a difficult question, but well, it is worthwhile working on it.
Okay, so that means that brings us more or less at the end of this talk to the next step.
And that is what do you mean from recognition to understanding.
So can you understand what is said?
And we in collaboration with Nijmegen University, Twente University and two companies, we had
a question from the Dutch police force and they had the database with 45 hours of verbatim
transcription. Well, they had the audio we needed to give them the transcription,
the part of speech tagging and also the emotional edition. And one of the companies was Pandora
in Amersfoort and they made a nice film that is in English and I tried to figure out what was said
And so I'm using the modern engine to do the recognition.
Let's see if it is working.
Yeah, that's the wrong version, so you will have it in the, if I send it to you, you will
get to the English and also the eventually Dutch transcription below it.
And, but the recognition is more or less a hundred percent.
And that's really, I mean, of course it's a good recording, but
still so good is astonishing.
Okay.
If you look to the future of artificial intelligence lead speech technology.
And we see the next step or the step in the coming years will be from the recognition what was said to the understanding what was meant.
So that will be an absolutely important part of our research.
And what is someone's emotional state? And so how can you deal with that person?
person. And of course we have to figure out how to use the speech recognition for the
smaller languages and even Dutch is a small language. I mean, given the Chinese and the
Indian and the American English population, we are doing well. But I mean, Friesian or
some heavy dialects or languages that are spoken Icelandic only by 300,000 people. Can
do the recognition for that language as well. And we need to speed up the velocity so you can use
it in all kinds of real time situations. And what the kind of dream that you are talking
with a Chinese engineer and he's speaking in Chinese and you're speaking in Dutch,
his Chinese will be translated automatically to subtitles in Dutch and my Dutch will be
be automatically translated in Chinese.
So, I mean, it is more or less possible,
but we need to speed it up a little bit.
So to have that conversations in the coming years.
And okay, this was more or less the end of,
well, a smaller version of my presentation
about AI and speech technology.
And now we're going back to Whisper.
And Whisper, as said before, is an open source,
so you can download the models, at least at the moment.
It works more or less, at least for American English,
on the human level, so it outperforms it a little bit
or not, well, it depends.
And you can ask yourself, why using Whisper?
Well, here are some statements, but it is,
I mean, the basic answer is it is working absolutely
gorgeous. Yeah, it's September, so it's five, six months old. And
here are some slides where it's more or less explained. However,
you can look it up yourself at the GitHub repository. And they
say that it more or less outperforms the most speed
recognition engines available.
However, if you have some dedicated,
very specific conversations,
then some other engines are doing better.
But overall, Whisper is the winner at this moment.
Okay, installing Whisper, is it difficult?
No.
Well, you need to install first Python on your computer
And it is version 3.8 till 3.10.
I started first with 3.11.
And then there were some problems.
So I went back to 3.99.
And now it works.
You need to install PyChart.
And there's a lot of information on the internet
how to do it, or you may have it already.
You have to install FFmpeg Python.
And once you have installed these three basic packages,
you need to install Git from GitHub.
So the Whisper Git, I mean, and that's all.
It takes a couple of minutes and then it's installed
and you have your engine at your computer.
Using Whisper, there's a good, well,
helping facility from the parameters
that you can add to it.
However, the most important is the model.
And it is a fast computer and a GPU.
It is 32 times faster than real time.
So one minute will be in two seconds done.
This is the base version.
It is slower.
And this large model, well, it is a relatively speed of one.
But I doubt it.
It is slower, I believe.
But I mean, these are the models that you can load.
And of course, the large model is a 1.5 gigabyte.
And that's a lot.
And the tiny model is only 40 megabytes.
So normally, I'm using the medium version.
And because I do have a lot of recordings in English,
but also in Dutch, in Italian, and other languages,
I'm not using medium.en, but medium as such.
five gigabyte and well, I mean, you need that on your computer and it is twice the velocity
of the real time. So one hour takes you half an hour, a little bit more, but also depending
on your computer. But I mean, you just give the model at your recognition and you say,
well, I want to do it in this time is medium only English because they are native English
want to do it with English only, you type in medium.en.
And if it's not available, it will be downloaded.
It takes some minutes, and then you
have the medium.en model at your computer as well.
And you can do what you want.
Then there are some tools.
I'm particular fond of WhisperX, because the results of Whisper
are they taking frames of 30 seconds.
and giving the recognition results.
The 30 seconds sometimes for subtitles
or for all kinds of other research is too low.
And this WhisperX, you get a word level
more or less accurate recognition.
So they do the recognition
and then do a kind of forced alignment based on it.
And you get a subtitle with one word timeframe.
And it's very useful.
For the Macintosh, of course, you can use it with the Python
on your computer, but there's also Mac Whisper.
I believe it's Dutch guy who developed it,
but it is based on the C version of the Whisper engine.
And the free version can use the small and the tiny models.
But if you want medium or large, you
have to pay one time 15 euro.
So it's not that much.
And well, here's the interview.
You have just a screen.
You download the video or the audio
you want to be recognized.
And it starts to record the recognized thing.
And here you see the results.
And there are some small things like change some words that,
for example, Janssen, that can be written
on very different ways.
And you say, no, this is Janssen.
It's two S's.
And then, well, you can do a search
and replace on the results.
but you will see if you use it, it's easy to see.
Okay.
I mean, that was the conclusion.
So Whisper, Python, PyTorch and FFmpeg,
the basic engines you need to install.
And once you have done that, pip install,
git hcps, well, you can see it there.
And you have the best engine at your computer.
And again, at the computer I'm giving this presentation,
I do have a graphical card, NVIDIA.
So that means I do have a GPU,
so I can use the GPU version of the Whisper.
If you don't have it, it will take much more time
because the relatively speed is based on a fast GPU.
If you don't have a GPU, yeah, but it takes you more time,
but you can do it in the evening
and the next morning it will be ready.
So, yeah.
Okay, conclusion.
Automatic speech recognition is working well at this moment.
I mean, it's absolutely astonishing how good it is.
We do need, and then, I mean,
the community of speech technologies,
we do need to give a little bit more attention
to people who are less well-represented,
people with a heavy dialect or people coming from outside
and trying to speak Dutch, people with diseases.
So there is of course something to do,
but I do believe that in the coming years,
these will be, well, we probably will solve this.
And that's the next step,
but that is more the research topic.
I mean, we need to go from the recording
to the understanding.
We know, we do need to know what the intention
someone was when we did the recognition and not only a verbatim transcription.
This is the end and I have the question, what questions do you have for me?
Before you start this, there will be, if you like it, a follow up.
Utrecht University will organize that. Rijnike will send you mails. But I can imagine that
initially the first steps are always more difficult than you want them. So if there are five or more
people who want to continue trying to do the recognition with their results and on their own
computer. We will organize some classes, I mean meetings where I will be there eventually another
technical involved people trying to help you to install the Python and the stuff.
And then also help you in the recognition and to show you which model to use when things like that.
But it depends heavily on you. I mean if you are technical quite well involved you can do it yourself.
If you want us to help you, let us know and we will organize that.
And now it's open for questions.
And if there are no questions, that's fine for me as well.
But yeah, Daphne.
Hello.
Wait, let me turn my camera on so you can see me.
So, I'm not a researcher, but I'm actually a privacy officer at the university.
So, my interest in this is, well, privacy related.
And I was wondering how you feel or how you want to deal with the privacy issues that come up with this.
Because obviously, if you transcribe interviews, then it's more easily accessible to lots of people.
Absolutely so, but first we have to go one step back.
I mean, the current version, well, more or less outdated,
but still working version is that we developed a server
at the Nijmegen University.
You go there, you make your login and your password
and you need to show that you are not in commercial company
but working at university of that kind of organizations.
And you upload a file, you select,
you push the button, start.
And then after a couple of minutes,
or depending on the length of your recording,
you get the results.
However, if it is quite a sensitive recording,
it means that it's going to another company
and that's not doing any harm.
But still, given the GDPR, it is a little bit tricky
If that is allowed, yes or no.
If you have very sensitive material,
you can make a contract and then it is not done
via the internet, but you sent a USB stick to them
and they will handle it with all the privacy
related items involved.
However, with the current version of Whisper,
I mean, you can do it on your own computer.
So that means that you can listen to it
and make an handwritten transcription
or you can give it to Whisper on your own computer.
So there's no, I mean, you can do it without any internet
and you get the results.
And okay, so there is no difference
between an handmade transcription
or a transcription done on your own computer with Whisper.
What you do after that, and then of course,
it's a sensible point that you mentioned.
I mean, once you have the transcription
and you place that somewhere on a website
or things like that, yeah.
but there's no difference between the use of whisper
on your own computer or doing it,
the transcription by hand yourself.
So yeah, the privacy is still very important,
but whisper is not going to change that, so to say.
Is that more or less what you wanted to hear?
Yeah, very much. Yeah, thank you.
Okay.
Hi, I was wondering, you mentioned the difference between specific languages, like Dutch, and
that more attention needs to go towards that.
But what would you say are the biggest differences in the models currently used, for example,
in sound and vision, which has ASR?
Do you see patterns in mistakes that it makes, or in what kind of ways does that differ from
whisper, for example?
Ah, well, Whisper is, I mean, the error rate on sound and vision that is using the Kaldi
recognizer is, I mean, is more depending on the quality of the speech and things like
that.
But for Dutch, it is between 15 and 12 percentage.
So it means that you have 85% of the words are recognized well, and for the other 15%,
you absolutely need to check them.
is at the human level so that is between three and five percentage. So it is only one-third
of the errors made at with the Calde recognizer are made by this part. Moreover, it's if you
say well I'm Arjan and I'm living in Utrecht, Arjan and also Utrecht is written as a capital
and with the Calde recognizer all the words are lower capital and there are no question marks
There are no commas, there are no dots, and all those reading stuff is not available.
With Whisper, it is available.
And I will show you.
But that's, yeah, I'm still in.
It's my web. Can you see what I'm doing?
Yeah, OK.
And here is an example. It is in Dutch, but well.
And here you do see it is,
either end starts with a capital I and then there's a comma
and another comma and it's a dot on the end.
The Taal Uni is written with capitals T and U and et cetera, et cetera.
So this is a very nice conversation.
Well, and even Caldi was doing it quite well.
But the recognition in Caldi is only with the lower capitals.
And this one is, it knows how to write the Taal Uni, it knows how to write
the Dutch and things like that.
So, I mean, this is more or less, and you don't need to repair this.
So, coming back to your question on sound and vision, even sound and vision will move to whisper as soon as possible.
I mean, Roeland and I, we are working both at the University of Twente and we know this and they start as quick as possible.
I mean, we need a couple of weeks, months to make it.
we need to to build it and yeah it is style unique that's right it's written
different but anyway and but we mean I mean I mean it's now March so I believe
that for the summer this per will be the default recognizer for Dutch that we
offer so that means that you can do it at sound to vision you can redo all the
stuff that you did in the past or you can say well only the new material will
done with Whisper, but that's something Sound and Vision has to deal with.
Is this more or less answering your question? Yeah, it does. Thank you. Okay.
Other people.
No questions at all.
OK.
Well, or it was completely not understandable, or it was 100% understandable.
Reinike, may I give the word back to you?
Yes, you may.
As Ariane already indicated, we will schedule to follow-up workshops with sufficient interest,
so from four people on.
If you're interested, you can send an email to CDH at uu.nl, I already also put it in
the chat.
And we will try our best to schedule two workshops that work for everyone on location in the
Utrecht City Center.