34 stories
·
0 followers

Can you help me gather open speech data?

1 Share

Screen Shot 2017-06-12 at 3.18.46 PM

Photo by The Alien Experience

I miss having a dog, and I’d love to have a robot substitute! My friend Lukas built a $100 Raspberry Pi robot using TensorFlow to wander the house and recognize objects, and with the person detection model it can even follow me around. I want to be able to talk to my robot though, and at least have it understand simple words. To do that, I need to write a simple speech recognition example for TensorFlow.

As I looked into it, one of the biggest barriers was the lack of suitable open data sets. I need something with thousands of labelled utterances of a small set of words, from a lot of different speakers. TIDIGITS is a pretty good start, but it’s a bit small, a bit too clean, and more importantly you have to pay to download it, so it’s not great for an open source tutorial.  I like https://github.com/Jakobovski/free-spoken-digit-dataset, but it’s still small and only includes digits. LibriSpeech is large enough, but isn’t broken down into individual words, just sentences.

To solve this, I need your help! I’ve put together a website at https://open-speech-commands.appspot.com/ that asks you to speak about 100 words into the microphone, records the results, and then lets you submit the clips. I’m then hoping to release an open source data set out of these contributions, along with a TensorFlow example of a simple spoken word recognizer. The website itself is a little Flask app running on GCE, and the source code is up on github. I know it doesn’t work on iOS unfortunately, but it should work on Android devices, and any desktop machine with a microphone.

Screen Shot 2017-06-12 at 3.24.10 PM

I’m hoping to get as large a variety of accents and devices as possible, since that will help the recognizer work for as many people as possible, so please do take five minutes to record your contributions if you get a chance, and share with anyone else who might be able to help!








Read the whole story
ssorc
163 days ago
reply
Melbourne, Australia
Share this story
Delete

Thoughts on the wearables studies (including The Stanford Wearables study)

1 Share

As might be obvious these days, publishing a study on wearables is the fashionable thing to do.  There seems to be a new major (or at least noticed in the mainstream media) wearable study published every month.  Sometimes more often.  And that ignores the likely hundreds of smaller and never picked up wearable studies that probably occur each year in all manner of settings.

And in general – these studies are doing good work.  They’re aiming to validate and hold accountable manufacturer claims.  That’s always a good thing – and something I aim to do here as well.  Many times these studies will focus on a specific claim – such as heart rate accuracy, or step accuracy.  As those are generally somewhat easy to validate externally through a variety of means.  In the case of steps it can be as simple as manually counting the steps taken, and in the case of heart rate it may be medical grade systems to cross-reference to.

All of which are 100% valid ways to corroborate data from wearables.

Except there’s one itty bitty problem I’m seeing more and more often: They’re often doing it wrong.

(Note: This isn’t the first time I’ve taken a supposedly scientific study to task, you’ll see my previous rendition here.  Ironically, both entities screwed up in the same way.)

Why reading the manual matters

Most studies I’ve seen usually tackle 5-7 different devices to test.  Almost always one of these devices is an Apple Watch, because that has mainstream media appeal.  Additionally, you’ll usually find a Fitbit sensor in there too – because of both mainstream media interest as well as being the most popular activity tracker ever.  After that you’ll find a smattering of random devices, usually a blend of Mio, sometimes Garmin, sometimes Polar, and then sometimes totally random things like older Jawbone or others.

From there, devices are tested against either medical grade systems, or against consumer grade systems.  The Polar H7 strap is often used.  Which while not quite as ideal as medical grade systems, is generally a good option assuming you know the obvious signs where a strap is having issues (remember – that’s still a very common thing).

But that’s not what’s concerning me lately.

(Actually, before we go forward – a brief aside: My goal isn’t to pick on this Stanford study per se, but that’s probably what’s going to happen.  I actually *agree* with what they’re attempting to say in the end.  But I don’t agree with how they got there.  And as you’ll see, that’s really important because it significantly alters the results.  Second, they’ve done a superb job of publishing their exact protocol and much of the data from it.  Something that the vast majority of studies don’t do.  So at least they’re thorough in that regard.  Also, I like their step procedure for how they are testing it at different intensities.  One of the better designed tests I’ve seen.

Next, do NOT mistake what I’m about to dive into as saying all optical HR sensors are correct.  In fact, far from it.  The vast majority are complete and utter junk for workouts.  But again, that’s not what we’re talking about here.  This is far more simplistic.  Ok, my aside is now complete.)

Instead, what’s concerning me lately is best seen in this photo from the Stanford study published last week (full text available here):

image

(Photo Credit: Paul Sakuma/Stanford)

As you can see, the participant is wearing four devices concurrently.  This is further confirmed within their protocol documents:

“Participants were wearing up to four devices simultaneously and underwent continuous 12-lead electrocardiographic (ECG) monitoring and continuous clinical grade indirect calorimetry (expired gas analysis) using FDA approved equipment (Quark CPET, COSMED, Rome, Italy).”

First and foremost – the Mio Alpha 2 on the left wrist (lower one) is very clearly atop the wrist bone.  Which is quite frankly the single biggest error you can make in wearing an optical HR sensor.  It’s the worst spot to wear it.  Don’t worry though, this is covered within the Mio Alpha 2 manual (page 6):

image

But let’s pretend that’s just a one-off out of the 60 participants when the camera came by.  It happens.

The bigger issue here is them wearing two optical HR sensor devices per wrist (which they did on all participants).  Doing so affects other optical HR sensors on that wrist.  This is very well known and easily demonstrated, especially if one of the watches/bands is worn tightly.  In fact, every single optical HR sensor company out there knows this, and is a key reason why none of them do dual-wrist testing anymore.  It’s also why I stopped doing any dual wrist testing about 3 years ago for watches.  One watch, one wrist.  Period.

If you want a fun ‘try at home’ experiment, go ahead and put on one watch with an optical HR sensor.  Now pick a nice steady-state activity (ideally a treadmill, perhaps a stationary bike), and then put another watch on that same wrist and place it nice and snug (as you would with an optical HR sensor).  You’ll likely start to see fluctuations in accuracy.  Especially with a sample size of 60 people (or 120 wrists).

I know it makes for a fun picture that the media will eat up – but seriously – it really does impact things. Similarly, see in the above picture how the Apple Watch is touching the Fitbit Blaze?  That’s also likely impacting steps.

Another fun at-home test you can do is wear two watches side by side touching, just enough so while running on a treadmill they tap together.  This can increase step counts as a false-positive.

Which isn’t making excuses for these watches.  But it’s the simple reality that users don’t wear two optical HR sensor watches in the real world.  But honestly, that’s probably the least of the issues with this study (which is saying a lot, because at this point alone I’d have thrown out the data).

In case you’re wondering why this did this – here’s what they said:

“1) We wanted to check for any positional effects on the watch performance –i.e. does right vs left wrist matter? Does higher or lower on the wrist matter? So watch arm & position was randomized (see supplementary tables in manuscript).

2) We are more confident in results obtained from same workout rather than separate workouts for each device.

3) Purely practical — having the same subject perform the protocol 4 – 7 times is infeasible. It would have been challenging to get compliance in a sufficient number of subjects.”

I get that in order to reduce time invested, you want to take multiple samples at the same time.  In fact, I do it on almost all my workouts.  Except, I don’t put two optical HR watches per wrist.  It simply invalidates the data.  No amount of randomizing bad data makes it better.  It’s still bad data.

And when we’re talking about a few percent mattering – even if 1 out of 5 people has issues, that’s a 20% invalidate data rate – a massive difference.  There’s no two ways about it.

Let’s Talk Fake Data

Another trend I see over and over again is using one-minute averages in studies.  I don’t know where along the way someone told academia that one-minute sport averages are acceptable – but it’s become all the rage these days.  These studies go to extreme degrees on double and triple regression on these data points, yet fail to have accurate data to perform that regression on.

Just check out the last half of how this data was processed:

image

Except one itty-bitty problem: They didn’t use the data from the device and app.

Instead, they used the one-minute averages as reported by various methods (most of which aren’t actually the official methods).  For example, here’s how they accessed the Mio Alpha 2:

“The raw data from the Mio device is not accessible. However, static images of the heart rate over the duration of the activity are stored in the Mio phone app. The WebPlotDigitizer tool was utilized to trace over the heart rate images and to discretize the data to the minute level.”

Translation: They took a JPG image screenshot and tried to trace the image to determine the data points.

Pro Tip: They could have simply connected the Mio Alpha 2 to any phone app or any other watch device to gather second by second HR data via the rebroadcasting feature. After all, that’s kinda the main selling point of the Mio Alpha 2.  Actually, it’s almost the only selling point these days.

Or here’s how they did the Microsoft Band:

“The mitmproxy software tool [15] was utilized to extract data from the Microsoft Band, following the technique outlined by J. Huang [16]. Data packets transmitted by the Microsoft phone app were re-routed to an external server for aggregation and analysis. Sampling granularity varied by activity J. Pers. Med. 2017, 7, 3 5 of 12 and subject. In cases where multiple data samples were collected each minute, the last data sample for the minute was utilized in the analysis.”

So, let me help you decode this: They didn’t use the actual data recorded in the app, but rather, they picked data at one-minute intervals in hopes that it’d represent what occurred in the previous minute.  Yes, the Microsoft app sucks for data collection – I agree, but this isn’t an acceptable way to do deal with such suckiness.  You don’t throw away good data.

Or, here’s how they did the Apple Watch data:

“All data from the Apple Watch was sent to the Apple Health app on the iPhone, and exported from Apple Health in XML format for analysis. The Apple Health app provided heart rate, energy expenditure, and step count data sampled at one minute granularity. For intense activity (running and max test), the sampling frequency was higher than once per minute. In cases where more than one measurement was collected each minute, the average measurement for the minute was utilized, since the minute average is the granularity for several of the other devices.”

So it gets better in this one.  They acknowledge they actually had the more frequent data samples (they’d have had 1-second data samples), but decided to throw those out and instead average at the minute.

But what’s so bizarre about this is how convoluted this study attempt was when it came to collecting the data.  Remember, here’s roughly what each participant did:

image

So you see what are effectively three and a half sports here: Walking, Running, Cycling, and…Sitting.

That’s fine (I like it actually as I said before). There’s complete validity in testing across all three and a half things.  But where the mistake was, is trying to treat it as a single entity and record the data streams live.  They skip over in the study procedure documents whether these devices were even switched between running and cycling mode for example.  None of the devices they tested were multisport devices.  So did the participant stop and start new activities?  The graphs above certainly don’t show that – because doing so on most of these devices isn’t super quick.

None of which explains the most obvious thing skipped: Why the not use the activity summary pages from the apps?

Every single one of the devices they tested will give you a calorie total at the end of the activity.  Here’s a few pages from these respective devices that show just that (left to right: Fitbit, Apple Watch, Microsoft Band):

FitbitCalories AppleWatchCalories MicrosoftBandCalories

Calories is prominently displayed for these workouts on all three of these apps.  This is the number they should have used.  Seriously.  Companies make this pretty little page so that every one of us on this planet can easily figure out how much ice cream we can eat.  It’s what 98% of people buy activity trackers for, and in this case they didn’t use the one metric that the entire study is based upon.

Instead, they tried to triangulate the calories using minute averaged data.  Which…isn’t real data anymore.  It’s alternative facts data.  And thus, you get inaccuracies in your results.

I went back to them and asked about this too, here’s whey they didn’t use the totals in the app:

“This is a good idea, but unfortunately per-workout totals are reported as a sum of calories for a given workout. We were instead interested in per-minute calorie expenditure, which would not be reported in the per-workout summary. The reason for  our interest in the per-minute values is that there is some adjustment as a person transitions from one activity to another (in both heart rate and energy expenditure). Consequently, in the 5 minute protocol for each activity, we  used the energy expenditure and heart rate for the final minute of the protocol (to ensure that a “steady state” rather than transient measurement was obtained).”

I get what they are saying – but again, that’s not giving an accurate picture of the calorie burn.  Instead, it’s only looking at the *average* of the one minute per each protocol chunk.  I’m not sure about you, but I don’t typically throw away my entire workout, save the last minute of it.  Also, by focusing on a single minutes worth of data, it serves to exaggerate any slight differences.  For example if you take one minute where one unit may be off 1-3 calories, but then multiply it out over a longer period – it exaggerates what didn’t actually happen.  We don’t know what happened in those other minutes, because they were thrown away.

And that all assumes they got the right numbers (for example, the JPG graph conversion is notoriously easy to get wrong numbers from).

Note: I did confirm with them that they configured each device within the associated app for the users correct age/gender/weight/etc as supported by that individual device.  So that’s good to see, a lot of studies skip this too – which also would invalidate the data by a huge chunk.

Wrap-up:

Early on in the study, they state the following:

“All devices were bought commercially and handled according to the manufacturer’s instructions. Data were extracted according to standard procedures described below.”

The only thing likely true in this statement was that all devices were bought commercially.  After that, nothing else is true.  The devices were not handled in accordance with manufacturer’s instructions.  Further, the data was not extracted according to manufacturer’s intent/instructions.  And then to imply the methods they used were ‘standard’ is questionable at best.  The standard method would be to just look at the darn activity pages given on every single app out there.  Did the calories match?  It’s really that simple for what their goal was.

Instead, they created a Rube Goldberg machine that produced inaccurate results.  Which is unfortunate – because I actually agree with the theory of what they’re saying: Which is that every company does something different with calories.  Unfortunately, they didn’t prove that in this study.  Instead, they proved someone didn’t read the manual on how to use the devices.

Which ironically is exactly the same problem the last time I dug into a study.  The folks in question didn’t use the products properly, and then misunderstood how to analyze the data from it.  No matter how many fancy algorithms you throw at it, crap data in means crap data out.

Once again, you have to understand the technology in order to study it.

Sigh.

Read the whole story
ssorc
167 days ago
reply
Melbourne, Australia
Share this story
Delete

NewsBlur’s Twitter support just got a whole lot better

4 Comments and 13 Shares

It was a little under a year ago that I declared Twitter back, baby on this blog. In that time, NewsBlur users have created over 80,000 Twitter feeds in NewsBlur. Since it’s such a popular feature, I decided to dive back into the code and make tweets look a whole lot better.

Notice that NewsBlur now natively supports expanding truncated URLs (no more t.co links).

And NewsBlur also supports native quoted tweets, where a user links to a tweet in their own tweet. NewsBlur expands the quoted tweet and blockquotes it for convenience.

Plus retweets now show both the original tweet author and the retweeting author. This means that you can quickly scan tweets and see where the retweet originated from. And retweeted tweets that quote their own tweets also get expanded.

It’s almost as if NewsBlur is inching closer and closer to becoming its own full fledged Twitter client. While NewsBlur already hit Zawinski’s Law (“Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.”) by supporting email-newsletters-to-rss, Twitter is coming up fast.

Speaking of which, I have this idea I’ve been noodling about better supporting Twitter habits that need to become less of a habit. I want to be able to automatically follow people from my Twitter tweetstream in NewsBlur based the frequency of their posting. I want to be able to infrequently dip into Twitter but still read the tweets from people who only post once a week or once a day.

In other words, I want Twitter, the RSS killer, to better integrate with an RSS reader so that I can pull out the good stuff from the unending flow of tweets. RSS means never missing anything, but Twitter’s main use case is anathema to how we use RSS. I don’t like to preannounce features, but this one intrigues me and if you agree, please share this story to let me know or to give me feedback on how you would like to see NewsBlur be a better Twitter client.

Read the whole story
ssorc
197 days ago
reply
Melbourne, Australia
Share this story
Delete
4 public comments
Groxx
198 days ago
reply
I read every tweet, and twitter makes that very hard to do (which is why I'm a happy user of Talon), so that sounds great :) Infrequent tweeters are often my favorites.

Personally, I'd be happy with a twitter folder, with a "feed" for each follow. With a bit of thumbs-training (and a per-folder focus setting? or something) I'd probably never go back to twitter.com at all.
Silicon Valley, CA
TheRomit
198 days ago
reply
I like the last idea (I current have some users as columns in Tweetdeck but RSS may be a cleaner solution).

How about Yahoo Pipes-like combination of some users' Twitter feeds that make one RSS subscription?
santa clara, CA
samuel
196 days ago
Well you can read multiple twitter feeds in a folder, and with Focus mode, you can already filter them. And folders have RSS feeds, so you could then pipe that out somewhere else.
TheRomit
192 days ago
Ah! That's a good idea. Thanks for the tip!
samuel
198 days ago
reply
I'm thinking something more automatic than post freq as training. A slider or segmented control and you are subscribed to users who post at the frequency or below. Possibly even multiple sub folders with different frequencies: 1 week, 1 day, and then all the rest.
The Haight in San Francisco
eldritchconundrum
197 days ago
You say "RSS means never missing anything", but to me that isn't true. Posts more than 30 days old cannot be in an "unread" state. The marking as read that occurs automatically after 30 days is literally "missing something". I can't find it anymore among the rest, once it's been marked as read.
samuel
197 days ago
I think a month is a fine line to draw. Everything on the Internet, at a high enough level, is measured in monthly usage. If you're not using NewsBlur once a month then you're not really using NewsBlur.
eldritchconundrum
197 days ago
Certainly one month is fine, for news items. But if I want to e.g. accumulate one hundred of updates of an ongoing story-intense webcomic, then read them all in one go, then I have to track manually where I last stopped, because unread doesn't mean unread. In other words, NewsBlur is not a general-purpose RSS reader, it is specialized for reading recent news. Well, the name says it upfront, so I guess I can't complain. News... And Blur.
samuel
197 days ago
To be fair, those stories aren't gone, they just aren't unread. That's why the default is "All Stories" and not just "Unread Stories".
superlopuh
197 days ago
The unread stories effectively disappear into the void though. I've definitely lost posts that I would have liked to read at some point this way. It would be cool if there was some sort of banner for recently/soon-to-be autoread posts. Automatically sending them to pocket would also be a useful feature.
eldritchconundrum
196 days ago
They are not gone, they are lost in the haystack. If I can't find them, that's the same thing.
tingham
198 days ago
reply
I think I'd like to have my entire feed in a separate drawer here and then be able to use training to push users I follow down (or up) in the visibility stack. Right now NB and TW are the only two pinned tabs in my browser. Getting this down to one would be fantastic. Post frequency as a function of training would be a really nice feature compression.
Cary, NC

Five great government grants for Australian startups

1 Share

Key points at a glance

  • While innovation and R&D are risky ventures for any business – the stakes are highest for Australia’s thousands of startups and small businesses.
  • Through a range of government grants and assistance, Australian startups and entrepreneurs can apply for extra funding to keep R&D dreams alive.
  • Top grants include the Entrepreneurs’ Programme, the Export Market Development Grant and the R&D Tax Incentive.
Five great government grants for Australian startups

No matter which way you slice it, innovation comes with added risks and costs. And in a business landscape studded with large incumbents and smaller upstarts, these risks are proportional – a scuttled research and development project will create a headache for any large company’s balance sheet, the same failed attempt could see a more modest outfit shut up shop.

Fortunately, help is at hand for small Australian businesses in the form of government grants and assistance. There are financial awards provided – if certain conditions are met – to a business or entrepreneur to help achieve a common goal. What’s more, such subsidies don’t need to be paid back, making a real difference to the bottom line.

We’ve compiled a shortlist of the top grants and assistance schemes that aid Australian innovation. You never know – you or your business might be eligible for one or more of these grants:

1. Entrepreneurs’ Programme

What is it?

Part of the federal government’s National Innovation and Science Agenda, the Entrepreneurs’ Programmea is a federal government scheme aimed at helping Australian startups and entrepreneurs on a several fronts – from accelerating competitiveness and productivity to commercialising new products, services or processes.

What’s on offer?

A range of support functions are provided, including co-funded grants, business advice, collaboration opportunities and incubator support.

2. Export Market Development Grant

What is it?

Aimed at current and future Australian exporters, the Export Market Development Grant (EMDG)b reimburses marketing and promotional costs for businesses in international markets. Examples include free samples, trade fairs, or marketing visits.

What’s on offer?

Eligible export businesses can have up to 50% of their promotional expenses covered, provided the expenses are over $15,000. Grants range from $5,000 to $150,000. Each applicant is entitled for up to eight consecutive grants.

Nifty Forms can help individuals and businesses access the EMDG.

3. Public Sector Innovation fund (VIC):

What is it?

At the state level, Victoria’s State Government oversees the Public Sector Innovation fundc, which issues grants aimed at fostering collaboration between the public and private sectors; leveraging innovative new ways to solve complex problems in policy and service delivery. This could include new ways to collaborate, experiential learning, or overhauling procurement systems.

What’s on offer?

The Victorian government provides grants ranging from $50,000 to $400,000 to organisations working alongside or within Victoria’s public sector.

4. R&D Tax Incentive

What is it?

Spearheaded by AusIndustry, the R&D Tax Incentived aims to cultivate new innovations and knowledge in Australia by reimbursing portions of a small business or startup’s research and development costs. It provides a tax offset that encourages more companies to ‘have a go’ and take the plunge into new R&D initiatives.

What’s on offer?

For eligible companies with an annual turnover of less than $20 million, up to 45% of R&D expenses (this changes to 43.5% for cost incurred after 1 July 2016) can be recouped through a tax offset. Unused offsets may also be able to be carried forward to future income years.

Nifty Forms can help companies check their eligibility for the R&D Tax Incentive.

5. Instant Asset Write-Off

What is it?

It’s not technically a grant or rebate, however the Instant Asset Write-Off was a flagship part of the government’s 2015 budget agenda. Coming into effect in November 2016e, the scheme is now available for startups and small businesses around Australia and could be a lifeline for investing in much-needed new equipment.

What’s on offer?

Businesses earning up to $10 million a year can take advantage of the scheme, with assets valued up to $20,000 able to be deducted.

a https://www.business.gov.au/assistance/entrepreneurs-programme-summary
b https://www.business.gov.au/assistance/export-market-development-grants
c http://www.vic.gov.au/publicsectorinnovation
d https://www.business.gov.au/assistance/research-and-development-tax-incentive
e https://www.ato.gov.au/Newsroom/smallbusiness/Lodging-and-paying/Instant-asset-threshold-increase-to-$20,000-now-law/

Read the whole story
ssorc
220 days ago
reply
Melbourne, Australia
Share this story
Delete

It is done – 52 cafes in 52 weeks……

1 Share

It is the 30th December 2016 and we have done it!  We have visited a different cafe every week for the whole year….that is a new cafe every 7 days. We have meet the challenge and now, I think, can consider ourselves to be absolute Brunswick ‘hipsters’.  We have had more coffee than we need.  We have eaten out far more often than we thought we would and much more than we need and have probably each gained a couple of kilos because of it. It has been a great adventure wandering through Brunswick and discovering the weird and the wonderful and those little secrets an old suburb keeps well hidden. At times it has been onorous to squeeze in a cafe amidst a busy week or on a weekend when all we have wanted to do was hide from the world. As the year progressed it was a little challenging to find the next one. But every time we have found it to be a good thing to do – to create a bit time and go for a walk and sit for a bit, no matter how short, and spend a few moments talking together and watching the people around us.

Our overall feeling about it is that it has been fun and that we have enjoyed the challenge.  Would we want to continue it into the new year?….no! Would we want to do it again?…..no way!  Do we want to try a new challenge in 2017?…….absolutely not! It has been fun but we are glad it is done.  

We have been asked many times which has been our favourite and which has been the worst but we have resisted reaching a conclusion on this.  Each place has had its good bits and its bits that probably could be done a bit better.  Certainly some places were much ‘slicker’ than others, some were very ‘organic’ and some really need to make some changes if they are to survive in the very competitive world of cafes in Brunswick.  I suspect that our experience each week has been strongly influenced by our own head-space….whether we were happy, stressed, busy, sad, melancholy, excited, lazy….. So much of life is shaped by our own attitudes and mood, it seems wrong to judge people working hard at their chosen path in life when they are giving it their best shot.

So it is thanks to all the baristas who have made us coffee and to all the wait staff who have tended our every whim and desire and to all the business owners who have greeted us at their door and thanked us for our custom and to all the people in all the cafes who have allowed us to take photos.

And one last image for the year of the things you see when you wander the back streets.  This extraordinary tree (we think) discovered on the way to cafe no. 52…..

rubbish-tree

 






Read the whole story
ssorc
324 days ago
reply
Melbourne, Australia
Share this story
Delete

Getting Started in the Seizure Prediction Competition: Impact, History, & Useful Resources

1 Share
Seizure Prediction Kaggle Competition

The currently ongoing Seizure Prediction competition—hosted by Melbourne University AES, MathWorks, and NIH—invites Kagglers to accurately forecast the occurrence of seizures using intracranial EEG recordings. This competition uniquely focuses on seizure prediction using long-term electrical brain activity from human patients obtained from the world first clinical trial of the implantable NeuroVista Seizure Advisory Sytem.

In this blog post, you'll learn about the contest's potential to positively impact the lives of those who suffer from epilepsy, outcomes of previous seizure prediction contests on Kaggle, as well as resources which will help you get started in the competition including a free temporary MATLAB license and starter code.

This competition is sponsored by MathWorks, the National Institutes of Health (NINDS), the American Epilepsy Society and the University of Melbourne, and organised in partnership with the Alliance for Epilepsy Research, the University of Pennsylvania and the Mayo Clinic.

Seizure Prediction Kaggle Competition

Melbourne University AES-MathWorks-NIH Seizure Prediction Contest: Problem Statement

For many people with epilepsy, seizures reoccur at random times and greatly disrupt their cognitive and emotional state, their ability to work and drive, and their social and economic situation. Being able to predict epileptic seizures will greatly improve the quality of life of people with epilepsy by either giving them a warning of an impending seizure so they can move to safety or activating an implanted seizure control device that can avert seizures through drug delivery or electrical stimulation of the brain.

How can we predict epileptic seizures using electrical recordings of brain activity? Or more specifically, how can we distinguish between brain activity a short time before a seizure from brain activity that is temporally distant from seizures?

section-divider

Background

Epilepsy

Epilepsy is characterised by the recurrence of seizures, abnormal brain activity events that have many types of cognitive and behavioural manifestations. Seizures are often referred to as fits or convulsions. The most well-known type of seizure being tonic-clonic seizures where a person loses consciousness, muscles stiffen, and jerking movements are seen. These seizures usually last 1-3 minutes and take can take a long time to recover from.

Around 1% of the world’s population has epilepsy and there are tens of millions of people who have drug-resistant epilepsy, where seizures are not satisfactorily controlled by anti-epileptic drugs and often drug doses that might control seizures lead to unwanted side effects. The most common treatment for people with drug-resistant epilepsy is resective brain surgery to remove the seizure generating region of the brain. Such surgery poses the risk of removing normal functioning brain tissue and sometimes seizures can return post-surgery due to potentially complex brain networks involved in an individual’s type of epilepsy. A newer option is the use of brain implants that electrically stimulate the brain to avert or control seizures (e.g. the Neuropace RNS system or the Medtronic Activa PC system), however, these devices do not incorporate seizure prediction algorithms and only offer improvement in roughly half of patients. Therefore, more improvements are needed. Reliable seizure prediction algorithms offer a chance to yield such improvements by activating interventions in a more timely manner when a seizure warning is generated.

Seizure Prediction

Given the ability of some epilepsy patients to predict their own seizures through cognitive or behavioural signs called ‘prodromes’, it is believed that although seizures typically seem to emerge through a random process there is an underlying deterministic or dynamical component to the process of seizure generation that could be reliably detected for the purposes of seizure prediction. At present no universal ‘pre-seizure biomarker’ has been discovered. This is likely due to the large degree of heterogeneity in the types of epilepsy. Therefore, a patient-specific strategy is likely to be the most successful option, however, seizure prediction approaches that can be applied to as broad a class of patients as possible will have the greatest utility.

The standard approach to monitor electrical brain activity is to record electroencephalography (EEG) using electrodes placed on the scalp (scalp EEG) or intracranially on the surface of the brain or within the brain (intracranial EEG). Electrocorticography (ECoG) refers electrodes placed on the surface of the brain and Stereo EEG refers to electrodes placed deep within the brain. Scalp EEG is non-invasive but more prone to artifacts and less aesthetically pleasing for people to wear for long-periods of time. Intracranial EEG is invasive and high risk, but has better signal-to-noise qualities and can be more localised to the seizure generating brain tissue. Because of the risks, intracranial EEG is only performed on hard-to-treat epilepsy patients with limited options as a precursor used in the planning of resective brain surgery. Usually recordings of intracranial EEG only last up to two weeks as the electrode implants are temporary and there are risks of infection due to the skull opening required for cables to pass through the skull to the outside data acquisition system. Scalp EEG recordings are also limited in duration because of degrading electrode fidelity and discomfort over long periods of time. On the other hand, seizures occur over a large range of frequencies from say 1 a month to 100 a day depending on the patient. Therefore, in a two week recording it is often the case that one only records a handful of seizures, which is inadequate for reliable training and testing of patient-specific seizure prediction algorithms. Much of the seizure prediction literature is based on such recordings and hence there is a great degree of uncertainty surrounding the clinical utility of much of the published literature.

Seizure Prediction Kaggle Competition

In 2013, a world first clinical trial of an implantable seizure advisory system was completed by NeuroVista Corporation and the University of Melbourne and its partner hospitals. Intracranial EEG electrodes were implanted on the surface of the skull and electrical brain activity was recorded for periods of up to three years in 15 patients. The device was fully implantable and communicated wirelessly with an external warning unit that emitted a red, white and blue light to indicate high, moderate and low risk of having a seizure. Because the device was fully implantable various risks common to standard intracranial EEG recordings, such as infection, were minimised, thus allowing for much longer recordings. Of the 15 patients, nine patients provided adequate data for reliable evaluation of seizure prediction algorithms. Three of these patients achieved very high sensitivity and very low percentages of ‘red-light’ high-seizure-risk warning time. For other patients it was difficult to predict even 50% of their seizures and novel improvements are needed. The current Kaggle seizure prediction contest provides data from three patients from this clinical trial whose seizures were difficult to predict.

Seizure Prediction Kaggle Competition

In 2014, a Kaggle seizure prediction contest was run using long-term data from dogs (obtained using the implantable seizure advisory system mentioned above in the human clinical trial description) and short-term data from humans. The current contest aims to see if any of the winning seizure prediction algorithms from the previous contest generalise to long-term human data or whether other novel approaches can be found. The findings and winning algorithms of the previous contest were recently published in the journal Brain and the forum webpage for the previous contest contains summaries of winning algorithms. Resources to other EEG-derived seizure prediction features published in the literature that might be useful for the current contest also appear below.

section-divider

Resources

Below is a list of resources for those interested in learning more about seizure prediction and entering the Melbourne University AES-MathWorks-NIH Seizure Prediction Contest. Note that two of the links below connect to research articles that require subscription. Check the links to see if you can access the document pdfs, otherwise try the options provided at the bottom of this link.

Seizure Prediction Review Articles and published EEG-derived seizure prediction features

Mormann F, Andrzejak RG, Elger CE, Lehnertz K (2007). Seizure prediction: the long and winding road. Brain 130: 314–333.

Most cited review in the field. Introduces the problem and has a great appendix that summarises the equations of many of the EEG-derived features considered in seizure prediction.

Gadhoumi, K., Lina, J. M., Mormann, F., & Gotman, J. (2016). Seizure prediction for therapeutic devices: A review. Journal of neuroscience methods, 260, 270-282.

A review of the most reliable studies on seizure prediction between 2007 and 2015.

Kuhlmann, L., Grayden, D. B., Wendling, F., & Schiff, S. J. (2015). Role of multiple-scale modeling of epilepsy in seizure forecasting. Journal of clinical neurophysiology, 32(3), 220-226.

Explores the new area of seizure prediction based on computational models of the brain.

The paper on the world first clinical trial that created the contest data

Cook MJ, O’Brien TJ, Berkovic SF, Murphy M, Morokoff A, Fabinyi G, D’Souza W, Yerra R, Archer J, Litewka L, Hosking S, Lightfoot P, Ruedebusch V, Sheffield WD, Snyder D, Leyde K, Himes D (2013) Prediction of seizure likelihood with a long-term, implanted seizure advisory system in patients with drug-resistant epilepsy: a first-in-man study. LANCET NEUROL 12:563-571.

The Previous Kaggle Seizure Prediction competition

Brinkmann, B. H., Wagenaar, J., Abbot, D., Adkins, P., Bosshard, S. C., Chen, M., … & Pardo, J. (2016). Crowdsourcing reproducible seizure forecasting in human and canine epilepsy. Brain, 139(6), 1713-1722.

You might also find the Kaggle blog on a review of the previous seizure prediction contest full of ideas for algorithm approaches.

Standard EEG analysis approaches

Thakor, N. V., & Tong, S. (2004). Advances in quantitative electroencephalogram analysis methods. Annu. Rev. Biomed. Eng., 6, 453-495.

Summarizes many linear and nonlinear time series analysis techniques from FFTs and wavelets through to mutual information. A good potential resource for basic EEG analysis and EEG-derived features.

Stam, Cornelis J. “Nonlinear dynamical analysis of EEG and MEG: review of an emerging field.” Clinical Neurophysiology 116, no. 10 (2005): 2266-2301.

Reviews EEG-derived nonlinear features.

See videos of talks from the 7th International Workshop on Seizure Prediction (www.iwsp7.org). Including a video on the outcome of the previous Kaggle seizure prediction contest that describes some of the winning algorithms that were applied to long-term dog and short-term human data, and a video introducing the problem of seizure prediction and standard approaches.

Free temporary MATLAB license and MATLAB starter code

MathWorks the creator of MATLAB is offering a free temporary license to use MATLAB and many of its signal processing and machine learning related toolboxes for the purposes of the current seizure prediction contest. It is also offering starter solutions that you can use and adapt for the contest. Find out more about free MATLAB and the starter code.

EPILAB: A non-proprietary MATLAB-based toolbox for seizure prediction

EPILAB is free. You should be able to extract MATLAB .m file scripts/functions for various EEG-derived seizure prediction features from the download.

How to read the contest data

Although MATLAB is not needed to analyse the contest data, the data are stored in MATLAB’s .mat file format as described on the contest data page. If you have MATLAB then reading in the contest data is easy. The contest forum also has some tips on reading in the data if you use R or if you use Python (option 1, option 2, option 3).

Other resources

Check out the forum for the current contest.

Seizure Prediction Kaggle Competition Forums

Ignore all of the above and jump right in!

Melbourne University AES-MathWorks-NIH Seizure Prediction Contest

section-divider

Bio

levin
Levin Kuhlmann is a senior research engineer in the Department of Electrical and Electronic Engineering at the University of Melbourne and the Brain and Psychological Sciences Research Centre at Swinburne University of Technology. His research focuses on signal processing, control theory and computational neuroscience applications to neural engineering, neuroimaging, anaesthesia, epilepsy and vision. He is interested in how the brain processes information at multiple-scales, from neuron to whole brain, and utilizing such an understanding to engineer improved diagnostics, interventions and therapies for brain-related medicine.

Read the whole story
ssorc
352 days ago
reply
Melbourne, Australia
Share this story
Delete
Next Page of Stories