Another ResScan Anomaly Question

[jerainey]

Maybe this has been covered in other threads, but I don't know how to search for it. My problem is lost data when I use the S9 for an afternoon nap. My practice has been to download the nap data selecting "Discard" for duplicate data. After reviewing I replace the SD card into the machine and use it normally at night. Next morning download again using discard and get nothing but the nap data in detailed graphs but the "statistics" claim that I slept 17:39 hours. I'm pretty sure I got up after my 3 PM nap and stayed up to my 10 PM bedtime This has happened before. Am I doing something wrong or is this another example of the brilliance of the ResMed software engineers? Have you had that problem? How did you solve it?

[billbolton]

Next morning download again using discard and get nothing but the nap data in detailed graphs...

Since you told ResScan to discard the later data from the same 24 hours period, ResScan is faithfully following your instructions.......

Am I doing something wrong ...

Yes.

Cheers,

Bill

[jerainey]

Thanks billbolton. So I guess the correct thing to do is to tell it to overwrite the "Duplicate" data. That is very non-intuitive and if there are instructions somewhere in ResMeds manuals, I haven't seen it.

[Blrfl]

Be careful how you use overwrite. If the machine has removed older detailed data to make room for newer, the summaries from older sessions will still be on the card. ResScan will see those summaries, and if you overwrite them, any detailed data stored on your computer for the same dates will go kerflooey. (That's a technical term. ) I guess ResScan isn't smart enough to gracefully deal with additional data within the same day.

One thing you could do to is reformat the card frequently enough that it never fills up and causes you to have a mix of summary and detailed data. If you always have detailed data, using overwrite should be safe because ResScan will overwrite what was stored with the same data from the card. Any dates that are already on your computer won't be touched if they don't show up on the card.

I run ResScan in a virtual machine that I can snapshot, so I can experiment to see how it behaves without risking any of my own data. If you'd like, I'll try some experiments sometime between now and the weekend and see if that's a safe thing to do.

--Mark

[billbolton]

I guess ResScan isn't smart enough to gracefully deal with additional data within the same day.

How is ResScan going to determine what your personal desires are in terms of data management? Telepathy features in software are notoriously difficult to implement

Cheers,

Bill

[Slinky]

It is simple, really. Just keep in mind that these PAPs have an internal, 24 hour, noon-to-noon clock. So one "day" to them is from noon one day until noon the next day.

Did you take your nap after your morning download of the night before and BEFORE noon? If so, Overwrite. (Your Tuesday MORNING nap data goes w/your Monday NIGHT data.

Your Tuesday AFTERNOON nap data goes w/your Tuesday NIGHT data.

[Blrfl]

I guess ResScan isn't smart enough to gracefully deal with additional data within the same day.

How is ResScan going to determine what your personal desires are in terms of data management? Telepathy features in software are notoriously difficult to implement

No telepathy required. Like every other application on the planet, you tell it what your preferences are. There's there's no reason ResMed can't give you some options for how to handle the data. Of course, whether they will or not is another story.

Some background: Every time you start a therapy session (i.e., you press the start button), the S9 creates a set of four files that contain the detailed data from that session. The files are made unique by including the date and time in the file name. For example, there's a set of files on my card from last Wednesday that are named 20101110_231157_xxx.edf, where xxx identifies which of the four files it is. Because there will never be another November 10, 2010 at 23:11:57, that set of files will be forever unique to my machine. Combine that with the machine's serial number (also stored on the card) and you have a means of identifying any session that takes place on any S9 that will never collide with any other.

There are three changes I'd make to ResScan to make its data handling more intelligent:

First is to retrieve and store sessions it hasn't seen before without raising a fuss because it's seen other sessions in the same 24-hour period. A session is a session and it's either been stored or it hasn't, and there's no reason for a 24-hour block to even be a factor.

Second is to ignore sessions on the card that have already been seen and stored. Each session has a checksum (like a fingerprint) generated by the machine, which makes it easy to tell whether or not what's on the card is the same as what's been stored. If what's on the card for a particular session differs from what's stored, then ResScan can throw up its hands and ask the user what to do. Unless some CPAP user takes up time travel and keeps going back to last Wednesday to sleep, I can almost guarantee you're not going to see duplicate sessions. (Someone intentionally fiddling with the clock doesn't count. If they break it, they own both pieces.)
Third is to keep a copy of the checksums for each session's detailed data stored with its summary data. Because the machine deletes old detailed data from the card when it needs more room, keeping the checksums would enable ResScan to identify summaries for sessions that have already been seen. That would make it possible to ignore already-stored sessions and would pose less risk of clobbering stored detailed sessions by overwriting it with a summary from the card.


I've been writing software for over 30 years, 25 of it professionally, and ResScan is a project I could do in my CPAP-augmented sleep. Unless the product is a complete mess on the inside, nothing I described above is difficult to do, and the user could be given options to make those behaviors automagic.

--Mark

[jlk]

That would be quite a step forward for resscan, too much for this year! I am not a software engineer, but do agree many changes woud be quite simple, at worst case maybe a larger sd card needed to hold a bit more data. Maybe Resscan 3.99 will allow more user friendly choices. john

[billbolton]

There's there's no reason ResMed can't give you some options for how to handle the data.

ResScan asks what you want you want to do on a case by case basis where there is an ambiguous situation, so you already have options

and you have a means of identifying any session that takes place on any S9

It already does that.

A session is a session and it's either been stored or it hasn't.....

If you read through ALL the different usages that have-been/are-now made for analysis software that have been described here over the years, you will see its not that simple for quite few applications of the software for specific purposes.

Unless some CPAP user takes up time travel and keeps going back to last Wednesday to sleep....

Crossing the date line is just one simple case of where problems arise.

I've been writing software for over 30 years, 25 of it professionally

I would have thought that you'd have learned something about data management, usability, the impacts of unanticipated usage (and crossing time zones) in that case

and ResScan is a project I could do in my CPAP-augmented sleep

I eagerly await the release of MarkScan version 1.0

Cheers,

Bill

[Slinky]

Yes! MarkScan! Shucks, someone wrote MyEncore to compliment EncorePro and James Skinner wrote EncoreAnalyzer to further compliment EncorePro. We ResScanner's would appreciate some freeware complimenting ResScan too. And gzv has been working on coordinating the ConTec CMS-50 oximeter software w/the ResScan software. God bless his heart! And James Skinner's heart. And the fellow who wrote MyEncore's heart as well.

[idamtnboy]

at worst case maybe a larger sd card needed to hold a bit more data.

The card is plenty big enough already. The limit of seven days high resolution data, 30 days of detail data, and 365 days of summary data are hard coded into the S9 firmware. My card back on Nov 1 had 95 days summary, 7 days hi res, and 30 days detail data. Total file size was 28.7 MB. The STR.edf file,which is the 365 day file, was only 16 kb. Expand that to 64 kb for a year's data and we're still way under the 1 Gb capacity of the card.

[Blrfl]

If you read through ALL the different usages that have-been/are-now made for analysis software that have been described here over the years, you will see its not that simple for quite few applications of the software for specific purposes.

Everything I've discussed has to do with collecting and storing data without losing any of it. The data is what it is no matter how it's going to be sliced and diced later. I'll go back and see if I can find the discussions you're talking about, but in the mean time, how about if you enlighten me a bit by telling me what use case is met by the behavior jerainey described?

Crossing the date line is just one simple case of where problems arise.

Neither the S9 firmware nor ResScan has any concept of time zone. If they did, all times and dates could be handled internally as UTC and sessions stamped with the time zone where they started to provide the correct local time during reporting. That would result in zero possibility for overlap no matter how many times you change time zones. Unix has used that approach for decades with great success.

I would have thought that you'd have learned something about data management, usability, the impacts of unanticipated usage (and crossing time zones) in that case

That's okay, Bill, you seem old enough that I would have thought that you'd have learned something about making your point without being insulting about it. Let's keep it constructive, shall we?

The OP has what I see as a legitimate problem. If he downloads the previous night's data in the morning and takes a nap in the afternoon, ResScan only gives him two options: the loss of some of his earlier data ("Overwrite") or the loss of the data from his nap ("Discard"). Neither allows him to retain all of the data his machine collected. Worse, after a discard which should have left that part of the data stored on his computer unchanged, ResScan shows a different number of hours slept for the previous night. In my world, the data loss problem is a design flaw and the incorrect sleep time is a bug. Maybe I'm just naïve, but doesn't all of this fall squarely under the headings of data management (not losing any of it), usability (making sure the user understands the implications of what he's about to do before taking a destructive action) and the impacts of unanticipated usage (taking therapy during naps in the middle of the day and retrieving the data from the card more than once in a 24-hour period)?

I eagerly await the release of MarkScan version 1.0

Sometime this winter (my winter, not yours) I plan to start producing at least two charts ResScan doesn't: detailed flow with events overlaid and pressure compared with a 30-day average for the same time of night. I don't anticipate releasing a program because it would have to be for Windows to be useful to a large number of people and I don't develop for that environment. I will, however, post some samples of what comes out of this effort. Maybe there's a business opportunity in value-added processing of people's CPAP data.

--Mark

[Slinky]

... If he downloads the previous night's data in the morning and takes a nap in the afternoon, ResScan only gives him two options: the loss of some of his earlier data ("Overwrite") or the loss of the data from his nap ("Discard"). Neither allows him to retain all of the data his machine collected. ...

Say what?? If he takes a nap in the afternoon that data goes on the next night's data. Go ahead, corn-fuse me some more! I've often taken a nap in the afternnon and I've not lost the previous night's data nor that afternoon's nap data. Maybe it is something w/the S9s? I've only used the S8s.

As far as the hours, just check the Summary graphs and put your cursor on the Usage graph bar, each "mask session" for that night and any nap before noon that day plus any nap after noon the day before will be listed w/the duration and the beginning and ending times of the "mask session".

For instance: I take a nap between 2 and 4 PM on Monday, 15 Nov 10, then go to bed for the night at 11 PM and sleep until 6 AM on Tuesday, 16 Nov 10. But I'm not feeling so hot so I take a nap from 9 AM to 11 AM on Tuesday, 16 Nov 10. When I do a download BEFORE noon on Tuesday, 16 Nov 10, the data from both naps and the night will be there. 3 mask sessions w/their times and duration on the Summary Graph Usage bar for 15 Nov 10 and the data from all three will also be included in the Statistics AND the Detailed graphs. You HAVE to keep that noon to noon clock in mind, not midnght to midnight.

If you opt to set the PAP clock from midnight to midnight but go to bed before midnight and nap for a couple of hours after noon the previous day and before midnight that night and then again after rising in the morning but before noon you are going to run into the same situation.

Or - if you want to download your afternoon nap data before going to bed that night, go ahead and do so, but don't save it. When you then download the data from your following sleep session that nap data will be downloaded again along w/that following sleep session.

[billbolton]

Let's keep it constructive, shall we?

Another case of what's good for the Goose is not good for the Gander.

Mea culpa, for feeding the troll!

[jerainey]

Mark and Bill : I really appreciate you knowledgable guys trying to educate me and I eagerly await ResScan 3.xxxx which will allow me to do what I want to do, and, hopefully, tell me how to do it. My 20 or 30 years as a dumb Apple user has indoctrinated me in the idea if I THINK something should work like so I expect it to work like I think. For years I told people "I don't do Windows". But now that I want to know what my sleep experience is I'm forced to do Windows, and I'm confused and disappointed. Oh well; Thanks for trying.