Just adding report count (number of people) since 1 April 2010. Many of these people say they see this problem frequently even starting from an empty project.   

- 16 OS X mostly 10.5+ (7) 
- 15 Windows various (5)
- 6 Linux (our tarball or SVN) (2)

Number in brackets is number of people reporting that choosing the "safe" option to delete orphans left silence in the project (usually all of it). 

At least half of these cases suggest cutting audio is to blame. Several cite "Cut some audio then File > New (paste into it or not) then blockfiles are missing in the project cut from".

(In reply to comment #1)

Thanks. That's a total of 37 which is 0.002% of the 1,636,597
downloads of 1.3.12. Have *we* ever been able to replicate it? If not, shouldn't its status be UNCONFIRMED? 

I spent a good deal of time reviewing the relevant code, and made some clarity changes. I could do more review, but it might not be worthwhile if we cannot replicate it. (And I'm holding off pending Michael's work on that code, for bug 35.)

Is "we" developers? I can definitely replicate the *tendency* for this to happen. I have seen these errors literally dozens of times on Windows XP and 7 (in Unicode Release). 

As for "unconfirmed", my impression is that this isn't really relevant to our case where the bugzilla is not open access. Almost everything entered here will have been confirmed (as moonphase if needs be). I think that is the point of that term.

(In reply to comment #3)

A) I have seen three reports in the last 10 days where Audacity has apparently put .au files from one project into the _data folder for another project. Sadly the projects are no longer available or too large for user to forward on. The scenario isn't clear but it seems likely that multiple projects were open at the same time. 

B) I think Michael is still looking at a problem where a project I sent him with a Unicode character in the file name gave orphans when opened on Mac, and then deleting the orphans destroyed the project. It's not clear if this happened because of the Unicode bug. I'm going to try sending other Win-created projects to people on Mac.

I'm going to try harder to reproduce these problems if I have time. I have been very close to getting a scenario for A) but it won't repeat. I'm assuming (A)  could be made reproducible if sufficient time was given to it and to a lesser extent the other scenarios that have been reported.

Just to add grist to the mill…

The following is based on using Audacity 1.3.13 built 9Sep2010.

I am within hours of completing a major project which involved editing many thousands of hours of raw audio which was mainly MP3s and MP4s ranging in length from 15 minutes on the short end to 10 hours on the longer end. Each raw audio was brought individually into Audacity, Silence Finder was run and the result was saved as a project for later review.

Occasionally (no more than once or twice per 100 project openings) I will get the warning about the project containing "orphan block files". At first I chose to quit immediately – sometimes reopening (after closing and then reopening Audacity) the project did not result in the open block files warning. If I did get the warning with a subsequent reopening of the project I went back to the raw data and re-created from scratch. The last few days I have tried accepting the option of deleting the orphaned files. With one exception (out of maybe a half a dozen experiences) the resulting audio seem to be entirely intact. On the one exception there were large gaps in the audio which when I went back to the raw data were not there.

Is it possible that some of these problems are the result of some other program (a virus checker??) having a lock on a datafile at the same time that Audacity is trying to access it?

(In reply to comment #5)

> 
> The following is based on using Audacity 1.3.13 built 9Sep2010.

Might be better to test with newer one, but I did a quick review of commits since then, and don't think anything significant in related code. 


> 
> I am within hours of completing a major project which involved editing many
> thousands of hours of raw audio...<snip>

Great that you're testing it so hard. Thanks!


> 
> Occasionally (no more than once or twice per 100 project openings) I will get
> the warning about the project containing "orphan block files". At first I >chose
> to quit immediately – sometimes reopening (after closing and then reopening
> Audacity) the project did not result in the open block files warning. If I did
> get the warning with a subsequent reopening of the project I went back to the
> raw data and re-created from scratch. The last few days I have tried accepting
> the option of deleting the orphaned files. With one exception (out of maybe a
> half a dozen experiences) the resulting audio seem to be entirely intact. On
> the one exception there were large gaps in the audio which when I went back to
> the raw data were not there.

Do you have a "before" for any of those, preferably a small one?

Were any of the problem cases resulting from having multiple projects open at once?


> 
> Is it possible that some of these problems are the result of some other >program
> (a virus checker??) having a lock on a datafile at the same time that Audacity
> is trying to access it?
> 

Probably not, as they'd probably access it for read-only. And those would be for missing files, not orphaned.

Observation: The small blockfile option in running Audacity might be useful in testing here.  It causes many more blockfiles and so might increase the probability of orphans.

(In reply to comment #7)
> The small blockfile option in running Audacity might be useful in
> testing here.  It causes many more blockfiles and so might increase 
> the probability of orphans.
It hasn't helped me reproduce the problem when I have tried it. 

I think the highest probability of catching at least one of the problems is to have many projects open at the same time. Sooner or later I am 100% positive Audacity will put the blockfiles for one project in the _data folder of another project.    

I still see this bug once or twice a week on Win just creating a few tones in a single project window, then deciding to save the project and re-open it. Where I have had time to look, the orphans are usually correctly reported as such and are silent if dragged in (though there was no silence in the project). But sometimes the project is corrupted and has silent, apparently lost audio (although there are usually no reported missing blockfiles). This is "once or twice" out of circa 50 or 60 project saves.  

I continue to use 1.2.6 exclusively (on Win) for real world projects because I  do not completely trust 1.3. Bill on Mac seems to have more confidence. To be fair, 1.2.6 does not have consistency checking so its projects may have harmless orphans. But it has never once left me with holes in a project.  

In terms of user reports, adding in the average 2 - 3 per month on feedback, it's about 10 - 12 per month in 1.3.12 or 1.3.13 (mostly Win, but Mac and Linux as well).  

I know the developers do not agree but I consider this bug as a perfect case for allowing P2 moonphase in an early release of a stable series. While it (possibly) should not block a 2.0 release (which in most ways would be vastly superior to 1.2), there is no way in my book this is a P3.   


> I think Michael is still looking at a problem where a project I sent him
> with a Unicode character in the file name gave orphans when opened on Mac, 
> and then deleting the orphans destroyed the project. It's not clear if this
> happened because of the Unicode bug. I'm going to try sending other 
> Win-created projects to people on Mac.
I tried this a few months ago on the Forum (after Michael's specific Unicode fix). If the Win projects contained Unicode characters they were silenced when opened on Mac (due to what looked like errors "translating" the Unicode between platforms). I assume we still have this issue and that it would be a separate, repeatable but easily fixable P2. Michael, are you in a position to help test out a project on Mac if I send you one?

James wrote:
> There is a suggestion in comment 5 that a virus checker could be part of 
> the problem i.e. in causing orphans.  There was scepticism about this, 
> but....   a virus checker reading a file could prevent it being deleted 
> and so lead to it being an orphan.
I stopped Avast! scanning my 1.3.13 Unicode folder after reading Ed's comment, but it has made no impact on the instances where I get orphans on re-opening the project.

James wrote:
> The title needs to change.  There are/were I think really two bugs - (a) the 
> creation of orphans and (b) recovery from orphans
No direct work has been done on bug 137 to my knowledge. Isn't "recovery from orphans" bug 20? That work has not reduced incidences of bug 137. 

The bug title is correct (unless you want to split off "unreliable re-opening of a project that was simultaneously open with other projects". When Audacity puts the wrong .au file in the wrong folder, there are of course missing blockfiles.

Paraphrased from feedback@ report today (1.3.12 on Mac) - will see if I can persuade him to try 1.3.13.
"after spending hours so many file when I open them, come up with due to a bug or something, the files have been lost and i have the option to fill the space with silence. The thing is repeatedly losing entire projects. I have not moved the files or anything. Just save, go to sleep, wake up, open, gone."

I hesitate to say, but after a review of bug 20 and this one I'm still worried in the "Orphan Blockfile(s)" warning about the unqualified "They should be deleted to avoid disk contention." I still think it's reasonable to say that when that dialogue appears in crash recovery, but it seems a risk to do so with this bug if the orphans are valid data misplaced by Audacity that belongs in other _data folders.

I suppose that while this bug is open, the options are not to show that sentence, or say something like "These files should normally be deleted to avoid [disk contention], but if they are valid data for this or other projects, deletion would damage those projects". And possibly replace [disk contention] with something a bit less geeky like "possible errors".

>I tried this a few months ago on the Forum (after Michael's specific Unicode
>fix). If the Win projects contained Unicode characters they were silenced when
>opened on Mac (due to what looked like errors "translating" the Unicode between
>platforms). I assume we still have this issue and that it would be a separate,
>repeatable but easily fixable P2. Michael, are you in a position to help test
>out a project on Mac if I send you one?

Sorry, missed this.
Yes, if there is any testing needed that I can help out on send it my way.

(In reply to comment #12)
>>I tried this a few months ago on the Forum (after Michael's specific Unicode
>> fix). If the Win projects contained Unicode characters they were silenced when
>> opened on Mac (due to what looked like errors "translating" the Unicode between
>> platforms). I assume we still have this issue and that it would be a separate,
>> repeatable but easily fixable P2. Michael, are you in a position to help test
>> out a project on Mac if I send you one?
> Yes, if there is any testing needed that I can help out on send it my way.

Thanks, Michael. You can get the downloadable projects and see what happened back in August from this Forum thread:
http://forum.audacityteam.org/viewtopic.php?f=17&t=38446

I can replicate the scandinavian (i think) project file missing block file errors which open fine on pc.
The problem seems that the file unzips with a different file name due to encoding issues (the 'A' with an umlaut it becomes and 'e' with an accent grave and several other issues).
This appears to be an issue with mac unzipping.

Here is the error:
13:30:40: Warning: Missing data blockfile: '/Users/admin/Downloads/Win projects/Det ér VI éndÜ/Det Är VI Ändå_data/e00/d00/e0000924.au'

because the _data folder has the wrong name.
Not sure what to do as the issue seems to be outside of Audacity (the unzip function).
I'm not too knowledgeable about it but maybe it is possible to do something smart with encodings that would give us the badly translated name, or we could force a lookup of the project file's name plus "_data"

Created attachment 96 [details]
fixes unzipped project missing blockfiles error

In response to comment 14, I got this working on mac.
I just made audacity try a _data folder with the .aup's base filename if the one stored in the .aup fails.

Also, if Vaughan or other devs could look at the patch that would be great as I am not so familiar with DirManager and there is a change I had to make that I don't understand.
For some reason for mac only DirManager::SetProject was explicitly not returning false if the _data dir didn't exist.  
This line was entered in revision 702 by dominic so we may not get an explanation, but I can't see the reason for it.
The patch changes this behavior to be the same on mac as win and linux.  I tested a bunch of different scenarios without causing bad behavior.
However if the explicit mac behavior I removed is needed we can do an explicit check before calling DirManager.


again, I don't know if we want to fix this or not, since this is a zip encoding issue.  But it seems like it will affect many international users since zip formats do not include character encoding.

(In reply to comment #15)
Thanks for looking at it, Michael. So does Finder display the characters correctly, and unzipping breaks it? 

Ubuntu's default Archive Manager can't even extract the folder that has the Swedish project. The name appears to Ubuntu as "Det ?r VI ?nd?" and Archive Manager spits out a lot of "filename not matched" errors. Ubuntu reads "Det Är VI Ändå.aup" as "Det ?r VI ?nd?.aup" even when it sees it as is on a networked Windows 7 machine and dragging it over does not help.   

Hmm. The fix would be handy for Mac, but realistically perhaps we should not fix it, but note it as another restriction on this Wiki page
http://wiki.audacityteam.org/wiki/Sending_your_work_to_others#projects ? 

It's a shame because it breaks our ideal of projects being portable between OS'es.
I don't know how feasible it would be, but could we add to Bug 136 a suggestion that the Project Manager proposed therein could convert a project with foreign/accented characters to one that is suitable for sending cross-platform? If they were sending to a user with a Unicode version of Audacity then File > Save Project As should be sufficient, but for sending to someone who can only use ANSI, Audacity would I assume also have to "convert" metadata, track and label names and so on...

http://www.gaclrecords.org.uk/modified_project.zip (10 MB but too large for bugzilla) contains two folders:

* Archive - Fixed: A project called "ShadowS" where I had edited the metadata tags and saved the project.  The metadata edit I had made was to remove control characters from the .aup which 1.3.12 on Mac had inserted. This is a backup copy of the project at the time I had saved it. Another project was open at the time I saved this project, but it has not been affected.

* Archive - Fixed - Modified: The same "ShadowS" project immediately after Audacity had reopened it, and I had seen a subliminal dialogue "inspecting project files and folders" (or similar) and had seen "missing blockfiles" reported. This is a copy of the project at that time, before taking any action in the consistency dialogue. I then inspected the _data folder (also before taking any action in the consistency dialogue), and saw that e0001278.au and e0001f1d.au had been moved on re-opening from ShadowS_data\e00\d01\ to ShadowS_data\e00\d00 and had been given an updated timestamp. I then closed the project immediately without changes upon which Audacity removed the empty "d00" folder. The copy in "Archive - Fixed - Modified" of course has the empty "d00" folder. 

About the warnings given, blockfiles 0001278.au and e0001f1d were correctly stated as missing:

04:50:34: Error: can't open file 'E:\modified_project\Archive - Fixed - Modifiied\ShadowS_data\e00\d01\e0001f1d.au' (error 2: the system cannot find the file specified.)
04:50:34: Error: can't open file 'E:\modified_project\Archive - Fixed - Modifiied\ShadowS_data\e00\d01\e0001278.au' (error 2: the system cannot find the file specified.)
04:50:34: Warning: Missing data blockfile: 'E:\modified_project\Archive - Fixed - Modifiied\ShadowS_data\e00\d01\e0001278.au'
04:50:34: Warning: Missing data blockfile: 'E:\modified_project\Archive - Fixed - Modifiied\ShadowS_data\e00\d01\e0001f1d.au'
04:51:21: Warning: Project check found file inconsistencies inspecting the loaded project data. 

Should e0001278.au and e0001f1d have been stated as orphans to the d00 folder rather than being disregarded because they are not orphans to the project? At one level that could be confusing to present, but at another it identifies exactly what the problem is (the files are in the wrong folder). Or could we check all folders in a project and if the missing files are found, put the files back in the expected place? Of course we want to stop Audacity moving files inappropriately, but we know users do that too.   

I have tried making extra copies of the corrupted Mac project, repeating my edit, saving and re-opening the resultant project but without reproducing a problem.

Opening copies of the project in "Archive - Fixed" I have got Audacity once to move those same two .au files on project reopening into an added folder called "s" containing "ShadowS.aup" i.e the same .aup file as the project .aup:


Archive - Fixed (folder):

      s (folder)
-->      ShadowS.aup

      ShadowS_data (folder)
-->         e00 (folder) 
   -->           d00 (folder) 
   -->           d01 (folder)
 
      ShadowS.aup
        

Can't reproduce it it though. Most of the time that folder "s" does not cause an issue.     


----

I noticed that although when we open the same project a second time with File > Open or File > Recent Files we show an error "is already open in another window", there is no block on executing the .aup from your file manager and opening as many copies of the project as you like. I had not done that when Audacity moved the files around in error the first time, but can we block this anyway?

(In reply to comment #11)

On 2/17/2011 7:38 PM, bugzilla-daemon@audacityteam.org wrote:
> http://bugzilla.audacityteam.org/show_bug.cgi?id=137
> 
> 
> 
> --- Comment #11 from Gale Andrews <gale@audacityteam.org> 2011-02-18 03:38:06 GMT ---
> I hesitate to say, but after a review of bug 20 and this one I'm still worried
> in the "Orphan Blockfile(s)" warning about the unqualified "They should be
> deleted to avoid disk contention." I still think it's reasonable to say that
> when that dialogue appears in crash recovery, but it seems a risk to do so with
> this bug if the orphans are valid data misplaced by Audacity that belongs in
> other _data folders.

In auto-recover mode, orphans are ignored, except for a log entry ("They will be deleted when project is saved.")

So I'm not sure what you mean -- how does the dialog appear in crash recovery?



> 
> I suppose that while this bug is open, the options are not to show that
> sentence, or say something like "These files should normally be deleted to
> avoid [disk contention], but if they are valid data for this or other projects,
> deletion would damage those projects". And possibly replace [disk contention]
> with something a bit less geeky like "possible errors".
> 

Not sure who originally worded that "disk contention", but it does not match my understanding of the term. Per http://download.oracle.com/docs/cd/F49540_01/DOC/server.815/a67775/ch20_io.htm, "Disk contention occurs when multiple processes try to access the same disk." 

I think this comment in Audacity is only about not wasting disk space with unreferenced orphans, not about multiple processes accessing the same disk space. Right?


And at a higher level, if we reword it that way, how is the user to know whether this is a "normally" case? Seems to just give them more reason to worry and no real direction on what to do.

(In reply to comment #15)

> Created an attachment (id=96)
> fixes unzipped project missing blockfiles error
> 
> ...
> 
> Also, if Vaughan or other devs could look at the patch that would be great as I
> am not so familiar with DirManager and there is a change I had to make that I
> don't understand.

I don't understand why that wxDirExists(projFull) call is bad on Mac, but your change doesn't impact Win, so that's fine from here. And the changes to Project.cpp look right to me. Please patch and commit.



> For some reason for mac only DirManager::SetProject was explicitly not
> returning false if the _data dir didn't exist.  
> This line was entered in revision 702 by dominic so we may not get an
> explanation, but I can't see the reason for it.

Probably been fixed in wxW. That's a very old revision number.

(In reply to comment #19)

Applied Michael's patch (http://bugzilla.audacityteam.org/attachment.cgi?id=96) in commit r11022.

(In reply to comment #17)

Thanks for the attempt, Gale, but I don't get any problem opening "Archive - Fixed". I stepped through the code in DirManager::ProjectFSCK() (which puts up the "Inspecting project file data" progress dialog) and it found no errors, missing blockfiles or otherwise.

So I've still never been able to reproduce this moon phase bug.



> I noticed that although when we open the same project a second time with File >
> Open or File > Recent Files we show an error "is already open in another
> window", there is no block on executing the .aup from your file manager and
> opening as many copies of the project as you like. I had not done that when
> Audacity moved the files around in error the first time, but can we block this
> anyway?

This is a different bug from Bug 137, repeatable, and I think lower priority. But I went ahead and fixed it with commit r11025.

Also fixed previously unnoted bug where AudacityApp::MRUOpen() returned true when it actually failed to open the file. Also removed AudacityApp::OnMRUProject() cruft.

Unfortunately, this still creates a new, empty project window whereas Open and Recent Files commands do not. I think that constitutes a new, separate P5 bug. 

Overall, this is another aspect of what I was talking about in http://bugzilla.audacityteam.org/show_bug.cgi?id=322#c26. Opening files vs projects got conflated for convenience, but this code is hacked in some regards, rather than being a good design, and that's why this type of bug shows up.

(In reply to comment #21)
> Thanks for the attempt, Gale, but I don't get any problem opening "Archive -
> Fixed".
Thanks for trying. I am still hoping to reproduce something in this bug if it kills me.:=) 

There are also known scenarios that can set off this bug:  

* Having multiple projects open, editing then saving them all, then re-opening one or more of those projects. 

* Deletes/cuts or pastes at the end of tracks.

Both those are often reported but I have still not repro'd it. 

Is there anything you can do to just tidy up how the code behaves, in the hope that helps?

> fixed previously unnoted bug where AudacityApp::MRUOpen() returned true
> when it actually failed to open the file... Unfortunately, this still creates
> a new, empty project window whereas Open and Recent Files commands do not.
> I think that constitutes a new, separate P5 bug. 
Thanks. Now at bug 332.

This reproduces on Ubuntu and Win 7 for me. Only a harmless orphan unfortunately. 

1 New project.
2 Generate 30s tone.
3 Select between 5s and 6s and Edit > Cut
4 File > Save Project
5 File > Close
6 File > Recent Files and open the saved project.
7 Choose "Continue without deleting" or "Close immediately" in the orphan dialogue
8 File > Exit
9 Relaunch Audacity, File > Recent Files and open the saved project. No error. 

I didn't realise Audacity would delete the orphans on exit. I would be very concerned about doing that while we know orphans can sometimes be necessary for projects due to this bug.    

Vaughan wrote:
> Not sure who originally worded that "disk contention", but it does not match 
> my understanding of the term. Per
> http://download.oracle.com/docs/cd/F49540_01/DOC/server.815/a67775/ch20_io.htm,
> "Disk contention occurs when multiple processes try to access the same disk." 
> I think this comment in Audacity is only about not wasting disk space
> with unreferenced orphans, not about multiple processes accessing the same 
> disk space. Right?
Yes I found that URL too. I would assume the comment to be about wasting disk space, but it also might possibly be about increasing the risk of internal confusion in the project. I don't know if that's even possible now but I recall it mentioned as a concern in the past. 


> > "These files should normally be deleted to avoid [disk contention], but if
> > they are valid data for this or other projects, deletion would damage those 
> > projects"
> if we reword it that way, how is the user to know whether this is a 
> "normally" case? Seems to just give them more reason to worry and no 
> real direction on what to do.
No way to know unless they cross-check all their project files and folders. But in my opinion it's unsafe to say "they should be deleted" with the current state of this bug. Maybe just comment it out with a TODO to reinstate it when this bug is fixed? Partly moot (and potentially misleading) if orphans are being deleted on program exit irrespective of that setting. 

I may have got confused about orphans in recovery dialogues. Please ignore that unless I comment again.

(In reply to comment #23)
Confirmed on Mac PPC 1.3.13-alpha March 26, but ...

On step 8, Audacity crashes.
Exception Type:  EXC_BAD_ACCESS (SIGBUS)
Exception Codes: KERN_PROTECTION_FAILURE at 0x0000000000000000

On step 9, the file is no longer available in "Open Recent", so I opened it with File > Open.

(In reply to comment #23)
In case it helps...
In step 2, generate a chirp going from amplitude 0 to 1.
Step 2a: wait for the clock to tick over a minute
Do step 3
Look in the temp folder, mine is at
C:\Users\Martyn\AppData\Local\Temp\audacity_temp
Sort by time (hence why you waited a minute).  You can see the modified blocks,
including the one which is the deleted audio.  You can import the later blocks
into Audacity if you like to verify what they are - you can tell which bits are
which because you used a chirp.
Now do step 4.  You can still undo the cut, right?  That's why all the files
got copied across to the saved _data folder.
Now at step 5 (closing) the history get cleaned up but the extra file/s don't,
so they are no longer useful.  If there is a problem I think this is where it
is, in this example.

But I haven't spent much time looking into DirManager before.

(In reply to comment #24)

> On step 8, Audacity crashes.
> Exception Type:  EXC_BAD_ACCESS (SIGBUS)
> Exception Codes: KERN_PROTECTION_FAILURE at 0x0000000000000000

This is actually Bug 334, and is not specific to this bug. Sorry.

(In reply to comment #26)
Oops again --- see Bug 337 - Audacity crashes on exit if no project window is open, not 334.

(In reply to comment #25)
Yes I appreciate the "orphan" in this case is required for the undo process. In this case, the orphan includes the entire audio that was cut, extending from zero to just beyond where the cut finished. Clearly there is something specific to this case that means the orphan is not deleted on File > Close (but is deleted when you exit). Even this case isn't always repeatable (it's stopped happening for me now after several runthroughs of those 9 steps). 

So to clarify, my concern was that if Audacity finds orphans on re-opening a project, and you don't choose the option to delete them in the dialogue, then shouldn't it leave those orphans alone even when exiting?

(In reply to comment #22)

Forgive me more threading on this, please.


from Gale Andrews <gale@audacityteam.org> 2011-03-26 15:06:55 GMT ---
>> Thanks for the attempt, Gale, but I don't get any problem opening "Archive -
>> Fixed".
> Thanks for trying. I am still hoping to reproduce something in this bug if it
> kills me.:=) 

Please don't go that far! :-)

Thanks for work on the rest of topics I'm not responding to further, for brevity.


> 
> There are also known scenarios that can set off this bug:  
> 
> * Having multiple projects open, editing then saving them all, then re-opening
> one or more of those projects. 
> 
> * Deletes/cuts or pastes at the end of tracks.
> 
> Both those are often reported but I have still not repro'd it. 

Will do more on that. 


> 
> Is there anything you can do to just tidy up how the code behaves, in the hope
> that helps?

That's much of what I did on bug 20, so thought it fit per the title of this bug. Seems this bug is really about "unreliable project saving", e.g., maybe writing blockfiles to wrong project, rather than re-opening. Will look into more saving code.

(In reply to comment #23)

from Gale Andrews <gale@audacityteam.org> 2011-03-26 19:46:49 GMT ---
> I didn't realise Audacity would delete the orphans on exit. I would be very
> concerned about doing that while we know orphans can sometimes be necessary for
> projects due to this bug.    

Sorry, that's only for auto-recovered projects, not the case when the "Close project immediately with no further changes" dialog is shown.

>> I didn't realise Audacity would delete the orphans on exit. I would be very
>> concerned about doing that while we know orphans can sometimes be necessary 
>> for projects due to this bug.  
> Sorry, that's only for auto-recovered projects, not the case when the "Close
> project immediately with no further changes" dialog is shown.
I hope I'm not missing the point, but when steps 1 - 9 in comment 23 reproduce,  the "orphan" is retained when you choose "Close project immediately with no further changes" then re-open the project with File > Recent Files. But if you then File > Exit, the orphan is always deleted at that point. So it seems we are trusting that orphan was no longer needed for that or other projects. If that "orphan" was wanted for some other project, which we know happens, that other project will now fail when opened.

(In reply to comment #31)
> --- Comment #31 from Gale Andrews<gale@audacityteam.org>  2011-03-30 15:44:42 BST ---
>>> I didn't realise Audacity would delete the orphans on exit. I would be very
>>> concerned about doing that while we know orphans can sometimes be necessary
>>> for projects due to this bug.
>> Sorry, that's only for auto-recovered projects, not the case when the "Close
>> project immediately with no further changes" dialog is shown.
> I hope I'm not missing the point, but when steps 1 - 9 in comment 23 reproduce,
>   the "orphan" is retained when you choose "Close project immediately with no
> further changes" then re-open the project with File>  Recent Files. But if you
> then File>  Exit, the orphan is always deleted at that point.

You are correct Gale, the orphan gets deleted on exit if the project was opened then aborted etc., and so the problem with the project is not there the next time Audacity is launched as it's been cleaned up on exit.  This is not what the dialog says will happen.

The 'cut' orphan does get seen if you reopen the file without an exit, as it is still there.  But it may be needed for a 'paste' operation, so we can't delete it on close, in the current scheme of things.  This needs more thought from devels.

Deleting the orphan on exit is not what we tell the user we are going to do.  The comment at the top of DirManager::ProjectFSCK may be relevant but DirManager::SetProject may be more relevant I feel.

> So it seems we
> are trusting that orphan was no longer needed for that or other projects. If
> that "orphan" was wanted for some other project, which we know happens, that
> other project will now fail when opened.
>

where is the evidence for that "which we know happens"?  Reproducible case? Maybe  I didn't read all of the above

(In reply to comment #32)
> The 'cut' orphan does get seen if you reopen the file without an exit, as it
> is still there.  But it may be needed for a 'paste' operation, so we can't 
> delete it on close, in the current scheme of things.  
In the case in comment 23, the "orphan" is required for Edit > Paste to paste audio rather than silence. And when I've tried the steps in comment 23 but deleted at step 3 instead of cut, the "orphan" is deleted on File > Close at step 5. 

Even if we don't delete the "orphan" after cut in step 3 then close at step 4, the user could do so because we are pretty much inviting them to do so in the dialogue. This s possibly a P3 edge case (how do we present the orphan to the user in that specific case) but I think deleting orphans against the indications of the dialogue is worth a P2.     


> > So it seems we are trusting that orphan was no longer needed for that or 
> > other projects. If that "orphan" was wanted for some other project, which 
> > we know happens, that  other project will now fail when opened.
> where is the evidence for that "which we know happens"?  Reproducible case?
See comments 8, 10 and 22. Of the "10 - 12 per month" cited in comment 8, about half are definitely reporting losing data (which might be in other _data folders than it should be, if user or Audacity hasn't deleted it). Some of the others may be seeing the comment 23 case. 

These reports come from Forum, feedback@, -users list and my own mailbag.  I can dig up some forum examples if it helps where we have a bit more meat as to what happened. Yes the numbers are not staggeringly high, but by definition they are under-reported because most people don't report bugs. And the absolute numbers would scale up when people migrate from 1.2 to 2.0. Reproducible, well that's we'd all like.

(In reply to comment #33)
> Even if we don't delete the "orphan" after cut in step 3 then close at step 4
"Step 4" meant "Step 5", just a typo...

Changed URL field to link to wiki.

A clarification...  

The project checking dialog is incorrectly identifying the audio parented by the clipboard as 'orphans', since they are not children of the saved project.  If we ask that dialog to delete them, we delete clipboard audio and will paste silence when we next paste the clipboard.  If we ask the dialog to leave them alone, it will.  They will get deleted when we exit Audacity, because that's when the clipboard is no longer needed.  That deletion on end of life of the clipboard is what should happen.

So the error shown up by the steps in comment#23 is in identifying blockfiles as orphans which are not orphans.  We should be checking that things that look like orphans aren't actually children of the clipboard.

(In reply to comment #36)
Thanks, James. OK so I created an a missing blockfile and an orphan in a project having a copied-in WAV by renaming an .au file. I tried all three possible dialog scenarios where you could choose not to delete the orphan, exiting and restarting Audacity after each. The orphan was left alone. 

Then repeated in a project with a read-directly WAV that was available (so the "missing" error was for "Alias Summary File(s)"). All three scenarios tested OK. 

Then repeated in a project with a read-directly WAV that was unavailable (so the "missing" error was for "Aliased File(s)"). Again all three scenarios tested OK. 

Have I missed anything? If not, this is good because we can remove the piece in the Release Notes "Known Issues" about files being deleted against the dialog indications.
     
However if bug 137 is still in 1.3.14, I would still lobby for commenting out "They should be deleted to avoid disk contention" in the orphan dialogue (or some other agreed change). I don't think the current wording helps us get user input on fixing this if the user has suffered a loss.

(In reply to comment #36)
> The project checking dialog is incorrectly identifying the audio parented by
> the clipboard as 'orphans', since they are not children of the saved project.

That takes it on a step from what I said in comment 25.

> If we ask that dialog to delete them, we delete clipboard audio and will paste
> silence when we next paste the clipboard.  If we ask the dialog to leave them
> alone, it will.  They will get deleted when we exit Audacity, because that's
> when the clipboard is no longer needed.  That deletion on end of life of the
> clipboard is what should happen.

Indeed.  Accurate.

> So the error shown up by the steps in comment#23 is in identifying blockfiles
> as orphans which are not orphans.  We should be checking that things that look
> like orphans aren't actually children of the clipboard.

Agreed.  But a different (real/repeatable) bug to the unrepeatable one that is the main thrust of this thread.

I bet somebody with knowledge of FindOrphanBlockFiles, RemoveOrphanBlockfiles and clipboard could easily fix this.

It would address Gale's result of "I am still hoping to reproduce something in this bug if it kills me.:=)"... "Only a harmless orphan unfortunately" comments.

But not the main unrepeatable thrust.  Do we have a "CAN'T FIX" catagory for the main thrust to go in, and split this issue out?

(In reply to comment #38)
> Do we have a "CAN'T FIX" catagory for the main thrust to go in, and split this 
> issue out?
Yes we have a WONTFIX resolution, but this thread already is the Moonphase bug. What I would kill for is a way to repro the "main thrust". I've not given up yet. 

Have we really reached WONTFIX yet?

I won't have much time to look further into this in the next week or so, so Michael has agreed to look into it.

(In reply to comment #23)

I fixed this case in r11118:

> 1 New project.
> 2 Generate 30s tone.
> 3 Select between 5s and 6s and Edit > Cut
> 4 File > Save Project
> 5 File > Close
> 6 File > Recent Files and open the saved project.
> 7 Choose "Continue without deleting" or "Close immediately" in the orphan
> dialogue
> 8 File > Exit
> 9 Relaunch Audacity, File > Recent Files and open the saved project. No error. 

Is there still another case we are trying to fix for the P2 or is this enough to demonte it?

(In reply to comment #41)
> Is there still another case we are trying to fix for the P2 or is this enough
> to demote it?
I haven't tested your fix yet Michael (thanks). 

I think your fix is the only "repeatable" issue but I and (I believe James) are very concerned at the number of reports of errors and resulting data loss that we can't yet reproduce. I suggest you look at:  
http://wiki.audacityteam.org/wiki/Bug:137

which summarises things better than here. Unwanted file moving and renaming when there are multiple subfolders in the project and when there were multiple projects open simultaneously is a principal concern.

I see you used to be able to replicate the other cases of the bug with enough tries.
Another comment says this is more readily done with multiple projects.  Just in case If there is a single project method of recreating the bug please let me know so I can write a script to pressure test it 100s of times.  The scripting engine currently only deals with the frontmost project so the multiple project methods won't be scriptable unless we expand the scripting API.

(In reply to comment #43)
> I see you used to be able to replicate the other cases of the bug with enough
> tries.
Yes I used to get errors about one time in 40 but this was just arbitrary "pick your project, edit it and re-open it" testing. That error rate may have gone down now if your latest fix works because some of the problems were what you fixed. 

For some unknown reason, a few individuals get the problems so often that Audacity is unusable for them, as you can see.

> Another comment says this is more readily done with multiple projects. 
> Just in case If there is a single project method of recreating the bug please 
> let me know so I can write a script to pressure test it 100s of times.  The 
> scripting engine currently only deals with the frontmost project so the 
> multiple project methods won't be scriptable unless we expand the scripting 
> API.
I think having multiple projects open at once is our best chance. For the script we have now, I think you want to create a project with many subfolders.

We don't know what edits cause the problem or even if that is relevant. There is a suspicion that edits at the end of tracks (especially those that cut or paste at the end of tracks) is relevant.

We don't know if things go wrong when the project is saved on exit and/or when it is re-opened, but the re-opening stage is suspect because I have seen illegitimate file moving between subfolders at the stage (comment 17).    

There is also the case of projects with > 2^31 samples. We know those have issues with the whole set of .au files being declared as orphans on re-opening and silenced tracks. It's not known if that is repeatable, but it's assumed to be a special case. So probably best to keep below 2^31 samples, though the larger the project and the more subfolders, the better.

I gave a shot at scripting this, and got as far as getting a project to open from script by adding new command, but there are too many remaining issues that involve the scripting structure that won't allow the closing of projects, so I gave up for now.

I also did many edits with multiple projects and couldn't replicate anything.  We should watch if the frequency of the bug reports drop with the recent fixes.

(In reply to comment #45)
> I also did many edits with multiple projects and couldn't replicate anything. 
> We should watch if the frequency of the bug reports drop with the recent fixes.
Thanks for trying. 
 
In the last month, the frequency of reports (mostly with 1.3.12 but some 1.3.13 alpha/Beta) is the same. A couple of the latest reports (1 x 1.3.12, 1 x 1.3.13 Beta) have cited cutting and pasting a long track to multiple Audacity tracks above each other.   

I imagine the fix for the repeatable issue in comment 23 will mean the reports we still get will be most likely the headline moonphase issue.

http://www.gaclrecords.org.uk/bug_373c10.zip (being investigated for http://bugzilla.audacityteam.org/show_bug.cgi?id=373#c20) has verifiable orphan blockfiles 

e.g. bug 373c10_data\e00\d00\e00007c4.au 

that 1.3.12 sees when launching the project, but 1.3.14 does not see them at all.

(In reply to comment #47)
This problem was caused by commit 11118, around line 1729 in DirManager.cpp.  Since clipboardDM does not exist (line 1705), orphans never get added to orphanFilePathArray at line 1729.  It's a bug, for sure, but not relevant to the title.  But what is?

(In reply to comment #48)
> This problem was caused by commit 11118, around line 1729 in DirManager.cpp. 
> Since clipboardDM does not exist (line 1705), orphans never get added to
> orphanFilePathArray at line 1729.  It's a bug, for sure, but not relevant to
> the title.  But what is?
Thanks for the diagnosis. Now tested r11118. Just to confirm also, I ran through the tests in "\tests\Project Check tests" in the source code tree and "missing_blockfile.aup" is the only one that fails (no error when there should be). 

Since the bug fixed in r11118 was relevant to the bug title, but r11118 creates a new bug that orphans are not detected, one view might state the new bug belongs here. Another view might state that since we don't know why people receive the moonphase errors in the bug title, any irregularity with orphans and missing files must be reported here or on Wiki, irrespective.   

Yet another view states the new bug is repeatable (not moonphase) and it doesn't matter about raising blood pressure with new P1s and possibly "blaming" Michael for 11118. So let's do it your way and see if we prefer it: 
http://bugzilla.audacityteam.org/show_bug.cgi?id=395

See bug 409 for cases where empty subfolders are deleted. We should be sure this isn't relevant to this bug.

Forgot to note here that a fix for the r11118 regression was commited in r11171.
Will look at bug 409 and report back here.

To followup, I commited a fix for bug 409 in r11184, but I do not expect it to affect this bug.

Ooops, I meant I submitted a fix for bug 390.  I looked at 409 and I don't think it is relevant.

(In reply to comment #7)
Observation: The small blockfile option in running Audacity might be useful in
testing here.  It causes many more blockfiles and so might increase the
probability of orphans.

Please remind me, how do we do that?

This is clearly a long-running problem (with many sub/non-related issues in this thread).

Has anybody had the issue on a Debug version of the code?  Does it only happen on Release versions?

If so, does it give us a clue as to where the problem is?  A timing issue maybe?  Or is it just that nobody has had the patience to run enough tests under Debug?  1%-3% failure is what is being reported with simple operations like open, do something, close, reopen.  Why don't devs see it?

TTFN
Martyn

Just to remind, there is a more digestible summary at:
http://wiki.audacityteam.org/wiki/Bug:137

There is a suggestion there not made elsewhere that allowing the computer to sleep and wake up causes orphans:
http://wiki.audacityteam.org/wiki/Bug:137#Orphans_on_Sleep.2FHibernate

I've never yet reproduced a problem with orphans after waking up the computer (I have tried sleeping when the project is open and unsaved, open and saved, or just Audacity open). I suspect the reports could be the now fixed issue with misreported orphans on the clipboard.      

(In reply to comment #54)
>>(In reply to comment #7)
>> The small blockfile option in running Audacity might be useful in testing 
>> here.  It causes many more blockfiles and so might increase the
>> probability of orphans.
> Please remind me, how do we do that?
Run Audacity with the -blocksize option. This sets the Audacity block size for writing files to disk to nnn bytes.

I don't know if Michael tested with small blocksize. I have done so several times and it has not helped me reproduce the moonphase issue.

However there is a clear pattern of occurrence with either a) a single project having many subfolders or b) multiple projects open at once (even if the individual projects do not have many subfolders).   

(In reply to comment #55)
> Has anybody had the issue on a Debug version of the code?  Does it only 
> happen on Release versions? If so, does it give us a clue as to where the 
> problem is?  A timing issue maybe?  Or is it just that nobody has had the 
> patience to run enough tests under Debug? 
By definition most users reporting will not have a Debug version. When I and other users who are prone to see the problem have tried Debug builds, the issue won't happen. So I don't think there are *any* reports of this in Debug builds.  

Could running the computer under stress have an impact - like having Firefox open with 40 tabs when you test?

(In reply to comment #56)
>>> (In reply to comment #7)
>>> The small blockfile option in running Audacity might be useful in testing
>>> here.  It causes many more blockfiles and so might increase the
>>> probability of orphans.
>> Please remind me, how do we do that?
> Run Audacity with the -blocksize option. This sets the Audacity block size for
> writing files to disk to nnn bytes.

I can't make that work under debug in VS.  I have followed http://stackoverflow.com/questions/298708/debugging-with-command-line-parameters-in-visual-studio to no effect, but have had it working on a command line.

> I don't know if Michael tested with small blocksize. I have done so several
> times and it has not helped me reproduce the moonphase issue.

Maybe this is a red herring.

> However there is a clear pattern of occurrence with either a) a single project
> having many subfolders or b) multiple projects open at once (even if the
> individual projects do not have many subfolders).
>
> (In reply to comment #55)
>> Has anybody had the issue on a Debug version of the code?  Does it only
>> happen on Release versions? If so, does it give us a clue as to where the
>> problem is?  A timing issue maybe?  Or is it just that nobody has had the
>> patience to run enough tests under Debug?
> By definition most users reporting will not have a Debug version. When I and
> other users who are prone to see the problem have tried Debug builds, the issue
> won't happen. So I don't think there are *any* reports of this in Debug builds.

Hmm.  I'll have to think about that.

> Could running the computer under stress have an impact - like having Firefox
> open with 40 tabs when you test?

Who knows?  This whole thing is a mystery.

Martyn

I haven't tried the small blocksize.

I did look a lot at the static method GetCurrentProject() usage, and although there are a lot of places it is used where it shouldn't be, I didn't find any usage that would create this bug.

(In reply to comment #57)
>(In reply to comment #56)
>>>> (In reply to comment #7)
>>>> The small blockfile option in running Audacity might be useful in testing
>>>> here.  It causes many more blockfiles and so might increase the
>>>> probability of orphans.
>>> Please remind me, how do we do that?
>> Run Audacity with the -blocksize option. This sets the Audacity block size for
>> writing files to disk to nnn bytes.

>I can't make that work under debug in VS.  I have followed
>http://stackoverflow.com/questions/298708/debugging-with-command-line-parameters-in-visual-studio
>to no effect, but have had it working on a command line.

Setting arguments:
http://social.msdn.microsoft.com/forums/en-US/vcgeneral/thread/c655c026-e84d-4eab-b61f-a6f951643467/
Make it the startup project:
http://msdn.microsoft.com/en-us/library/a1awth7y%28v=VS.90%29.aspx

then it works.

At last I have a project that reproduces the main problem!!!

I was so happy to create this buggy aup file and _data!

http://dl.dropbox.com/u/1327769/test5168.zip

I used the -blocksize 1024 option in VS and looked at getting the number of blockfiles up towards what Gale and Ed had said they had seen errors at, about 36,000 - 53,000 blockfiles (in total).  I went for a smaller number than that for a test and captured this case on the first run.

I'm so happy with a real test case!  And it isn't a race/disk access problem.

Martyn

(In reply to comment #60)
Sorry!  That last comment was a red herring!  I had set "-blocksize 1024" and used floats but that is not a legal combination!

There is a check on the minimum "sequence maxsamples" value in an aup and the sequence will not be read if this value is less than 1024 (in Sequence::HandleXMLTag).  Setting the -blocksize to 1024 gives only 256 samples in "sequence maxsamples" and so the rest of the aup is ignored.  You don't get a message to say what has happened though!

REOPENED. I think the "DEVEL - FIX MADE" was for the repeatable clipboard orphans (comment 41). The core moonphase issues (and a possible approach to them focusing on testing Read and Write Returns) are very much open. 

I don't think anyone is clear as to the exact role of read and write errors here. We know that unwanted orphans and missing blockfiles can occur with up to about 5% frequency in stress-test situations, and in normal use in larger projects on less capable machines.

We know Audacity will occasionally delete or rename .au files when this is not wanted and break the project (it happened again to me recently as discussed with Martyn and Ed). 

So I think this should be re-opened. Ed did want to split off different failure scenarios to different P2 bugs, but I'm unsure it helps much other than to make a new working space. We do have the Wiki page Bug:137 to bounce ideas around. I don't object to splitting off if someone has a clear vision that this would help.

It's always arguable to demote this to P3 on the grounds "only" about 8 people a month now report something that sounds like the moonphase issues and "only" about 4 a month appear to claim data loss arising.  To demote responsibly would IMO require a specially prominent place in the Release Notes with a "back up to WAV" invocation.

A user has submitted a project created in 1.2 which as it's too large for bugzilla I've stored at:
http://www.gaclrecords.org.uk/bugs/bug_137_downing.zip

On Vista/1.3.13 (for the user) and Win 7/1.3.14 (for me), as soon as you accept the prompt to open the 1.2 project, the same 15 .au files are removed from the _data folder. This is 100% repeatable, so may give us a clue to the moonphase "deletion and/or renaming of .au files" issue that has been reported. The removed .au files are then not reported in a "missing blockfiles" dialogue, only as "Gap detected in project file" in the Log. 

Additionally, there is an orphaned blockfile in the  _data folder "b00000.au" 
which is not reported as an orphan.   

Two copies of the project are in the zip because the original project will be modified by the bug as soon as you open it. Note that opening the project will generate a "missing aliased audio file" dialogue: 

> Warning: Missing aliased audio file:
>'D:\andrew\programs\flash\momentum master\backups\matthew\crash sound.wav' 

I assume (am awaiting verification) that user has this WAV file in the correct location so does not see this error (and thus sees no error dialogues when opening the project, only the silenced tracks).    

I haven't opened a new bug for this but feel free to do so if preferred, as long as it links to this bug.

(In reply to comment #63)

I think this is a (possibly related but) different bug.  Please open a new bug for this.  PX.

This aup has lines such as:

<sequence maxsamples="262144" sampleformat="262159" numsamples="506880" >
	<waveblock start="0" >
		<simpleblockfile filename='b00005.au' len='506880' .../>
	</waveblock>
</sequence>

which clearly has len of the (single) simpleblockfile > maxsamples.  That shouldn't be, I believe. The sequence should have had more than one block? But 1.2 6 reads it OK, so 1.3 etc should as well, possbly with a warning.

What wrote it?  Not that it really matters.  We should really handle it, since it was most likely 'us'.

In HEAD Sequence::HandleXMLTag we are setting

            mDirManager->SetMaxSamples(mMaxSamples);

and

         else if (!wxStrcmp(attr, wxT("numsamples")))
            mNumSamples = nValue;

incompatibily with that broken file in Sequence::HandleXMLTag.  I haven't tracked exactly what causes the file deletions after that.

Index: src/Sequence.cpp
===================================================================
--- src/Sequence.cpp	(revision 11255)
+++ src/Sequence.cpp	(working copy)
@@ -790,6 +790,8 @@
          }
          else if (!wxStrcmp(attr, wxT("numsamples")))
             mNumSamples = nValue;
+         if(mNumSamples > mMaxSamples)
+            mDirManager->SetMaxSamples(mNumSamples);
       } // while
 
       //// Both mMaxSamples and mSampleFormat should have been set. 

is a quick fix, and enables this file to be read.  There should probably be a limit on mNumSamples where this would work without a warnng/stop.

So, an incomplete analysis of this (probably different) bug.

TTFN
Martyn

(In reply to comment #62)

Can you tell me how to stress test to get a 5% occurrence of the bug?
I have never been able to replicate this bug when doing (manual) stress testing.  Or do you mean out of all users stress testing, 5% can replicate it?
I searched the comments but didn't see anything related to this 5%.

(In reply to comment #64)
Thanks, Martyn. Creation and repeatable deletion of overlong blockfiles is now at bug 451.

(In reply to comment #65)
> Can you tell me how to stress test to get a 5% occurrence of the bug?
> I have never been able to replicate this bug when doing (manual) stress
> testing.  Or do you mean out of all users stress testing, 5% can replicate it?
Please contact Ed who will tell you how he gets failures when testing with a script. 

I don't mean that 5% of users who stress test can get some degree of failure. I suspect the percentage of users receiving at least one failure would be much higher than that if they were formally stress-testing on a production machine with other tasks going on. 

5% is about my failure rate from just working on 1.3 projects under normal operational circumstances (although many of such projects will only be tones in tracks and similar). The machine may be stressed in comparison with a pro developer's machine but that is not stress testing if by that we mean repeated opening and closing of dozens of simultaneous projects. If I attempt that it does (slightly) increase the failure % as far as I can tell. 

Remember too I am talking about Unicode Release not Unicode Debug. I don't regularly use Debug but I have never had a bug 137 incident in it to date.

(In reply to comment #67)

Well that's great news, I'm sure we can fix this if we can replicate it at least occasionally.  Is this new or have I missed this?

In any case I've asked Ed for some advice.

(In reply to comment #68)
> Is this new 
No it's a few months old. I didn't really follow the bug 137 discussion on -devel that was contemporary with what was going on, but I just added a note to the Wiki page about what Ed found and other observations of my own:
http://wiki.audacityteam.org/wiki/Bug:137#File_read.2Fwrite_errors
http://wiki.audacityteam.org/wiki/Bug:137#Stress_Testing

(In reply to comment #69)

(back from bug 451)

Gale, can you post a summary of remaining issues somewhere? I think "Summary Points" at http://wiki.audacityteam.org/wiki/Bug:137 is not up-to-date, right? And it's too abstract, and descriptive of symptoms rather than known causes, to figure out what to actually look into. The rest of that page is indeed indigestible, imo, a long trace of conversations.

(In reply to comment #70)
Symptoms (and *ideas* of possible causes) are all that are known, right? Otherwise it would be fixed already. Catch 22. Are the developers testing Unicode Release instead of Debug to try and provoke the bugs?  

Will take a look later at Wiki page but I don't sense there is all that much to add until bug 451 is finally wrapped up, unless someone said something pertinent on -devel that they did not add to Wiki.

(In reply to comment #71)

> Symptoms (and *ideas* of possible causes) are all that are known, right?

I was asking whether there are any known repeatable ways to make it happen at this point, as I think there have been at times, for this bug. 

IIRC, there have been several symptoms that were lumped together in this bug, some fixed by debugging reproducible causes, some fixed by code review. We've also added a lot more error-checking and asserts. If there's no clear series of steps to cause the problem, we should not hold up the next beta release for this. 


> Otherwise it would be fixed already. 

Not necessarily. Even being able to consistently reproduce the symptoms doesn't mean it's instantaneously, easily fixable.


>Catch 22. Are the developers testing
> Unicode Release instead of Debug to try and provoke the bugs?  

We might, if we understand what to do that is likely to cause the bug. We've made a lot of iterations on fixing this bug, and I was simply asking for an update. I cannot act on it until I know what to look for.

Meantime, I'll take a look at Michael's scripting code for stress testing.


> 
> Will take a look later at Wiki page but I don't sense there is all that much to
> add until bug 451 is finally wrapped up, unless someone said something
> pertinent on -devel that they did not add to Wiki.
> 

Afaik, there are no known tasks left to do on bug 451.

And rather than your adding to the wiki page, I'm primarily seeking an updated, succinct list of what's still outstanding. It's fine if you want to put it there, but the summary is out-of-date, I think some of those items are finished, and I don't see anything actionable there, at the current "sometimes it seems related to X" level of description.

Thanks.

(In reply to comment #71)
> Symptoms (and *ideas* of possible causes) are all that are known, right?
> Otherwise it would be fixed already. Catch 22. Are the developers testing
> Unicode Release instead of Debug to try and provoke the bugs?  

Yes, I've been using my scripting stuff to test on linux and mac, release build.
In total, I've processed about 200,000 project open/edit/saves over 10 days on three computers.
I've been varying the job/project size so it's not that consistent, but I haven't gotten this bug to pop up during this time.
I've done some of the suggestions such as keep a window open in the data folder, which forces the finder/ubuntu to calculate disk size etc on the blockfiles that come and go, and stuff like cpu strain, firefox, etc, but no antivirus software.

I haven't done windows yet though, since I need to do some work to restore my partition.  Due to some time constraints I will get around to this probably after dec 5, and give you a new patch that has mod-script-pipe working in win as well.  If you want to run the script on there before, just let me know, I can advise.

(In reply to comment #73)
> Yes, I've been using my scripting stuff to test on linux and mac, release
> build. In total, I've processed about 200,000 project open/edit/saves over
> 10 days on three computers. I've done some of the suggestions such as keep a 
> window open in the data folder, which forces the finder/ubuntu to calculate 
> disk size etc on the blockfiles that come and go, and stuff like cpu strain,
> firefox, etc, but no antivirus software.
> I haven't done windows yet though, since I need to do some work to restore my
> partition. 
Thanks, Michael! The only thing that I think Bug 137 Wiki page could be updated for is more on  "unable to write - disk full?" errors when trying to save projects. These are reported several times a week in 1.3.13 (mostly but not exclusively on Windows) and I am sure few of them are disk full or permissions/file name problems. 

If (Win 7 x64) I use a drive that is down to about 20% free space but still defragged as far as it can be, I get about 1 in 20 project saves and 1 in 40 autosaves to error. Maybe 1 in 30/1 in 50 on other drives, or my Win 7 i386 netbook. 

Either way, the drive still has ample space to save the project and Audacity is happy to export WAV to autosave folder or the folder it won't save the project to. I think this problem has only once corrupted a project for me (i.e. orphans/missing files result) but a couple of others have lost most of their project to orphans/missing files directly after seeing write errors (and abandoned Audacity as a result so I can't offer details). Even Ed gets this save failure occasionally on his super fast spacious Win 7 x64 system. I'll add his comments to Wiki. 

I cannot help feeling that something is wrong here, also that other write errors may occur that are not trapped/messaged and this may be even the most important remaining root cause of bug 137. 1.2.6 does not demonstrate "unable to save project" errors here although I still use it a lot for projects.

(In reply to comment #73)

>[...]Are the developers testing
>> Unicode Release instead of Debug to try and provoke the bugs?  
> 
> Yes, I've been using my scripting stuff to test on linux and mac, release
> build.
> In total, I've processed about 200,000 project open/edit/saves over 10 days on
> three computers.
> I've been varying the job/project size so it's not that consistent, but I
> haven't gotten this bug to pop up during this time.

Does that mean you've extended the script beyond the initial post? Might be good to post updates on this bugzilla entry.

It occurred to me that rather than adding commands, that would still be the same run every time, it could loop a random number of times (different each iteration) and insert a random number of commands/edits each loop, in randomized order...


> [...]
> I haven't done windows yet though, since I need to do some work to restore my
> partition.  Due to some time constraints I will get around to this probably
> after dec 5, and give you a new patch that has mod-script-pipe working in win
> as well.  If you want to run the script on there before, just let me know, I
> can advise.
> 

I've been running it in Windows. Am I missing?

We've been discussing this bug on the -devel ml.
Sorry for not updating the bugzilla thread - I forgot to do both.
There is a new patch there (from dec 18) that allows changing between projects via scripting.
Martyn also added a multi project script that tests copy paste between tracks that require rate conversion.

I've been testing in Release with some windows open in the data folder so the system is actively monitoring the files as suggested.
However, we haven't found any 137 replications with this yet.


I've been thinking about how to proceed with this.
Gale since you get an approx. 5% replication rate and so far we have gotten 0% over a large sample and all platforms, my suspicions are rising that this is a bug related to interaction with external software, and we may simply not be able to replicate on some systems.
If this is the case, we might be best off spending some time identifying the system component that causes it on your system and then install it on our systems and then debug it.

For example, I noticed Ed mentioned that he only gets errors when norton antivirus is running.  Do you have norton or some antivirus running when you get this 5%?  If you disable it, do you still get these errors?  Can you also tell me the version of your antivirus?

In the meantime, can we please ask the people who get this bug on mac if they have some particular antivirus or file observer application running on their system?

(In reply to comment #76)

> Gale since you get an approx. 5% replication rate and so far we have gotten 0%
> over a large sample and all platforms, my suspicions are rising that this is a
> bug related to interaction with external software, and we may simply not be
> able to replicate on some systems.
> If this is the case, we might be best off spending some time identifying the
> system component that causes it on your system and then install it on our
> systems and then debug it.
> For example, I noticed Ed mentioned that he only gets errors when norton
> antivirus is running.  Do you have norton or some antivirus running when you
> get this 5%?  If you disable it, do you still get these errors?  Can you also
> tell me the version of your antivirus?

Thanks, Michael. Avast! 5.0.677 on both my Win 7 systems. Disabling Avast! is not something I have done often, but it makes no difference. I have had Avast! several years but I normally do not allow it to scan in temp or project folders. When I have allowed it to scan there, I have seen no real difference.     

Neither Win 7 system has plentiful RAM - both the minimum (2 GB on the x64 and 1 GB on the x86). 

Firefox is almost always running with more tabs than I should have (15 in one window and 10 in another as I write).  

I always have xplorer2 (an Explorer clone) watching five or more windows as follows:
* the Audacity AutoSave folder
* the Audacity data folder (where audacity.cfg is stored)
* the Audacity temp folder
* the open Audacity project folder(s) 
* the folder audacity.exe is running in

The last unwanted (but harmless) orphans I got were when reopening a saved compressed project a few weeks ago. After that, on a whim I increased Firefox 8 (now v9) priority to "High" on the x64 while not changing any other app priorities. I noticed it made quite some difference to the overall responsiveness of the computer. 

I had not seen the orphan issue again even though repeating my steps for the compressed project. I returned Firefox to normal priority a few hours ago and have been trying the compressed project steps again. I got the orphans in attempt 3 out of 6, again they seem a duplicate of audio that wasn't edited.

Autosave or project save errors reduced considerably when Firefox was on higher priority (say once every 200 saves instead of once per 20 - 40 saves). There were no save errors in the compressed projects that had orphans. There was one autosave error in attempt 5 out of 6 on tonight's compressed projects.      

Shutting down all other apps seems to make little difference. I rarely have the luxury to do this but have still had orphans and unwanted file moves in the past with only Audacity running. I don't get any consistent sense others who have reported bug 137 problems have especially stressed systems.

I have tried using nearly full drives and nearly empty drives without seeing any difference in behaviour. 


> In the meantime, can we please ask the people who get this bug on mac if they
> have some particular antivirus or file observer application running on their
> system?
There haven't been any definitive reports on Mac since 1.3.13 that I am aware of. Maybe one Linux one in 1.3.13 on the Forum. All on Windows.

There is a chance http://gaclrecords.org.uk/bugs/cross_fade_out.zip (which refers to bug 528) may be useful for this bug. 

After saving the original project (which contained tones cross-faded using various Nyquist plug-ins) then reopening in 1.3.2, four legitimate orphans were found. That is the state the project in the above URL is in. 

After rendering that project and saving it in 2.0.1 alpha it now has 12 orphans. I think I probably rendered, then did and undid some cuts and repairs. I cannot reproduce the additional orphans now in a fresh download of the above, but if / when you want to look at this, you may find something here.

There are currently about six or seven credible reports a year of what looks like bug 137. There have been three credible reports in the last few months in 2.0.x on Mac OS X. See http://wiki.audacityteam.org/wiki/Bug:137#Q.26A .

This probably wants splitting off into separate issues, but for tracking, I'm promoting back to P3. There are still at least two or three issues reported a month on the Forum of broken project reopening which IMHO are unlikely to be user error. http://wiki.audacityteam.org/wiki/Bug:137 has some more information. 

If we are going to move to a unitary project format in the next year as suggested, will that simply negate bug 137 and spinoffs?