Bugzilla – Bug 290
Mac: LAME: Audacity looks for LAME in a restricted folder that Mac no longer supports
Last modified: 2018-08-20 11:51:37 UTC
uname -a Darwin David-Favor-iMac.local 10.6.0 Darwin Kernel Version 10.6.0: \ Wed Nov 10 18:13:17 PST 2010; root:xnu-1504.9.26~3/RELEASE_I386 i386 Attempting to Audacity -> Libraries -> FFmpeg Library -> Locate... for using MacPorts or FFmpeg_20090729_for_Audacity_on_OSX.dmg fails silently and message about no FFmpeg libraries remains. Manually adding either library set (MacPorts or .dmg) to audacity.cfg surfaces a variety of architecture mismatches. Here's the otool -hv info for all all components. Please note only the LAME library is picked up and works correctly, so I include it's data for reference. All this data wraps and becomes unreadable, so I'll regenerate each part and add it as an attachment.
Created attachment 81 [details] otool -hv /Applications/audacity-1.3.12/Audacity.app/Contents/MacOS/Audacity
Created attachment 82 [details] otool -hv /opt/local/lib/libmp3lame.0.0.0.dylib
Created attachment 83 [details] otool -hv /opt/local/lib/libavformat.52.64.2.dylib FFmpeg from MacPorts.
Created attachment 84 [details] otool -hv /usr/local/lib/audacity/libavformat.52.dylib FFmpeg from Audacity (FFmpeg_20090729_for_Audacity_on_OSX.dmg) Interesting... /usr/local/lib permissions are set to drwx------, which is most likely broken. I'll try "rediscovering" the FFmpeg libs from Audacity after doing sudo chmod ugo+x /usr/local/lib to see if this at least fixes this library set.
Had to sudo chmod ugo+rx /usr/local/lib as all group + other rwx bits were flipped off. This allows local users to access the FFmpeg libraries recommended by the Audacity site. Ah... Success for the these libraries. Great! MacPorts libraries still fail, so this is the only outstanding issue now. And... During Audacity install, ensure correct permissions set on each path component of /usr/local/lib. Whew...
Thanks for the entries, David. I know Leland will be taking a look at them. As a report like yours is not the first on OS X 10.6.x, Leland and I had already been discussing whether we should install the LAME and FFmpeg libs elsewhere on Mac. As you saw from my initial response by e-mail, we suggest sudo chmod -R 755 /usr/local/lib/audacity as one workaround for the user. It's fair to say though that in some reported cases where the user has upgraded their OS X version, their permissions are really messed up. The users needed a complete permissions reset to do anything much on their computer. If we remain with installing into /usr/local/lib, is there anything specific we can do? You referred to Audacity "dying" after editing audacity.cfg so as to enable FFmpeg. Was that a freeze or crash and can you still reproduce that? > MacPorts libraries still fail, so this is the only outstanding issue now It's covered by bug 176, but for most users (those installing the .dmg or zip) it's a non-issue - they should only be installing our FFmpeg package.
(In reply to comment #6) Seems like there are several possible quasi-fixes. 1) Add into the Audacity.dmg installation process changing permissions as in sudo chmod -R 755 /usr/local/lib/audacity 2) Add this same chmod into the FFmpeg library .dmg installation file 3) Change /Applications/audacity-1.3.12/Audacity.app/Contents/MacOS/Audacity to run setuid, do the chmod at startup, drop back to user EUID/EGID after issuing the chmod #1 & #2 are kludges and at least allow people recovering easily from changes to permissions. #3 is best solution and requires a bit of logic consideration. #3 is probably best implemented as parsing FFmpegLibPath and MP3LibPath then doing the chmod on each. This ensures correct permissions are set independent of which library set is being used.
(In reply to comment #8) #3 will definitely not happen! Way to many security exposures and too big of a hammer for a single problem. As far as permissions go, this really shouldn't have to be a concern for Audacity. Whatever is changing the permissions in the /usr/local hierarchy should be keeping their bloody fingers off of it. That is a standard (even according to Apple's docs) location for storing dylibs. Unfortunately, the Audacity team is the one that has to deal with the issues, so a solution needs to be devised and moving the Audacity specific libraries out of the /usr/local/lib folder will probably be the resolution. Ignoring permission issues for now, I see the main problem as being that non-Audacity specific ffmpeg and lame libraries exist on the system and those libraries are causing the library detection routine some headaches. Refer to this doc for an explanation of search order for dylibs: http://developer.apple.com/library/mac/#documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/DynamicLibraryUsageGuidelines.html#//apple_ref/doc/uid/TP40001928-SW21 When Audacity searches for the ffmpeg and lame libraries, it uses absolute pathnames, so the search order would be: 1. $DYLD_LIBRARY_PATH using the filename 2. The given pathname 3. $DYLD_FALLBACK_LIBRARY_PATH using the filename Notice how no matter where the user tells Audacity the libraries reside, there's still the possibility of an incompatible set being encountered by Audacity from some directory listed in the DYLD_LIBRARY_PATH environment variable. Say, for instance, you have built the MacPorts ffmpeg package and /opt/local/lib is listed first in the DYLD_LIBRARY_PATH variable. Audacity will find the (possibly) incompatible libraries in /opt/local/lib and stop looking. I'm still researching for the best solution, but I do have one possible workaround that will provide some relief to the issue. Leland
Thanks, Leland. I've widened the bug title to "LAME and FFmpeg detection problems". It's still only a Mac issue at the moment, but the problem of a competing library preventing detection of Audacity FFmpeg is also evident on Windows. Is the issue there similar - Audacity looks in Path first, as appears to be true from looking at the log?
(In reply to comment #10) Yes, Windows detection problems could be a similar situation. Here's the search paths for Windows: With both implicit and explicit linking, Windows first searches for "known DLLs", such as Kernel32.dll and User32.dll. Windows then searches for the DLLs in the following sequence: 1. The directory where the executable module for the current process is located. 2. The current directory. 3. The Windows system directory. The GetSystemDirectory function retrieves the path of this directory. 4. The Windows directory. The GetWindowsDirectory function retrieves the path of this directory. 5. The directories listed in the PATH environment variable. There's 4 possible locations from which the "wrong" DLL could be loaded (the first 4) since we use the PATH variable to specify the selected directory. Let's deal with them a little later since you aren't getting too many reports from the Windows users. Leland
I've done a little more research on the Mac issues and while I have found a way to use the user's selected directory, it is not a trivial change to Audacity. If the change were made, it would most definitely be Mac specific, so any risks would be isolated to Mac users. Since we also know that Windows could/does have similar detection issues where incompatible versions of the ffmpeg and lame libraries get located before the Audacity versions, how about we simply rebuild the ffmpeg and lame libraries with different names like: libavformat.audacity.dll libavformat.audacity.dylib It's not ideal and it doesn't really solve the root problem, but it might work. I haven't thrown in the towel just yet on finding a real solution, so we could just keep that one in our back pockets. Leland
I have a working solution for the Mac. It give the user full control over which directory to use (as long as the libraries are compatible with Audacity). Now, I just need to figure out how to add it to the Xcode project. I so, have a solution for Windows that will work for Windows XP SP1 or greater. Leland
(In reply to comment #13) Thanks, Leland. The problem with "other" FFmpeg versions stopping "ours" working is definitely a problem on Windows too. Maybe it does not get reported so often as Mac, but it's provable where it occurs. Delete the other FFmpeg, then Audacity will see its own FFmpeg. I suspect the LAME detection problem on Windows is partly different because it can still occur when there isn't a single other lame_enc.dll on the system. I'll drop you a line off-list about that. > I have a working solution for the Mac. It give the user full control over > which directory to use (as long as the libraries are compatible with Audacity). > ... have a solution for Windows that will work for Windows XP SP1 or greater. OK. We still nominally support Windows 2000 (and 98/ME if we release an ANSI 2.0 as I would like to). Does the OS X solution support 10.4 and greater? I think repackaging the FFmpeg installers to contain "libavformat.audacity.dll" or "libavformat.audacity.dylib" might not be a bad idea anyway, though less pressing if your better solution works fine.
Created attachment 88 [details] Fixes detection problems on Mac and possibly Windows This should fix the detection problems on the Mac and may even help with issues on Windows and Linux. On any of the platforms, the main issue is the search path for libraries and how absolute path names are handled. The Mac seems to be especially susceptible since there isn't one concrete location where libraries are stored. Windows handles absolute paths differently and allows runtime updates to the environment variables (and if push comes to shove, provides the SetDllDirectory() function), so if problems still exist, they should be easy to circumvent. This patch does four things: 1) It adds a shell script on OSX that takes care of starting Audacity after reassigning and clearing the DYLD_LIBRARY_PATH environment variable. This will allow loading of libraries from their absolute path rather than searching DYLD_LIBRARY_PATH first. This script should be transparent to the user, but it will affect people running Audacity with gdb as they will have to specifically target Audacity.app/Contents/MacOS/Audacity instead of the Audacity.app bundle. Not big deal really. If ppl no enough to use gdb from the command line, they should be able to figure it out. 2) It corrects detection of a monolithic FFmpeg library. This is one where avformat, avcodec, and avutil have all been linked into one large library. The main issue here was that under OSX and Linux, looking for symbols that should reside in avutil and avcodec, would always succeed since dlsym() on these platforms not only scans the requested library, but any dependent libraries as well. And when avformat was loaded, it would pull in it's dependent libraries as well. Another issue here was the if it was determined that the library was not monolithic, the library was never unloaded, so those dependent libraries may have come from the wrong location since they would have been loaded via the normal search paths and not by absolute path name. 3) It adds a method to FileNames which returns the full path name of a loaded module that contains an address. In the case of FFmpeg, it is used to verify that a routine from a specific library is actually being used from that library or not. It is also used to show (in Help->Show log) the directory from which a library was actually loaded. 4) It provides one possible means of using the latest FFmpeg libraries. I have not actually tested whether the libraries do what they should, but it does allow Audacity to at least load them. Look for INITDYNALT() in FFmpeg.h and FFmpeg.cpp. This bit can be easily removed if it proves troublesome. Leland
Added "patch" keyword. Thanks again, Leland. So should we commit this so that it's in 1.3.13 Beta and we get a chance to find any problems before 2.0 e.g. to see if we need more forcing of the absolute path on Windows? And also wait to see if the reduction in OS X issues to "pure permissions problems" lets us keep the libs in /usr/local/lib? Do the OS X people see any kind of new dialogue when loading FFmpeg? Wasn't sure what you meant by "transparent". For FFmpeg, what happens if (3) verification that "a routine from a specific library is actually being used from that library" fails?
(In reply to comment #16) I think it should be committed, but I did not do so as I'm not sure of the current review process. What I meant by transparent was the they should not see any differences in startup or functionality...other than the FFmpeg libs should be located better. Currently, the "verification" is only used to determine if the located avformat library is monolithic or not. I did start to add a check that would inform the user that the libraries were not loaded from the chosen location, but thought that could be added later if desired. The log will still show the actual results, so there is a way to tell. Let me know if you want me to commit and I will. Leland
Created attachment 89 [details] Fixes detection problems on Mac and possibly Windows Just added some protection when printing actual paths.
Created attachment 90 [details] Fixes detection problems on Mac and possibly Windows I removed the "support" for later versions of FFmpeg. It worked with a few of the filetypes, but would fail every time with WMA. Proper support would have to be done with something more along the lines of what Benjamin submitted. Leland
(In reply to comment #17) >I think it should be committed, but I did not do so as I'm not > sure of the current review process. I gave it a quick scan. Looks right to me, except I can't comment on the Mac-specific stuff. I say commit it, mark as DEVEL - FIX MADE, and let QA test it in the nightly build.
Fix committed.
(In reply to comment #21) As soon as 1.3.13 releases I'll test and comment on what occurs. Hopefully this will include a rebuild of ffmpeg-0.6.1 also.
(In reply to comment #22) You don't have to wait for 1.3.13 to appear if you want to test it out now. Nightly builds of Audacity can be found here: http://audacity.homerow.net/index.php?dir=mac (Didn't know if you knew about them or not.)
(In reply to comment #23) Thanks for the nightly build heads up. Install seems to work. And still ffmpeg libraries from 2009. Should I open a new ticket requesting a rebuild of ffmpeg-0.6.1 to bring libraries up to latest?
(In reply to comment #24) > Install seems to work. Great. Did you test on a clean .cfg (that only contains NewPrefsInitialized=1)? > Should I open a new ticket requesting a rebuild of ffmpeg-0.6.1 to bring > libraries up to latest? No. That's bug 176. There are far too many bugs in Audacity-with-FFmpeg to consider releasing an Audacity installer containing FFmpeg 0.6.1. Probably we want a specific ticket or more for those bugs. If you want to use 0.6.1, I suggest you use it at the command line (for export, you can use the Audacity command-line encoder).
Hi Leland, I am posting my errors building Unicode Release on Win 7 x64: FileNames.cpp ..\..\..\src\FileNames.cpp(227) : error C2065: 'HMODULE' : undeclared identifier ..\..\..\src\FileNames.cpp(227) : error C2146: syntax error : missing ';' before identifier 'module' ..\..\..\src\FileNames.cpp(227) : error C2065: 'module' : undeclared identifier ..\..\..\src\FileNames.cpp(228) : error C2065: 'GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS' : undeclared identifier ..\..\..\src\FileNames.cpp(228) : error C2065: 'GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT' : undeclared identifier ..\..\..\src\FileNames.cpp(229) : error C2065: 'LPCWSTR' : undeclared identifier ..\..\..\src\FileNames.cpp(229) : error C2146: syntax error : missing ')' before identifier 'addr' ..\..\..\src\FileNames.cpp(228) : error C3861: 'GetModuleHandleEx': identifier not found ..\..\..\src\FileNames.cpp(230) : error C2059: syntax error : ')' ..\..\..\src\FileNames.cpp(230) : error C2143: syntax error : missing ';' before '{' ..\..\..\src\FileNames.cpp(231) : error C2065: 'MAX_PATH' : undeclared identifier ..\..\..\src\FileNames.cpp(232) : error C2065: 'DWORD' : undeclared identifier ..\..\..\src\FileNames.cpp(232) : error C2146: syntax error : missing ';' before identifier 'nSize' ..\..\..\src\FileNames.cpp(232) : error C2065: 'nSize' : undeclared identifier ..\..\..\src\FileNames.cpp(234) : error C2065: 'nSize' : undeclared identifier ..\..\..\src\FileNames.cpp(234) : error C2065: 'module' : undeclared identifier ..\..\..\src\FileNames.cpp(234) : error C2065: 'MAX_PATH' : undeclared identifier ..\..\..\src\FileNames.cpp(234) : error C3861: 'GetModuleFileName': identifier not found ..\..\..\src\FileNames.cpp(235) : error C2065: 'nSize' : undeclared identifier ..\..\..\src\FileNames.cpp(235) : error C2065: 'nSize' : undeclared identifier ..\..\..\src\FileNames.cpp(235) : error C2065: 'MAX_PATH' : undeclared identifier
(In reply to comment #26) Oops...guess I'm out of practice. I'd only build the Unicode Debug version. Fix committed.
(In reply to comment #25) Someone please post the build script used to create the current ffmpeg libraries Audacity uses. I'll see if I can merge this into the MacPorts build to produce both a 64-bit (normal MacPort version) and 32-bit (for Audacity) library set as an option for people using Audacity + MacPorts.
(In reply to comment #28) Here's the build scripts and instructions: http://audacity.homerow.net/cgi-bin/dl?ffmpeg/FFmpeg_Libraries_for_Audacity_on_OSX_Build.tar.gz I tried it last week and it does not work for later versions of FFmpeg. I did not take the time to work out the issues. Leland
(In reply to comment #29) > Here's the build scripts and instructions: I already posted those at bug 292 comment 6 where David also asked for them. I think it is probably better discussing the possibility of porting Mac Ports FFmpeg to 32-bit there.
RESOLVED - FIXED for "LAME and FFmpeg detection (permissions problems and competing libraries)" as no problems were found detecting our FFmpeg on Windows when other versions of avformat-52.dll were in Path). REOPENED as (OS X) "LAME and FFmpeg installation (Apple permission problems)" which can be fixed after Bug 342 is resolved by installing LAME and FFmpeg to /Library/Application Support/audacity/libs (or similar).
(In reply to comment #31) > installing LAME and FFmpeg to /Library/Application Support/audacity/libs (or > similar). I think the frustration this causes OS X 10.6.7 users is unacceptable. If a new installer isn't possible for 2.0, I think the best answer is to only offer zips.
(In reply to comment #32) The current FFmpeg installer for OS X creates a folder named "~" at the root level of whatever disk you choose to install into if you choose the default installation. It is apparently trying to install to "~/Library/Application Support/audacity" but instead of the "~" being interpreted as the user's home directory the installer creates it instead. Then, of course, Audacity can't find the libraries. The LAME installer successfully installs to /usr/local/lib/audacity on Lion. Until the FFmpeg installer is working properly it should be removed from the download page, and there should be specific instructions on how to install using the zip version.
(In reply to comment #33) Bill, could you recheck that? This was a temporary problem on Buanzo's site around Christmas time and he said he had fixed it. Are you sure you are not getting a cached download of the FFmpeg installer? > The LAME installer successfully installs to /usr/local/lib/audacity on Lion. OK but for some people on Lion, Audacity won't be automatically allowed to read the dylib file there, which is the issue for both installers.
(In reply to comment #34) Cleared browser cache and downloaded FFmpeg_v0.6.2_for_Audacity_on_OSX.dmg. The DMG contains FFmpeg_v0.6.2_for_Audacity_on_OSX.pkg, created April 4 2011 10:32 am Run the installer, accepting all defaults, and it creates: /~/Library/Application Support/audacity and installs all the FFmpeg files in there. >> The LAME installer successfully installs to /usr/local/lib/audacity on Lion. > but for some people on Lion, Audacity won't be automatically allowed to read > the dylib file there, which is the issue for both installers. How widespread is that problem? As Leland notes, this is the standard location for dylibs on OS X. Is this only an issue for those running Audacity from a non-admin account? -- Bill
(In reply to comment #35) I've e-mailed Buanzo directly. He should I believe be serving http://www.audacity.homerow.net/cgi-bin/dl?ffmpeg/FFmpeg_v0.6.2_for_Audacity_on_OSX.dmg (package created 6 May 2011). >> The LAME installer successfully installs to /usr/local/lib/audacity on Lion >> but for some people on Lion, Audacity won't be automatically allowed to read >> the dylib file there, which is the issue for both installers. > How widespread is that problem? As Leland notes, this is the standard location > for dylibs on OS X. > Is this only an issue for those running Audacity from a non-admin account? The problem is common, as you can see from the Forum, but not universal. Someone was rattling on to feedback@ yesterday about it. /Library/Application Support/audacity/libs has been agreed as the new location with Leland: http://forum.audacityteam.org/viewtopic.php?f=17&t=48142&hilit=Leland+ffmpeg&start=20#p135711 I suggest we just do it, because if /usr/local/lib gets locked for read access to normal users, Apple lose the right IMO to dictate the "usual" place to install dylib files. If the problem happens, Finder shows /usr/local/lib as locked "no entry" (it's even locked to root or admin). You have to mark the locked folder, hit Apple+I, then in the window that opens, add rights for the admin group to read the folder. Then run Audacity as admin if you want to export MP3. Some people say you can add rights to a standard user, then export MP3 as a standard user. Most don't try as Apple support reps says to add admin rights. I do not know why this does not happen for all users on Lion. After giving the admin password to install LAME to /usr/local/lib/audacity, at least some can export MP3 from their standard user account. They cannot export from someone else's standard user account, right? That's crazy too. Let's make the change.
Leland's shell script and other changes committed in https://code.google.com/p/audacity/source/detail?r=10954 helped somewhat. What completely fixed the FFmpeg problems was to code Audacity to look in /Library/Application Support/audacity/libs for FFmpeg, and recommend an installer that installs there. Doing the same for LAME should allow us to close this issue.
Retitled as "(Mac) LAME: Audacity looks for LAME in a restricted folder that Mac no longer supports" to focus on the remaining issue. Promoted to P2. FFmpeg was corrected for this problem years ago. See Steps to Reproduce for the change that is required. Removed block on Bug 300 as this is not a building/linking issue any longer.
Created attachment 721 [details] Default to looking in /Library/Application Support/audacity/libs for LAME Attached patch by Leland to fix this bug. Audacity now defaults to looking in /Library/Application Support/audacity/libs for LAME, but also looks in the path used by the current installer (/usr/local/lib/audacity). If the old path is still available as well as the new, and there is no MP3LibPath set in audacity.cfg, Audacity prefers the new path. A new installer has been "found" that uses the current LAME 3.99.5 instead of the obsolete 3.98.2 used by the current installer. The new installer installs to the new default location and removes the old location (/usr/local/lib/audacity). A new build of 3.99.5 as a ZIP file has also been "found". The installer and ZIP file can be made available when the patch is available in a release. Changed code and the installer have been tested by Leland and Gale. The 3.99.5 libmp3lame.dylib file has been found to work correctly and supports the seven Audacity standard metadata fields on Snow Leopard, Lion, Yosemite and Sierra. Many thanks to Leland for his work on this.
Changed keywords to "patch".
DEVEL-FIX MADE https://github.com/audacity/audacity/commit/0efe931df26da43198b03972ed50b4c3c88b2662
I think this can be RESOLVED - it was extensively tested already by Leland and I.
I reopend this as (only) fix-made. It should be re-verified in a build after this commit: https://github.com/audacity/audacity/commit/dba49dd485d718a7ce5e42cf1cf4a7990d2e6a59
(In reply to Paul L from comment #43) Before running this test I reinstalled LAME as my privileged user. As per the instructions in the Manual it removed the old copy I had in /usr/local/lib/audacity and replced it with an installation in Library/Application Support/audacity/libs (which is what Apple recommends). Logged out of my privileged user account and logged in as Guest and then my Standard (non-privileged) Test account. For both of those Audacity (03Aug nightly) fired up with no library warnings. And on both of them LAME (and FFmpeg0 showed the correct Library/Application Support/audacity/libs pathing. And both LAME and FFmpeg worked on both those user accounts. Formerly when LAME was in my user account /usr/... directory, it could not be seen and had to be installed locally on the Guest a/c desktop - which was no problem, but of course it did get deleted each time I logged out of the Guest account. What we have now would appear to be much better. AFAICT this would appear to be fixed properly (certainly works for me) - but this is Macolyte under-the-hood stuff that I am certainly no expert in.
Tested on Mac Beta 02Sep17 on macOS Sierra 10.12.6 As in comment #44 Mac finds LAME and FFmpeg in their "correct" installed locations. For this test I did not delete and re-install, but rather left them wher I had installed in comment #44 so that I could test that 2.2.0 Beta found them properly there from there previous installation. I tested also on my "standard" non-admin account and on my totally unprivileged guest account and in both cases the libraries were correctly found I believe that this is no longer an issue on Mac - but leaving it open for now for someone else to close off
Testing on macOS Sierra 10.12.6 4a0fbf8 27Sep17 (and on the 2.2.0 Beta-1 for Mac) I still cannot get this to occur. Nor since the release of Beta-1 for Mac have I seen any complaints on the Forum of this occurring. I believe this is fixed and that we should close this bug as resolved.
Tested again on the official 2.2.20 RC1 (signed) on my Macbook Pro macOS 10.13 High Sierra LAME found properly, automatically, in my: a) privileged administrator account b) normal non-privileged user account c) totally unprivileged guest account Cliff has posted to say that he has no problems with this either - and I assume from the lack of reports from Bill and Steve that this all works fine on their Mac's too. The RC1 has been out in the wild for more general testing for a few days now and we have no reports of this issue recurring - so I am going to close these as RESOLVED
