Latest post Wed, Jan 30 2019 11:16 AM by derDoyle. 23 replies.
Page 2 of 2 (24 items) < Previous 1 2
Sort Posts: Previous Next
  • Fri, Jan 11 2019 12:21 PM In reply to

    • chuckkahn
    • Top 500 Contributor
    • Joined on Thu, Oct 13 2005
    • Toronto
    • Posts 350
    • Points 3,870

    Re: Anyone Fooling Around with Sparse Disc Bundles?

    Is there a way to use a Mac formatted drive with symlinks and SDBs on a Windows Avid?

    Avid Media Composer 2018.11 / Mac OS Sierra / MacPro5,1 / 6-Core Intel Xeon 2.66 GHz x2 / 64 GB 1333 MHz DDR3 ECC [view my complete system specs]
  • Fri, Jan 11 2019 4:47 PM In reply to

    • chuckkahn
    • Top 500 Contributor
    • Joined on Thu, Oct 13 2005
    • Toronto
    • Posts 350
    • Points 3,870

    Re: Anyone Fooling Around with Sparse Disc Bundles?

    Comparing ways to create Sparse Bundles....

     

    1) Disk Utility -- I set the SBD size to 8 TB and the size of the .sparsebundle file is 1.51 GB.

     

    2) CreateDiskImage app -- also set the SBD size to 8 TB and the size of the .sparsebundle file is 59.6 MB

     

    Why the difference in sizes?

     

    Also, how do I create a proper SBD in the OS X Terminal?   I looked at the options in the hdiutil command and they seemed quite numerous:

    https://pastebin.com/mEWL3i82

     

    So I tried this:

     

    hdiutil create -size 8tb -type SPARSEBUNDLE -volname "bkp_WM_S2_Masters" -fs "Case-sensitive Journaled HFS+" /Volumes/WM3_Edit_Bkp3/bkp_WM_S2_Masters.sparsebundle

    and the size was 1.53 GB.

     

    So why does the app make the .sparsebundles smaller?

     

    update:  I think I see the reason:  the Create Disk Image app defaults to 10 GB and so its .sparsebundles are smaller than the 8 TB ones I created.

     

    update: adding a -nospotlight option:

     

    hdiutil create -nospotlight -size 8tb -type SPARSEBUNDLE -volname "bkp_WM_S3_Imports" -fs "Case-sensitive Journaled HFS+" /Volumes/WM3_Edit_Bkp3/bkp_WM_S3_Imports.sparsebundle

    Avid Media Composer 2018.11 / Mac OS Sierra / MacPro5,1 / 6-Core Intel Xeon 2.66 GHz x2 / 64 GB 1333 MHz DDR3 ECC [view my complete system specs]
    Filed under:
  • Fri, Jan 11 2019 5:01 PM In reply to

    • chuckkahn
    • Top 500 Contributor
    • Joined on Thu, Oct 13 2005
    • Toronto
    • Posts 350
    • Points 3,870

    Re: Anyone Fooling Around with Sparse Disc Bundles?

    My use case is to back up project ISIS workspaces to an external drive to subfolders and then create sparsebundles with symlinks so that an editor can access the sub-foldered backup in Media Composer.   That way I don't have to predict what final partitions sizes I need and I can hide the MXF backups from Avid's scanning for new mediafiles -- the SBD/symlink is the gateway to making the subfoldered MXF backups mountable/unmountable.

    Avid Media Composer 2018.11 / Mac OS Sierra / MacPro5,1 / 6-Core Intel Xeon 2.66 GHz x2 / 64 GB 1333 MHz DDR3 ECC [view my complete system specs]
  • Mon, Jan 14 2019 3:30 AM In reply to

    Re: Anyone Fooling Around with Sparse Disc Bundles?

    chuckkahn, in direct response to your first question, to my knowledge - no. But I read your second post about your use-case and unless I've misunderstood what you're trying to do then I think I can advise you what steps to take.

    For the first problem, that of mac drives on windows machines, Windows doesn't support HFS drives natively, however, if you haven't already chosen what drive to use and haven't already begun your back up process before reading this, then you can use the option of formatting the external drive to the exFAT filesystem (rather than HFS) which will be accessible to both windows and MacOS. exFAT has drawbacks in reliability, and is reputedly more likely to have corrupted data (for example if something happens to interrupt a file transfer in progress) than NTFS or HFS+, I mention this because it would be bad not to disclose the risks however I'll also note that, you're calling this a 'backup' drive and since you're handing it off to editors there's presumably several backups of that backup that are left inactive specifically just to store the data and not be messed with. So then, as long as you take this corruption risk in to account, AND HAVE OTHER, SAFER BACKUPS IN PLACE THAT AREN'T BEING REGULARLY ACCESSED BY EDITORS, my anectdotal experience is that exFAT has worked perfectly well for editing with no increased incidence of issues compared to any other filesystem. Otherwise if there is a reason it must be Mac formatted, there are also paid software options to provied support for HFS drives in Windows, so one way or the other you can access the material on the mac drive on your windows machine.

    Symlinks created in MacOS do not work in windows, however windows is capable of creating symbolic links of its own, which likewise work only on windows, and also only on NTFS. Unfortunately, the mixing of platforms makes things complicated. Theoretically, since the filesystem itself allows for symlinks, I wonder if the paid solutions for HFS support on windows also provide such support, but I have no idea, and somehow I doubt it.

    If you are wanting to backup the contents of one Avid MediaFiles directory to an arbitrary folder (other than \volume root\Avid MediaFiles) in order to hide those backed up files from Avid's scanning, but you also want a quick way to make those arbitrary folders behave like a traditional (\volume root\Avid MediaFiles) that Avid can use it as a media directory, your means and ability to do so depends on the platform the editor will be using. I hope I'm not misunderstanding your question but I'm guessing you'd be giving the editor this drive either as a contingency in case an emergency has happened and the ISIS is down and work must continue using the backups, or because that particular editor is for some reason not given access to the ISIS for their particular work and must use the external drive. I also have to assume that there's a reason you can't use the simple method for achieving this and that's why you want to use symlinks and sparsebundles. I'll list what the simple method (which uses neither symlinks nor sparsebundles or any kind of virtual volumes) is below for reference but will also list what you need to do if indeed you can't use the Simple Method

    The Simple method

    So you are making backups from ISIS storage to an external drive and you want the destination folder for those backups to be something other than \volume root\Avid MediaFiles in order to avoid Avid scans. Presumably then, when a backup is complete, and you've handed the drive to the editor and you're on their machine, you can just cut and paste the mxf folder you backed up from the arbitrary folder you chose, to the drive root in a folder called Avid MediaFiles.

    Drawbacks:

    This method allows only one project accessible at a time on the editor's machine but I'm guessing on average each of the editors you'd hand the drive to would only need the one project at a time and if they need to switch you simply close Avid, move the folder back to it's original backup location and do the same trick with a different backup folder you may have on that drive.

    Using arbitrary folders as real working Avid media folders (windows method)

     

    1. 1) Although windows is capable of symlinks, if your goal is just as the title for this method describes, you don't need them. That's a good thing too because windows doesn't as far as I know have a sparse bundle equivelant meaning disk images would need to be huge otherwise. Actually achieving this goal is easier in windows and requires less rigmarole than MacOS.
    2. 2) To make a given folder act like an Avid media folder, you need to place an 'Avid MediaFiles' directory and MXF subdirectory inside it, as though the folder were itself a drive and then you need to right click on the folder containing the media, access it's properties and make it a shared folder.
    3. 3) Now access your own computer (the editor's computer) as a network location by typing \\COMPUTERNAME in to an explorer window address bar or through the run command, and you'll see all the sharepoint locations available on the computer, one of which will be the folder you flagged as shared earlier.
    4. 4) Right click the sharepoint and click 'map as network drive' and assign it a drive letter. You'll now have a new volume listed in the windows 'computer' menu with it's own drive letter, within this 'drive' there will be an 'avid mediafiles' folder meaning as far as Avid is concerned, you've plugged in a drive with an Avid MediaFiles folder on the root, which is all the preconditions necessary for Avid to the read the media from that location.

     

    Drawbacks

    Although you are accessing material on locally attached storage and not over a network of remote machines, it is still a 'network drive' as far as windows is concerned meaning the maximum available speed of the drive is limited to the speed of the machine's NIC which for most machines is 1Gb. This is mostly fine, and if you have 10Gb ethernet card or something even better then it's totally fine, but sometimes with certain material or in certain rare situations, this is problematic. You should test first for a given project. It also means that if you use the 'consolidate' function to move a lot of material around then the speed is compartively slower than it otherwise might have been on that locally attached drive (depending on the type of physical storage and data connection used). However if you're talking about a mechanical hard drive connected by USB 3.1, the read speeds of spinning platters isn't often that much higher than 1Gbps anyway. RAID0 drives can probably do better so you would be hampering the speed of your external RAID in such a case but again, with testing you'll probably find that it handles most requirements with ease.

     

    Using arbitrary folders as real working Avid media folders (MacOS method)

    The full method is described earlier in the thread, so this is just here for me to point out that, if the windows machine in the mix is just an intermediary being used for making the backups to the external drive, while the editors' machines are macs, there is no way you can prepare the backup drive ahead of time for editors to simply plugin and mount a sparsebundle and be ready to go, windows gets in the way here because it A: supports only its own special form of symlinks that aren't compatible with MacOS B: only supports symlinks on NTFS formatted drives C: has no equivelant of a sparsebundle.

    However, there's nothing stopping you from using the windows machine to perform the backup on to an external drive in an arbitrary folder that Avid can't scan, and then hooking up the drive to the MacOS computer where the editing is performed and creating both a sparebundle to act as a mountable volume and then a symlink to act as a conduit/funnel to redirect the media written to/read from that volume to the folder of your choice (in your case the existing folder where the media was backed up to).

    Drawbacks:

    This method will require that you give the drive to the editor using a mac and then perform a bunch of steps on that mac itself to make it work, meaning you can't just give them the drive and say "doubleclick on the .sparsebundle file and it'll work". Either you will have to take the steps on their machine prior to the editor starting, or they'll have to read the steps themselves and perform them, or if you have a spare mac, you can then take an extra step of pluggin the drive in there and preparing everything beforehand.

     

    I hope I've not completely misinterpreted what you're trying to do, but it sounds like you can skip all the complexity and just use The Simple Method.

     

  • Fri, Jan 25 2019 5:51 PM In reply to

    • chuckkahn
    • Top 500 Contributor
    • Joined on Thu, Oct 13 2005
    • Toronto
    • Posts 350
    • Points 3,870

    Re: Anyone Fooling Around with Sparse Disc Bundles?

    I'm backing up to multiple drives -- one backup copy on Bkp1 drive, another on Bkp2 drive, etc -- and I'm finding that the symlinks in the sparsebundles copied from Bkp3 to Bkp1 for example, point to the MXF files on Bkp3 not to Bkp1, even when Bkp3 is unmounted.   Is there a way to make a sparsebundle symlink and duplicate it to another drive without having to re-create the MXF symlink specific to each drive copy?  Or an easy way to edit the symlink so I can substitute the Bkp3 part of the path with Bkp1?   

    Avid Media Composer 2018.11 / Mac OS Sierra / MacPro5,1 / 6-Core Intel Xeon 2.66 GHz x2 / 64 GB 1333 MHz DDR3 ECC [view my complete system specs]
  • Sat, Jan 26 2019 11:25 AM In reply to

    Re: Anyone Fooling Around with Sparse Disc Bundles?

    Ah Ok, it took me a while to understand what you were saying here. This is an unexpected circumstance, but I get it. I'm not aware of a way of doing exactly what you ask though. The way symlinks work in conjunction with the sparsebundle disk images I think requires that an absolute path be specified.

     

    I did manage to make it work in the circumstance where all the drives have the same volume name, in such instance you can unmount the drive(s) you do NOT want the media redirected to leaving only the one that you do. I tested this and it works, you can just follow the usual symlink and sparsebundle instructions once, and then copy the sparsebundle image from drive to another. Obviously though this is far from ideal as it will lead to ambiguity when you rely upon file paths being respected and yet need to send data to one particular drive while they're all mounted simultaneously. I actually tested what would happen if you got MacOS to do something like making a directory at a specific path, where that path leads to a drive that has the same name as another drive that's already mounted. It resolves the ambiguity by simply writing to the drive that mounted first. I think internally the mounted volumes are given numbers, similar to how windows assigns letters. But if you specifically needed the directory to go to the drive that was mounted 2nd, but also needed the drive that was mounted first to remain mounted, then you'd be SOL. One thing you could try (I can't test this on my current machine as it doesn't have Avid), is specifying a drive based on it's position in the finder sidebar, this should hopefully work both for copying data at finder level, and also within Avid itself for things like consolidating. As far as I know, the positions of volumes in the sidebar is based upon the order they mounted, so you hopefully then if you simply named all your backups 'Bkp' then backup 3 will be distinguishable by being the 3rd icon down. This is pretty dodgy though and I wouldn't want to rely on such a system, though I don't have a particular example, it also just screams unintended consequences somwhere else for some unexpected reason and for something as important as data backups, that's best avoided.

     

    I'm not an expert on this but when you presented the problem to me I thought I'd figured out the general direction a solution would take and then began researching it and based on my loose understanding I think it's a dead end. I was aware that symlinks can link 2 directories at given paths, but I was also aware that this could be done relative as well as absolute and I figured this might allow one to create a symlink that will just link to a directory that accessible relative to wherever the link is created. That is how Apps for MacOS are distributed for example, they contain relative symlinks that work on anybody's system who downloads the app. Unfortuantely though it doesn't work for us because in such a case the app is a master folder containing all the subfolders, and as such one can create a symbolic link from one of those subfolders to another because the subfolders will always be in the same place relative to each other, but in our case we're actually having to create a link from a volume to another volume and the volume that it's going to is going to be different every time (in your case the numberal at the end of 'Bkp' so I don't think there's a way to get around that.

     

    In answer to your second question though, of an easy way to edit. I found this open source program that runs as a service on MacOS (make sure you read the install instructions):

    https://github.com/nickzman/symboliclinker/releases

    This will allow to you to create a symbolic link with a right click action from the MacOS contextual menu. It automatically makes the link in the same directory as the file or folder you clicked on and in order to create the link, but you can then move that anywhere you need it to be. Annoyingly, it will always add " symlink" suffix to the symlink that is created and that behaviour can't be changed, so there is an easy, but unnecessary extra step of you having to remove the " symlink" from the link name when you copy the symlink to the Avid MediaFiles folder on the root of the sparsebundle you create.

     

     

     

  • Sun, Jan 27 2019 8:05 PM In reply to

    • chuckkahn
    • Top 500 Contributor
    • Joined on Thu, Oct 13 2005
    • Toronto
    • Posts 350
    • Points 3,870

    Re: Anyone Fooling Around with Sparse Disc Bundles?

    Yes SymbolicLinker lets you quickly create symbolic links, but editing those that it creates?  I saw no way to do so.   If I could just edit the "3" to another number in the symbolic link's path for each MXF I had created, 'twould be much simpler than creating new "MXF symlink" files and moving them all into the matching sparebundle and deleting the non-working ones that are there and renamed the new "MXF symlink" to "MXF".

    Avid Media Composer 2018.11 / Mac OS Sierra / MacPro5,1 / 6-Core Intel Xeon 2.66 GHz x2 / 64 GB 1333 MHz DDR3 ECC [view my complete system specs]
  • Mon, Jan 28 2019 7:32 AM In reply to

    Re: Anyone Fooling Around with Sparse Disc Bundles?

    Oh I should have clarified. I meant, that in answer to your question about editing symbolic links, or at least somehow making the process more convenient: SymbolicLinker will somewhat improve the ease with which you manually create new links in each instance. I don't know of a any way of editing symbolic links after the fact, I think probably because they're designed specifically so that if you for example move the folder or file being linked, it breaks the link, unlike for example MacOS aliases. They're a more simplistic approach and perhaps because of that it wasn't foreseen that people would use them in this way where editing would actually be useful.

    Of course, it's entirely possible that there is a way to do that, or that there is a way to copy a symbolic link that links two folders on different volumes with different names without needing to recreate the link each time, but if so then I guess it is currently outside the limits of my Unix knowledge.

    I pieced together the whole symlinks in sparsebundles as conduits for arbitrary folders technique by synthesising different workarounds and tricks that I was able to glean by searching forums and speaking to other people who'd used Avid for a long time and pieced together different bags of tricks to do things they'd needed to do in the past. Consequently there could be potential efficiencies I'm missing and would spot if I came from a different background.

    I did invest a fair bit of time trying for you though. It seemed to me that it should have been possible to do it and I felt 'relative' symbolic links would be the answer but it seems that what you're doing is inherently absolute because you're referring to something outside of the volume the link will be on and hence will be different every time (unless all the volumes are identically named). If you were copying and distributing a folder heirarchy where everything is contained in the same folder and any symbolic links are referencing other locations within the heirarchy it might have worked but I have found no way to make it work in this specific context.

  • Wed, Jan 30 2019 11:16 AM In reply to

    • derDoyle
    • Not Ranked
    • Joined on Tue, Aug 18 2015
    • Berlin, Germany
    • Posts 89
    • Points 1,045

    Re: Anyone Fooling Around with Sparse Disc Bundles?

    Didn't read the whole thing but here in my company we're using this for years now for our assistant editors.

    Our IT guy made this possible.

    We have a big XSAN-Storage, on this are project-subfolders with all needed subfolders (AvidMedia Files, OMFI MediaFiles, Import, Export, Stuff, Dailies and so on). A little applet creates these per default in always the same manner, you only have to give a project name.

    The applet creates the subfolder-structure and an image. The inside of the image contains shortcuts to these subfolders.

    All via SymLinks as far as I know. The size of the image is "unlimited", so you always have all the space on XSAN available for each image.

    -XSAN

    --Projects

    ---Project A

    ----AvidMedia Files

    ----OMFI MediaFiles

    ----Dailies

    ----Documents

    ----Import

    ----Export

    ---Project B

    ----AvidMedia Files

    ----OMFI MediaFiles

    ----Dailies

    ----Documents

    ----Import

    ----Export

     

    Project folder and Image have basically the same content.

    Image "Project A"

    ----AvidMedia Files

    ----OMFI MediaFiles

    ----Dailies

    ----Documents

    ----Import

    ----Export

     

    Now you can mount the image from every suite which is attached to the XSAN. Parallel working is not possible, but that's ok.

    In Avid you can use the image as a work volume, sometimes the Media Creation jumps back to the XSAN root folder, so you have to check twice. You can rename your MediaFiles folders like on every normal volume.

    Unfortunatly I can't give you in depth information, not because I don't want to, but because I simply can't, this is outside of my understanding ;) But I wanted to say that it's possible. Has its minor flaws, but works so far.

     

    Alternatively for local machines I found this a very handy option:

    https://www.youtube.com/watch?v=mY-RBEK7W8g

     

    Doyle

    Clients: MacPro6.1 (2013 "Trashcan"), 3,5GHz 6-Core, 16GB (1866) RAM, AMD FirePro D500, 256GB SSD, macOS 10.12.2, Avid MediaComposer 8.7.2, NitrisDX... [view my complete system specs]

    Passionate In-House Avid Support

    Admin of the German Avid Community on Facebook

Page 2 of 2 (24 items) < Previous 1 2

© Copyright 2011 Avid Technology, Inc.  Terms of Use |  Privacy Policy |  Site Map |  Find a Reseller