managing meta data questions

Started by TheBadger, March 20, 2014, 12:37:33 AM

Previous topic - Next topic

TheBadger

Hey,

Not sure if these questions are OS specific or software specific. But I was curious about managing meta data.

On OSX I can hit "get info" of any file. And then I can see some info about that file:
[attach=1]

As you can see there is some useful info, a place to make notes and a lock icon. However in this case Im not sure what locking prevents anyone else from doing if I were to give that file to someone. I think it only prevents changes to the data shown in the image, but not to the file once it is opened in a compatible software, or to other data on the file not shown in the "get info" window. Does that sound right?
For raw camera files you can see camera data like in adobe bridge, but not as much.

Here is what you can see in Bridge:
[attach=2]
All of those tabs open to much more info. Mostly I don't have any files that have meta to go in those areas. For example there are spots for medical information and other things I don't deal with.


1) What I am wondering is, is there a easy way to know all of the meta that a file contains and to edit that data, regardless of the file type? OR at least of common file extensions?

2) If you open a file, then make changes, then save out a new file with the same or different file extension, does the old data remain in addition to the new meta? And can you decide if it remains or not? Or would this be part of notes added to a file (would I have to add the old notes to the new file in addition to new notes)? And in the case of bridge just for example, are those notes now part of the file or are they kept in bridge?

3) workflow exchange path; Can I know or make known by meta (not notes) that a file was first .obj, then .fbx, then .max, then .fbx, then obj?
Or does / can meta contain software routs; first hexagon, then maya, then, then... And so on?
OR can this only be known by making notes in a program like bridge?

4) is there any soft any of you can recommend, that allows one to see the full meta that a file contains, all of it? For example is the info I get in bridge all that there could be? So Im fine with Bridge, or is there more info then bridge is interested to show me?

What is really in a file? It seems like its a lot more than I was aware of before. And anyway, I would just like to start managing it, adding notes for my self and generally be better organized and more knowledgeable. And of course be security and privacy smart!

Lastly, I wanted to ask if a file can only be locked in the soft that creates it? So If I make an obj, than import that and make some other file kind, that file can only be locked from changes being made to it in the soft that created it?

Im on OSX 10.9.2 if it helps you to tell me anything specific to my OS or the soft I can/should use.

Thanks in advance for any teaching you feel like doing.




It has been eaten.

TheBadger

One other thing.

I often change .png to .jpg simply by clicking on the file and typing .jpg where .png was. On my os the file is converted then. But Im not sure what happens to the meta, or if the change is "real", like re saving in photoshop. Also, I sometimes will change the name of a file the same way. And Im not sure that is a good idea, though it seems to work. But again not sure what really happens to the file...

I need to start using a better naming convention too. I have a ton of files named "final" in different folders.
It has been eaten.

Dune

I think there's software that reveals metadata and can also change or add to it, came across it once, but a simple google search will reveal a lot. And regarding naming convention; I think it's good to add a date to every file. Gives it a unique number and some insight (a lot of insight) in its age. Though that's in the metadata of the file also  :P

TheBadger

QuoteThough that's in the metadata of the file also  :P

Yes indeed. But still a good idea to add the date. though for me personally Im not good at remembering dates. They don't hold the context that I remember things by the way they do for others I guess.

I wish the "get info" thing was much better. Running bridge requires starting and waiting then navigating. But "get info" is super fast. Going back into all my files and starting a good managing system for my self has made me really aware of how much time it takes to do anything. (should have done it in the first place :-[)

And of course I will google. But I was looking for some personal recommendations. There are now thousands of apps for anything you want. But no time to test them all, and I don't trust the reviews on line. But I give a lot of weight to what you guys say :)
It has been eaten.

Oshyan

Oh boy, this is a big, big topic Badger. I mean it's fundamental operating system and file system usage stuff, you know? Not to say that you should already know it just because you use a computer - far from it, most people don't! I unfortunately don't have time to delve into things too deeply (ok, I lied a little, see below...), but I'll give you a few pieces of info that will hopefully help narrow your search terms and find more relevant info.

First and foremost, unless OS X is doing something magical that I'm not aware of, changing the file extension a file is *absolutely NOT* changing the actual format. The reason it probably still works as you expect is because OS X doesn't actually give a crap about file extensions at all, it uses meta data (or some other mechanism?) to know what the file type is. So changing the extension is really just changing the name as far as OS X is concerned. Further to that even if you put the file (a ".jpg" renamed from ".png") on a Windows machine it would probably still be viewable but this is *only* because any decent image viewer will be able to read both formats and while it may load expecting one, it would likely be able to automatically determine the correct format and read it. However if a viewer were attempting to *only* load data in the file type specified by the extension it would *not* work, and certainly you do not get any of the actual benefits or qualities of a given file type by doing this, e.g. reducing file size with JPEG compression by simply renaming a .PNG. Saving to JPG requires a process, it completely reprocesses and rewrites the image data. Unless, as I said, OS X is doing something crazy, which frankly would in my opinion be ill advised for a number of reasons.

Now, regarding meta data, there are at least 2 major types, and probably more. Determining the difference in a given file is unfortunately more difficult than it should be. In OS X in particular (but also now to a lesser extent on Windows) there is OS/file system meta data that is *unique to the operating/file system*, i.e. it is not embedded in the file and is likely to only be readable by that specific OS/file system. OS X stores lots of data this way, including image thumbnails, notes, etc. In any situation where you have the exact same meta data options available for *any* random file, no matter the file type, this is probably OS-level meta data, not file-level meta data. This data shows up when you transfer a file to a PC and you get those "OS X _" hidden folders and stuff.

The other major type of meta data *is* more portable, it is generally intrinsic to the file (or stored in a "sidecar" file, sort of like the .MTL that goes along with a .OBJ), and for that reason it is also often more specific to the needs of that file type (e.g. image-specific meta data for image formats). Furthermore because it is file-specific and file-*type*-specific, not every file type will have such data. Meta data is defined on a per-file-type (format) basis, generally speaking. So for example with image formats there is Exif: http://en.wikipedia.org/wiki/Exchangeable_image_file_format
But there is *also* IPTC, which is actually quite old (from the 1970s), and now generally being replaced by XML lead by Adobe.
http://graphicssoft.about.com/od/glossary/f/metadata.htm
The thing to understand about this kind of meta data is there are two parts to it. There is the data that the file format itself is specified to be able to handle, the "standard", the agreed upon set of information that anyone and any supporting program will be *able* to read/write in a given format. File formats are not always well documented or specified, or even particularly compatible between programs - consider .OBJ for a good example. But in the best cases, for something broadly standardized like JPG for example, there is an organizing body of some kind that defines what data the file format can store.

The *other* part to file-level meta data though is the *program support*. Meta data in any given file format is generally defined in such a way that it is *optional* to support it. You could have a JPG reader that does not have any support for viewing - much less editing - of meta data. It could be totally unaware of Exif, IPTC, and the rest, yet still be able to load the core JPG image data and display the image. Similarly with Exif to IPTC to XML, there are imaging programs that *only* support Exif, ones that support both Exif and IPTC or Exif and XML, etc.

So to get meta data into a file, you need a file format that supports it, and a program that supports editing that data. Storing meta data in a file is generally best because then that data is entirely portable, i.e. it is not tied to a particular operating or file system, and if you send the file to someone else, the data remains intact. If either piece is missing - support for meta data in the file format, or support in your application for that meta data - then you can resort to operating system meta data, but remember that it is not portable and is likely to be more general (i.e. lacking fields specific to a given format's capabilities, like geometry count for an OBJ for example) as well as more limited (fewer fields).

Regarding saving of info like what programs a file has gone through, this is unlikely and not something I'm aware of being broadly available or supported, although some formats - e.g. Microsoft Word - do have some history/revision data, etc. in them. The key think to understand there is that it's not *just* the file format that needs to support such data, but the programs themselves need to *modify* the data. In other words let's say you have a file format that does have a meta data field for recording revisions and the programs that made them, and you have 3 programs that support loading, modifying, and saving that format. As I said before, a program can support reading of the basic format without supporting the meta data, or it can support some but not all the meta data possible in that file. So let's say Program 1 can load the file and it writes in the meta data "File X modified by Program 1 on XXXX date". Ok, great. But Program 2 doesn't support any meta data at all. You load the file and make changes, then save it, but it has no idea about meta data. Program 3 does support some of the meta data, but it's a bit older and doesn't support that specific meta data field! This is a real possibility, though if a file specification is designed correctly and the program is designed correctly, it should not be likely to happen. But these are just some of the complications you need to be aware of.

The bottom line is that meta data is very useful, but between the issues of portability and program support, it often cannot be relied upon outside of your own computer, unfortunately. Meta data support is currently best in image formats for various reasons, Exif, IPTC, and XML meta data are powerful and broadly supported now. I imagine 3D formats like FBX, Alembic, and Collada have their own meta data that is hopefully increasingly supported.

I could give specific recommendations of software that supports meta data read/modify/write, but it would be largely confined to the image processing domain. The key take-away though is that file-specific meta data relies on support in specific applications, it is not as likely to be supported by your operating system. This doesn't sound so bad if you can find programs that support the meta data you want to read/write, except remember too that a "chain of meta data" is only as strong as its weakest link. If you're lucky a load/save in a program that doesn't support the meta data will just result in unmodified meta data. In the worst case however the data will be entirely stripped/lost.

It's a complicated problem.

- Oshyan

TheBadger

#5
Oh man, thats a great post Oshyan! Thank you.
And yes indeed that provides a very good place for me to start from.

There is a lot in your post to go through. It will take me a while to decide what soft or managing convention I will be able to make use of.
But there is at least one thing that I can ask about right now.

QuoteFirst and foremost, unless OS X is doing something magical that I'm not aware of, changing the file extension a file is *absolutely NOT* changing the actual format. The reason it probably still works as you expect is because OS X doesn't actually give a crap about file extensions at all, it uses meta data (or some other mechanism?) to know what the file type is. So changing the extension is really just changing the name as far as OS X is concerned. Further to that even if you put the file (a ".jpg" renamed from ".png") on a Windows machine it would probably still be viewable but this is *only* because any decent image viewer will be able to read both formats and while it may load expecting one, it would likely be able to automatically determine the correct format and read it. However if a viewer were attempting to *only* load data in the file type specified by the extension it would *not* work, and certainly you do not get any of the actual benefits or qualities of a given file type by doing this, e.g. reducing file size with JPEG compression by simply renaming a .PNG. Saving to JPG requires a process, it completely reprocesses and rewrites the image data. Unless, as I said, OS X is doing something crazy, which frankly would in my opinion be ill advised for a number of reasons

Oshyan, please download one of the OP images above and try to look at its guts.
When I make an image capture my mac produces a .png. And I open the image in "preview" then crop it and save. (there is no normal "save as" function in preview). After that I click on the name and change the extension to .jpg.

Im not sure, but I think I may have started doing that because I thought that it reduced the file size. I don't remember how I got in the habit. Maybe it was just because .jpg is easy for everyone to deal with.

Anyway, knowing what it looks like off my system will go a long way to clarifying what you were saying. And I am aware of the mac hidden files that get created. What I am not sure of is if they are also created when I save from non apple software. For example if I sent you an .obj, would those hidden file be present? I think they used to be before lion but now?

Its funny, like you said people just let the OS do what it does and don't think about it. Thats one of the reasons to have a mac (less need to learn IT) But thats also the problem. You use a mac and its easy you stop learning as things come, so you can end up playing catch up when issues do arise.

But yes, please look at the image above and tell me how it appears to you. and what if any data you can see about me from looking at it.

I will probably pick through your post over time as I move through the information and find software to play with that works on my OS. And yes you are clearly right about this being a huge topic. But from my perspective any upward learning is better than where I am now. I have long neglected fundamental IT stuff because I thought I would never need it (someone else's job). But that thinking has just left me vulnerable to wrong information and untrue facts. And who knows what other security threats :-\

Thanks again.
It has been eaten.

Oshyan

Quote from: TheBadger on March 21, 2014, 03:11:50 PM
Its funny, like you said people just let the OS do what it does and don't think about it. Thats one of the reasons to have a mac (less need to learn IT) But thats also the problem. You use a mac and its easy you stop learning as things come, so you can end up playing catch up when issues do arise.

This. Exactly this. Yes, Macs are in some ways potentially easier if you never plan to do anything advanced with your computer. The problem is that you can never predict if or when that will happen, and for whom. My view is it's better to learn about how things work, at least in a basic way, from the start and as you go, rather than hit a wall and come to find out you've been missing all this potentially useful or important stuff for years.

Regarding your images above, yes they're PNGs. Using the first as an example, I can see this in 2 ways. First, load in my default image viewer XnView (XnView MP is a new version also available for Mac if you're interested, great viewer and simple editor/format converter), then look at Properties and it tells me it's PNG format. There does not appear to be any Exif or IPTC data, but in the comment field is this:
"XML:com.adobe.xmp: <x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 5.4.0">
   <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
      <rdf:Description rdf:about=""
            xmlns:exif="http://ns.adobe.com/exif/1.0/">
         <exif:PixelXDimension>514</exif:PixelXDimension>
         <exif:PixelYDimension>1020</exif:PixelYDimension>
      </rdf:Description>
   </rdf:RDF>
</x:xmpmeta>
"



Opening the image in a text editor, this is in the header:
‰PNG
or "data:image/png;base64" if I view it in my handy Chrome image info plugin.


- Oshyan

TheBadger

Well thats just ridiculous  :o !

Regardless of what you said about mac not caring. What does apple think, that mac users don't post images or talk to windows users?
I mean, even if the mac internally deals with things differently than a PC, why allow me to change the file extension in the way I described, if it does not acually change the file? Its misleading.

And what you were saying about readers being flexible rings really true to me now. Because this forum took the upload as a JPG, despite that its really a PNG as you have shown.
And I suppose thats a good thing, so regardless of some issues, people can still post without having the problems of strict file format readers.
But that also only reinforces the lack of understanding on this topic, because it helps to hide the true facts.

I don't even feel bad about not knowing much about this. I think I am well in the mainstream on this subject (in terms of lack of knowledge)

I think this is one of those things that only first seems unimportant and boring. But really is just fundamental like you said in your first sentences.

I just think it strange

But thanks for teaching! And also for proving your facts!
It has been eaten.

Oshyan

You can change file extensions on Windows in the same way (although it *does* warn you on Windows). The reason Mac doesn't is because it doesn't use file extensions, they're considered probably one of several things: messy, confusing, or potentially misleading/inaccurate. The first two are, in my opinion, stupid reasons. The latter would be because you can, as you've found, simply change the extension but it doesn't change the file type, so could be confusing, but to my mind the answer to that is not to make file extensions irrelevant, but rather to do what Windows does and teach and warn you when you're making possibly unintended changes.

File extensions are useful, especially for cross-platform format support, i.e. if a file format doesn't have built-in type meta data (or that data isn't recognized), and if the OS-level meta data is not cross-platform (most isn't), then how does an OS know what type of file it should expect? This is especially relevant for damaged files where some data may be readable if you make assumptions about what file type it is, but the header may not be intact to tell the software/OS what to look for, whereas a file extension is almost always intact. Anyway, Apple's stance against file extensions is just a bit less stupid than its long-time single button mouse philosophy IMO. ;)

I agree that it's good that systems like the forum here don't disallow you from uploading a ".jpg" that is really a PNG format. But I *do* think there should be warnings/messages whenever this happens, no matter the OS/system. Something like "Note: the file you uploaded had a .jpg extension but was in PNG format. This has been handled internally and this notice is purely for your information." or something. Like, a non-threatening, non-error message just to inform people. I think that would be good and operating systems could use more of that in my view, at least optionally. Few people are going to go through the "introduction to your operating system" type things that some OSs come with, but teaching as you use the system is good I think, especially if it's unobtrusive and genuinely useful. I think in particular providing gentle hints and info when someone dose something that seems potentially "bad" is a good idea. Like how Windows warns if you try to change file extension. Hell even I appreciate that and I'm a power user, it's annoying when I really *do* want to change the extension, but just as often I am working quickly and accidentally select the whole file name (including extension) and then I might accidentally overwrite the extension. There's always undo, though many people don't know you can do that in the file manager just like you can in say Word or whatever, hehe (Undo is pretty universal and super, super useful!).

Anyway, I could go on for ages but I won't. ;)

- Oshyan