Last week we shared a number of updates with our community of users, and now we want to share them here: At Mozilla, we work hard to make Firefox the best
I do agree with your point about auto-generated captions being better than no captions. But isn’t it bad to insert them automatically on creation? If we use these models to caption images shouldn’t it be done by the screen reader instead? That way people can benefit from future advancements of the tech and customize the captioning system for themselves. With the current system there is no way to tell if you got a crappy AI caption that you may want to replace with a better auto-generated caption or a human written caption.
So - I don’t think Firefox would be generating captions for PDFs on PDF creation.
But of the major ways that PDF’s do get created - converted from text editors or design software, I know that Microsoft Word automatically suggests captions when the document creator adds an image (but does not automatically apply captions), and I believe that some design software does, as well.
I think that, functionally, both suggesting captions at time of document creation, or at time of document read are prone to the same issues - that the software may not be smart enough to properly identify the object, and if it is, that it is not necessarily smart enough to explain it in context.
By way of example, a screenshot of a computer program will have the automatic suggestion of “A graphical user interface” (or similar), but depending on the context and usage, it could be “A virus installer disguised as ___ video game installer.” Or “The ___ video game installer.” Between the document creator and the creation software or screen reader, only the document creator would really know the context for the image.
Which is all to say that I think that Mozilla has the right idea with auto-tagging, but it will always fail on context. The only way to actually address the issue is to deal with it within the document creation software.
But I wouldn’t be opposed to ML on those that can auto-suggest things or even critique how content authors write their descriptions.
So - I don’t think Firefox would be generating captions for PDFs on PDF creation.
I’m not sure. The blog post is not entirely clear on that.
Between the document creator and the creation software or screen reader, only the document creator would really know the context for the image.
Agreed. Context is usually very important for images. But with an auto-generated caption embedded in the document itself, you already lose some context. Because if the automatic caption is incorrectly stored as “The ___ video game installer” you cannot decide anymore if this was written by the author with the context in mind or just generated. Which I would argue is worse than no caption, as it lowers your trust in all captions.
But I wouldn’t be opposed to ML on those that can auto-suggest things or even critique how content authors write their descriptions.
Absolutely, I think that will be by far the best solution. It could massively encourage users to write their own captions if in most cases you only need to accept the suggestion. But so far, that seems unlikely to be the way forward. Why do that when you can just throw even more “AI” at the problem?
I do agree with your point about auto-generated captions being better than no captions. But isn’t it bad to insert them automatically on creation? If we use these models to caption images shouldn’t it be done by the screen reader instead? That way people can benefit from future advancements of the tech and customize the captioning system for themselves. With the current system there is no way to tell if you got a crappy AI caption that you may want to replace with a better auto-generated caption or a human written caption.
So - I don’t think Firefox would be generating captions for PDFs on PDF creation.
But of the major ways that PDF’s do get created - converted from text editors or design software, I know that Microsoft Word automatically suggests captions when the document creator adds an image (but does not automatically apply captions), and I believe that some design software does, as well.
I think that, functionally, both suggesting captions at time of document creation, or at time of document read are prone to the same issues - that the software may not be smart enough to properly identify the object, and if it is, that it is not necessarily smart enough to explain it in context.
By way of example, a screenshot of a computer program will have the automatic suggestion of “A graphical user interface” (or similar), but depending on the context and usage, it could be “A virus installer disguised as ___ video game installer.” Or “The ___ video game installer.” Between the document creator and the creation software or screen reader, only the document creator would really know the context for the image.
Which is all to say that I think that Mozilla has the right idea with auto-tagging, but it will always fail on context. The only way to actually address the issue is to deal with it within the document creation software.
But I wouldn’t be opposed to ML on those that can auto-suggest things or even critique how content authors write their descriptions.
I’m not sure. The blog post is not entirely clear on that.
Agreed. Context is usually very important for images. But with an auto-generated caption embedded in the document itself, you already lose some context. Because if the automatic caption is incorrectly stored as “The ___ video game installer” you cannot decide anymore if this was written by the author with the context in mind or just generated. Which I would argue is worse than no caption, as it lowers your trust in all captions.
Absolutely, I think that will be by far the best solution. It could massively encourage users to write their own captions if in most cases you only need to accept the suggestion. But so far, that seems unlikely to be the way forward. Why do that when you can just throw even more “AI” at the problem?
Totally agree, that would be even better.