Bixby on the Samsung devices is not a virtual assistant, but also an advanced engine powered by AI power and third-party applications to offer additional features for the users. It uses the Camera but takes it one step further and increases its overall functionalities. All of these features are grouped under the name of Bixby Vision. There are several modes in it which are namely Translate, Text, Discover and Accessibility Modes.
In this post, we will give information about what these modes are, how to use them and how accurate the outcomes produced by these modes are in Bixby Vision.
Bixby Vision 101
Bixby Vision is a visual and text recognition technology developed by Samsung Electronics. All the services are Translate, Discover, Text and Accessibility Modes. Shortly, Translate translates a text on paper from a particular source language to the target language; Text scan and detects every single character on paper and allow the Samsung user to do something with it like copy and share; Discover demonstrates the similar or exact pictures of a something on the Internet that is captured by the user; Accessibility Modes are designed to enhance the experience of the Samsung users who need special attention.
The software system is integrated into Samsung mobile devices, however, the services are provided by third parties like translation by Oanda and Google Translate, discover by Pinterest and Syte and so forth. Because of the fact that it is wise to read the partner's privacy policies within the Bixby Vision before using it. Here is how to reach Bixby Vision:
- Open Camera.
- Go to "More".
- Tap "Bixby Vision" from above.
Unless it is enabled from Settings, there is no way to open Bixby Vision by tapping an app from either the home or apps screen. Though searching for "Bixby Vision" in the search box of the apps screen might be a possible remedy, it is better to add it directly to the apps screen as a separate app like any other one.
Here is how to add the Bixby Vision shortcut app to the apps screen:
- In the Bixby Vision, tap three-vertical-dots︙.
- Tap "Settings".
- Under General, enable "Show Bixby Vision on Apps Screen".
- In case want to see it on the home screen, tap and hold "Bixby Vision" on the apps screen and tap "Add to Home Screen".
Translate
As its name suggests, it makes translation from a source language to a target language but they are not limitless. There are 55 source and 104 target languages, that is to say, it can only detect 55 languages and translate them into 104 different target languages including the source language and the others. As it might be expected, converting process does not begin when it is detected that the source and the target language are the same.
How to Use Translate in Bixby Vision
- From the below, choose the source and target language. You may either set the source language to "Auto" or select the language to translate.
- Find some text to translate.
- Once you see the translation on the screen, tap on the screen to freeze in order to read it with ease. Literally, when it is tapped, it demonstrates the converted language on the screen as a picture and it does not disappear from the screen even if you do not point your camera to the text.
- Tap anywhere again on the screen to resume and translate any other text or the same.
- There are several options offered by Samsung to do with the converted and translated text:
- Copy All: It takes all the translated and converted text on the screen and saves it to the clipboard.
- Select Text: It opens the processed text in a plain text format and allows the user to select the whole or particular text within it.
- Save Image: It directly saves the image of what you see while you preview the result on the screen.
How Accurate the Results in Bixby Vision Translate
We put the Translate feature in Bixby Vision to test in order to see how accurate the results are. Before starting, it should be stated that text translation is mainly from Turkish to English and vice versa.
It is better to start with Turkish-to-English and then jump into English-to-Turkish.
It works really well when it comes to short sentences. "Küresel Ekonomi and Türkiye" is translated without any error as "The Global Economy and Turkey", the country's name changed though but it is not a big problem.
However, this is not the case when it comes to other headings and follow-up long texts. For instance, there is a subheading in the book called "Küresel Ekonomide Türkiye'nin Yeri" and a following minor heading named "Dünya Eknomisinin Görünümü", these can be translated as "Turkey's Place in the Global/World Economy" and "The Overview of Global/World Economy" respectively. Plus, a 6 lines paragraph follows these headings which starts with "2018 yılı itibarıyla dünya, cari fiyatlarla toplam olarak 84,7 trilyon dollar, satın alma gücü paritesine göre 13,5 trilyon dolar GSYH'e sahiptir". According to this relatively exact translation should be as follows:
Even though the result is not completely wrong, it still does not offer a reasonable outcome that can satisfy any single human being because it gets the first two or three words of the long texts and combine them with the subheading and minor heading which results in a useless translation.
For the ones who are interested in the result:
When it comes to English-to-Turkish translation, things go well. There are only minor mistakes and it translates in the context, not in a literal way.
So, the source texts say:
Translated one is:
Some particular accuracy problems can be understood by even non-native speakers like "the", unnecessary capital cases and so forth. There will always be the ones who can translate the text in a better way, but the outcome of AI software is quite sufficient and nearly close to perfect.
There are some shortcomings of the test like there is not enough comparison in any other languages. However, we assume that neither you have enough time to read a long text nor we have the knowledge in any other languages as we have in Turkish. Text is converted by Oanda or Google Translate, so we are hundred per cent sure in some languages these translation systems are better than the remaining ones. For that reason, this test is just a demonstration of what possible shortcomings you can find while you translate a text from a source language to a target language via Bixby Vision.
Text
It scans texts on paper or anywhere else, detects the majority of the characters and lets Samsung users copy and paste them somewhere else on their mobile devices without even writing every single character on their own. In this specific case, it is unnecessary to put it to the test because it really does a good job of what it is designed to do, except for some special characters like "•".
It goes without saying, sometimes it cannot determine line breaks and put a dash like nat-ions and etc but it really does not matter a lot. Since every single language is unique, there might be some characters it cannot determine as well.
What about handwriting? Unlike typing, things are not that well in handwriting because it occasionally detects the written handwriting error-free, but not always. Our handwriting might be the reason as well but we tried our best to make it as error-free as possible.
Some of the examples are:
How to Copy & Paste Text from Paper to Digital in Bixby Vision
- Point the camera to a text either on paper or somewhere else.
- Give a few minutes for the application to determine typing and handwriting and turn them into digital characters.
- Tap the highlighted words or enlarge the text cursor's area to select more than one.
- Choose what you want to do with the selected typing or handwriting from the toolbar:
- Copy: It copies the selected word or words to the clipboard.
- Translate: It translates the selected text from one language into another one with the help of Google Translate.
- Select All: There is no need to mention, it selects all the highlighted text on the screen.
- Search: It takes the selected text as a prompt or search question and searches it in search engines, generally Google Chrome.
- Picture: It only appears on the screen next to three-vertical-dots ︙ after you scan a particular text which shortly allows the user to select a picture or screenshot from the Gallery and do the same action on a digital screen rather than using the conventional methods.
Discover
It shows slightly relevant and similar photos of the object or anything else you point the camera at. It is not powered by Google itself, but by Pinterest which is also known as the search engine for pictures. Similar objects will be listed on the black panel below and tapping on any single picture directs the user to the Pinterest page of the photo. So with that feature, it is possible to learn the name of the object or get more information about it.
Once an object is captured, discovering the objects on the Internet by using the picture or screenshots in the Gallery feature is enabled. As a basis, it allows Samsung users to select a visual element from the Gallery instead of taking a picture of it. Here is how to use the search with pictures in Gallery:
- In order to enable the feature, capture an object.
- Tap the "Picture" symbol next to the three-vertical-dots︙ from above.
- Choose either Samsung Gallery or Google Photos.
- Select the picture and wait for the system to search for similar pictures in Pinterest's database and show you.
Apart from Pinterest, Samsung has a partnership with Syte which is like the commercial version of Pinterest. For example, it shows a similar or exact picture of a t-shirt and allows you to buy it from a particular website on the Internet. Keep in mind that, they are not sold by Samsung Electronics Co,. Ltd, so it is better to read the Legal Notice of Syte and check the credibility of the commercial website.
Accessibility Modes
Bixby Vision is not for the people whose all body works as expected, but at the same time for the ones who are visually impaired. There are four different modes in the accessibility modes which are Scene Describer, Text Reader, Object Identifier and Color Detector. In short, Scene Describer scans the environment where the camera point and tells what there is in it, Text Reader: directly reads out loud a text on paper or digital screen, Object Identifier: says the name of the object that is pointed with the camera, and finally Color Detector: as the name suggests, detects the color and read it out.
There are additional features in the former three options of Bixby Vision like "Choosing a picture from the Gallery" but it is not available for Accessibility Modes.
Here is how to enable Accessibility Modes in Bixby Vision.
- In the Bixby Vision, tap three-vertical-dots︙.
- Tap "Settings".
- Tap "Accessibility Modes" to enlarge the screen.
- Tap the toggle to enable it and choose and activate the modes (Screne Describer, Object Identifier, Text Reader, and Color Detector) from the below as well.
Scene Describer
Once you tap "Capture" below by pointing the camera at a scene, it describes the screen captured and reads it out. Any other accessibility modes do not require tapping "Capture" to prompt the system, but it is not the case for "Scene Describer". Others are detected automatically without the need for tapping "Capture", but they can be still enabled from the Settings:
- Tap three-vertical-dots ︙.
- Tap Settings.
- Under General, switch on the button next to the "Show Capture Button" to enable the "Capture" button for all the other accessibility modes.
The description generally varies. For instance, sometimes it says "a screenshot of a computer", sometimes "a computer screen with text on it" but it is just an explanation of the same screen with divergent combinations of words. The captured screen is not saved to the Gallery unless the "Download" button is pressed from above.
Object Identifier
Fundamentally, it identifies the object that you point the camera at. It suits perfectly the case when you see an object that you do not know the name of, otherwise, it only says what you are already aware of.
Sometimes it detects more than one object, and sometimes cannot detect the second object though there is one in the shooting angle. Accuracy is just tailor-made, at least for the objects we put the Object Identifier to test.
Text Reader
As the name suggests, it scans texts in a book or on a digital screen and reads them out. However, there are some shortcomings of it like the voice is only pronounced in the following languages English (US & UK), French (France), German (Germany), Italian (Italy), Portuguese (Brazil), Russian (Russia), Spanish (Mexico & Spain), any other text in a different language will be pronounced with one of the determined language's pronunciation in a weird way.
It directly takes the data from the system language and set it as the system default voice and sound, in order to change the language for each kind of different language, follow the steps below:
- Open Bixby Vision. At this stage activate "Accessibility Mode" if it is not enabled, if it is, pass to the following steps.
- Tap three-vertical-dots ︙.
- Tap Settings.
- Tap Accessibility Modes.
- Tap "Text-to-Speech Settings".
- Tap "Language".
- Select the language to pronounce the text in that specific language and change the language again and again because otherwise it is either not read or read in a weird way.
Another shortcoming is that it cannot automatically detect the language in Text Reader. For that reason, the language should be adjusted by following the steps above. Otherwise, either only detectable text will be read out but in a weird accent or all the text will be ignored.
How Accurate the Results for Text Reader?
As we did it in Translate, we decided to put the text reader to the test to see how accurate the results were.
Starting with, it nearly instantly detects and scans all the text on a particular page of the book and starts to read it out loud. However, as it was the case for Speech-to-Text of Voice Recorder on Samsung, there are some errors: it starts from above which is correct but starts from the second line of the paragraph by skipping the first line and once the second line comes to an end it turns to the first line. In our and others Samsunger's opinion, the speed is equally important but when there are some major errors, it makes it useless to some extent. Therefore, it is good for Samsung to balance these two and provide the best results.
Nevertheless, it is good to state that there will be some cases when it reads out the text error-free. We put it to the test in the English language, in other languages the results may vary. Long story short, the result is mainly based on how well the screen is captured, the space lines between each line of the paragraph and so forth.
Color Detector
The last mode of the Accessibility Modes in Bixby Vision is the Color Detector and the only function of it is to read out the color of the surface that you are pointing the camera at. It is very good at what it does, even if it classifies the colors according to the color tones.
Settings for Accessibility Modes
Accessibility mode offers additional settings for the modes that use text-to-speech outputs which means for all modes.
In order to see the settings, follow the steps below:
- Open Bixby Vision either from the apps screen or the Camera app.
- Tap three-vertical-dots︙ and select Settings.
- Tap Accessibility Mode and after that tap "Text-to-Speech Setting".
- Here are a few options to make it more unique:
- Preferred Engine: It allows Samsung users to change the preferred engine for the text-to-speech modes, but there is only one option for particular countries which is "Samsung Text-to-Speech Engine". The "Gear" symbol next to it gives information about the Samsung TTS (Text-to-Speech) Version and allows the users to change the language and voice which exactly offers 9 different options.
- Language: It is like the "Gear" symbol in Preferred Engine which basically allows the user to change the language for text detection and pronunciation.
- Speech Rate: It allows the user to set how fast and slow the engine will read out the text.
- Pitch: It has the same function as Speech Rate but is designed particularly for how low and high the pitch is to be.
- Play & Reset: When "Play" is tapped, it shows how you will hear the voice, so basically to control the settings and we think there is no need to mention what happens when "Reset" is touched.
Conclusion
Bixby Vision is powered by AI systems and other third parties to save the day of Samsung users with its unique opportunities. It directly translates one language into another one, demonstrates similar photos of an object you captured, allows the users to copy and paste text from paper into the digital world and offers some accessibility modes for the ones who need special attention. Even though the system works instantly, the accuracy is a big question because it does not successful at %100 per cent or even 80% per cent. That's why it needs a balance between speed and accuracy to persuade Samsung users to continue using it in their daily lives.
If you want assistance in Bixby Vision on Samsung Galaxy phones and tablets, kindly leave a comment below. If we have an understanding of the subject, we could give some of the answers to your question. Don't worry; in other cases, we can provide you access to particular articles published by members of the community that you can read and use to examine the problem you're attempting to solve.
Comments
Post a Comment