Automation for scraping text from images on iPhone

Recently, I was chatting with a family member looking for a solution to scrape/extract text from a significant set of images. They wanted a solution that was easy to run and didn’t require too much effort. We started looking for some apps that could provide this ability for free (or possibly for a minimal fee). During the search, it suddenly struck me that Apple recently introduced live text extraction from images in iOS 15, a feature I had used when it first came out. I was eager to learn if the “Shortcuts” app (more on this later) exposed that feature as a part of the API, and amazingly it did! At this point, the problem was solvable on the phone for free and with better privacy as it removed the need for uploading the images to a random service/app.

Note: I have not covered the details of using the Shortcuts app to create the proposed shortcut, as it would have resulted in a relatively long article. If you feel this is a topic that I should write about, or create a video on, then let me know on my social (Twitter/LinkedIn).

Picture Credit

“Shortcuts” app & my first usage of it

Shortcuts got introduced to iOS in 2019, which enabled scripting/block coding on apple devices by providing a friendly user interface. One interesting thing that I learned while writing this article is the history of the Shortcuts app. If you are curious, give this wiki a read.

I first used the Shortcuts app to solve my OCD about removing the phone from charging as soon as it was 100% charged, as I read somewhere that this was good practice for having a good battery life (not sure how true this holds today). The idea was to get a notification from the phone as soon as it reached 100% charge. I solved this worry of mine with ease using the Shortcuts app, and it was the time when I got hooked on them for my automation needs on my iPhone or iPad. Since then, I have created many shortcuts to solve my daily problems.

Ok, now let’s go back to the topic.

Problem Definition

Input: A set of images stored on the iPhone.

Output: A file containing the text from all the images. The output file was later fed to a data processor to extract useful information.

Remarks: The solution should be able to append data to an existing file if later it was run on more images.

Solution - Creating a new shortcut

1. Store all the images on iPhone in a new album so that they can be deleted after the extraction is complete.
2. Add a prompt to select the expected images.
3. Call the “Extract text” API on the selected images.
4. Ask the user for the output filename. In this case, I used the “Notes” app to create/extend a note with data.
5. If a note with the provided name didn’t exist, create one.
6. Append the extracted text to the note.

Note: I have not covered the details of using the Shortcuts app to create the proposed shortcut, as it would have resulted in a relatively long article. If you feel this is a topic that I should write about, or create a video on, then let me know on my social (Twitter/LinkedIn).

Image showing the shortcut definition for the described solution

You could download a copy of my shortcut from my iCloud URL.

Things to note

• I haven’t done any performance and quality benchmarks to test the limits of this solution. However, for our problem, we ran the shortcut on a set of 300+ images (sharing a similar layout) with a batch size of 50, and the accuracy of extracting the relevant information was 100%.
• Currently, the Text Extraction API only works for English, Chinese ( simplified and traditional), French, German, Portuguese, Spanish, and Italian.

Concluding thoughts

Shortcuts is a really powerful app to automate tasks on iPhones and iPads. I have used the app extensively to solve many of my daily problems, and seeing its application for this use case was just amazing. It makes me really happy that I could do more on the device apart from the marketed features. I hope this gives you some inspiration to automate your tasks and use your device to the fullest.