Software Development

Augmented Reality (AR) is becoming increasingly popular in contemporary times. The credit goes mainly to its ability to bridge the gap between the physical real world and its digital counterpart. AR connects and places virtual content in the real world. It has found implementation in diverse industry segments such as education, e-commerce, entertainment, gaming, and more.

This article will discover the utilization of Augmented Reality in Flutter, which is a popularly used framework for creating cross-platform mobile applications.

[A third-party plugin is employed to integrate AR into Flutter, as the Flutter engine doesn’t directly support AR.]

What is Augmented Reality?

Augmented Reality (AR) integrates digital information in real time with the external physical world. This info could be in the form of videos, imagery, sounds, or other kinds of representational data. It enhances the user’s awareness of the real world by overlaying digital data onto their physical surroundings.

AR is starkly different from Virtual Reality (VR) as VR provides an immersive experience to the user with simulations whereas AR overlaps virtual elements over the real world. 

Applications in various industries:

AR finds its applications in a variety of industries including gaming, entertainment, education, etc. It majorly transforms the way a user interacts with data and its gathered information, products, services, and resulting experiences. 

For instance, in the gaming and entertainment industry, AR has come up with many interesting and interactive elements that create a cohesive utilization in the real world which leads to immersive and highly engaging experiences for the user. Games like Pokémon GO have apprehended the depth of imagination of loads and bring forth virtual characters in the physical world.

In the field of education, VR has found its implementation mainly to augment learning experiences. Interactive 3D models, virtual tours, history-based restorations, and many more such activities let the students explore complicated subjects with ease. Even the medical community utilizes AR to generate visualizations of difficult surgeries and are able to perform practices. 

The architecture and Design community also benefits greatly from AR as they can compare the real designs with virtual blueprints in real time and can gauge the final impact. The retail and e-commerce industry has been able to flourish more as they are able to provide their prospective customers with visualized products in the real environment. This helps the customer make an informed decision before finalizing a purchase. Now one can even try virtual clothing on themselves as well as place virtual furniture or décor pieces in their space. AR has transformed the way we shop in real-time.

AR has also empowered today’s home improvement and interior design apps, enabling users to visualize complete space settings with paints, furniture assembly, and décor. Users can customize settings with design elements to aid in decision-making, making AR an essential tool.

Advantages of using Flutter for AR

Flutter is Google’s open-source UI framework. It has quickly become popular among the developer community due to its convenience, efficiency, and flexibility. Flutter has become an obvious choice for integrating AR into mobile apps, owing to its numerous benefits.

Cross-Platform Consistency: It generates immersive AR experiences that run effortlessly on both iOS and Android platforms from a single codebase, thereby saving time, effort, and resources.

Fast and Hot Reload: It accelerates AR development with real-time code changes and also enables quick modifications and testing.

Rich Widget Library: It employs pre-designed widgets for gesture interactions, animations, and graphics to seamlessly integrate AR functionalities.

Openness to Third-Party Libraries: It integrates ARKit and ARCore using plugins like the “ar_flutter_plugin,” providing access to the latest AR capabilities.

Access to Native Features: It enhances AR experiences by utilizing device features such as cameras, sensors, and location services.

Strong Community Support: It greatly benefits from an active and collaborative Flutter community that provides support, guidance, and solutions throughout your AR development journey.

Choosing ar_flutter_plugin

To build a Flutter Augmented Reality (AR) app, we browsed through various plugins available on pub.dev. Our goal was to detect a solution that restructures AR implementation while making sure that it results in a perfect experience across both iOS and Android platforms.

One plugin especially stands out amongst others, which is extremely promising in its implementation- ar_flutter_plugin. This is an excellent option because of the many advantages it offers. Let us understand why it has become a prioritized choice for developers and how exactly it boosts the AR execution in real-time.

Benefits of ar_flutter_plugin:

Harmony across Platforms: ar_flutter_plugin has achieved a milestone with its inherent compatibility quotient with both ARKit on iOS and ARCore on Android. This establishes a balanced foundation for coexistence and ensures your AR experiences transcend platform boundaries, facilitating an integrated and comprehensive user experience.

Community-Powered Excellence: Community-driven development synergizes with ar_flutter_plugin, as it heavily relies on community input. Its strength is derived from the deep pool of knowledge contributed by its creators. This collaboration and resulting synergy lead to seamless plugin evolution with improvements and updates, fostering thorough AR integration.

Redefined Integration: Through ar_flutter_plugin, the intricate framework of AR integration within your Flutter app becomes seamless. The plugin’s architecture is highly detailed, enabling the creation of an immersive user experience with technical precision.

Ecological Symbiosis: By implementing ar_flutter_plugin, developers can integrate with the broader Flutter ecosystem. This synergy ensures a seamless connection with other Flutter libraries, ensuring that AR-powered creations stand out for their exceptional efficiency.

Empowering 3D Possibilities: This plugin particularly showcases its strength with 3D models, molded into formats like GLB and GLTF, which serve as the foundation for elevated AR interactions. Continuous integration leads to immersive experiences, transforming how the virtual world is perceived in reality.

A Glimpse of Tomorrow: AR technology is in a constant state of transformation, and ar_flutter_plugin keeps you prepared for the future. It offers limitless possibilities with its cutting-edge features and upcoming refinements.

Prerequisites for Flutter Augmented Reality (AR) app

Initially, prepare 3D models in a format compatible with the plugin.

The most commonly utilized formats include GLB and GLTF. You can explore downloadable options from platforms like sketchfab.com, turbosquid.com, and free3d.com.

Also read: Creating Live Chat Functionality in Flutter using PubNub

Considerations and workarounds

Some GLTF objects may face execution issues on surfaces, acknowledged through this open plugin issue.

By replacing GLTF with GLB objects and storing them in the app’s document directory, this obstruction can be dodged in a practical way.

It is to be noted that the plugin does not offer any support for face/head detection currently. It does not even provide tools for measuring between anchor points on a surface. Plugin also does not place the 3D object automatically on the surface when it detects the surface. We need to show it manually on callback of surface detection.

Steps to integrate Augmented Reality in Flutter App

Creating a new Flutter project

• Create a new Flutter project by running the following command or from any IDE.

flutter create my_app

Adding dependencies

• Add the following dependency to the pubspec.yaml with the latest version.

ar_flutter_plugin: ^0.7.3vector_math: ^2.1.4path_provider: ^2.0.12

• Declare permission in AndroidManifest.xml
• Set android:required=”true” if AR is the primary feature of your app. 

<uses-permission android:name="android.permission.CAMERA" /><uses-feature   android:name="android.hardware.camera.ar"   android:required="true" />

• If you have added the permission_handler plugin OR facing any permission issues on iOS, add the following block in Podfile.

post_install do |installer|     installer.pods_project.targets.each do |target|       flutter_additional_ios_build_settings(target)       target.build_configurations.each do |config|         # Additional configuration options could already be set here         # BEGINNING OF WHAT YOU SHOULD ADD         config.build_settings['GCC_PREPROCESSOR_DEFINITIONS'] ||= [           '$(inherited)',           ## dart: PermissionGroup.camera           'PERMISSION_CAMERA=1',           # add additional permission groups if required         ]         # END OF WHAT YOU SHOULD ADD       end     end   end

Integrating ar_flutter_plugin in the Flutter project

• Replace your MyApp with the following code.

main.dart

import 'dart:io'; import 'package:ar_flutter_plugin/datatypes/config_planedetection.dart';import 'package:ar_flutter_plugin/datatypes/hittest_result_types.dart';import 'package:ar_flutter_plugin/datatypes/node_types.dart';import 'package:ar_flutter_plugin/managers/ar_anchor_manager.dart';import 'package:ar_flutter_plugin/managers/ar_location_manager.dart';import 'package:ar_flutter_plugin/managers/ar_object_manager.dart';import 'package:ar_flutter_plugin/managers/ar_session_manager.dart';import 'package:ar_flutter_plugin/models/ar_anchor.dart';import 'package:ar_flutter_plugin/models/ar_hittest_result.dart';import 'package:ar_flutter_plugin/models/ar_node.dart';import 'package:ar_flutter_plugin/widgets/ar_view.dart';import 'package:flutter/material.dart';import 'package:flutter/services.dart';import 'package:path_provider/path_provider.dart';import 'package:vector_math/vector_math_64.dart' as vector; class MyApp extends StatefulWidget {  const MyApp({super.key});  @override State<MyApp> createState() => _MyAppState();} class _MyAppState extends State<MyApp> { final asset = 'wooden_table.glb'; late ARSessionManager arSessionManager; late ARObjectManager arObjectManager; late ARAnchorManager arAnchorManager;  @override Widget build(BuildContext context) {   return Scaffold(     extendBodyBehindAppBar: true,     appBar: AppBar(       elevation: 0,       backgroundColor: Colors.transparent,     ),     body: ARView(       onARViewCreated: _onARViewCreated,       planeDetectionConfig: PlaneDetectionConfig.horizontal,     ),   ); }  @override void dispose() {  arSessionManager.dispose();  super.dispose(); }}

• If you want to detect plain surfaces as well as wall surfaces, then use PlaneDetectionConfig.horizontalAndVertical

Also read: Integrating Authentication and Navigation in Flutter Web with go_router

Callback for AR view creation 

• Add the below callback and it will be initiated when the native AR view is attached in Flutter.

Future<void> _onARViewCreated( ARSessionManager arSessionManager, ARObjectManager arObjectManager, ARAnchorManager arAnchorManager, ARLocationManager arLocationManager,) async { this.arSessionManager = arSessionManager; this.arObjectManager = arObjectManager; this.arAnchorManager = arAnchorManager;  this.arSessionManager.onInitialize(       handlePans: true,       handleRotation: true,       showWorldOrigin: true,       customPlaneTexturePath: 'assets/images/triangle.png',     ); this.arObjectManager.onInitialize();  this.arSessionManager.onPlaneOrPointTap = _onPlaneOrPointTapped;}

• handlePans: Set it to true if you want to drag a 3D object on a plane.
• handleRotation: Set it to true if you want to rotate the 3D object on a plane.
• showWorldOrigin: Set it to true if you want to show X, Y, and Z origin references on a plane.

Callback for plane or points tapped on view

• Below callback is triggered when you tap on a plane or object in AR View.

Future<void> _onPlaneOrPointTapped( List<ARHitTestResult> hitTestResults,) async { final singleHitTestResult = hitTestResults     .firstWhere((result) => result.type == ARHitTestResultType.plane); final newAnchor =     ARPlaneAnchor(transformation: singleHitTestResult.worldTransform); final didAddAnchor = await arAnchorManager.addAnchor(newAnchor); if (didAddAnchor == true) {   await _copyAssetModelsToDocumentDirectory();   final newNode = ARNode(     type: NodeType.fileSystemAppFolderGLB,     uri: 'wooden_table.glb',     scale: Vector3(0.5, 0.5, 0.5),     position: vector.Vector3(0.0, 0.0, 0.0),     rotation: vector.Vector4(1.0, 0.0, 0.0, 0.0),   );   final didAddNodeToAnchor =       await arObjectManager.addNode(newNode, planeAnchor: newAnchor);   if (didAddNodeToAnchor == false) {     arSessionManager.onError('Adding Node to Anchor failed');   } } else {   arSessionManager.onError('Adding Anchor failed'); }}

• Use a different NodeType based on where your 3D object is coming from. If you want to render your object from the Flutter assets folder then the “.GLTF” format of the 3D model is required.

Copying 3D models to the document directory

• This method is used to copy GLB object into the application’s document directory.

Future<void> _copyAssetModelsToDocumentDirectory() async { final docDir = await getApplicationDocumentsDirectory(); final docDirPath = docDir.path;  final file = File('$docDirPath/$asset');  final assetBytes =     await rootBundle.load('assets/glb/${asset}'); final buffer = assetBytes.buffer;  await file.writeAsBytes(   buffer.asUint8List(assetBytes.offsetInBytes, assetBytes.lengthInBytes), );}

• After that, run the app and when the surface is detected, you will see the triangle.png on the surface.
• When you tap on a surface, a wooden table object will be placed. You can drag it and rotate it using the help of two fingers.

Also read: Comparing OpenAI’s Language Models: GPT-3.5 and GPT-4

Conclusion

We can conclude with the fact that by combining augmented reality and Flutter, we are able to open the way for unlimited possibilities. AR has played a huge role in revolutionizing work in several industries like entertainment, gaming, education, services, etc., by creating a smooth blend of virtual and real-world scenarios and enabling creativity.

Flutter has adaptability features and the capacity to function on multiple platforms, and the compatibility strength of ar_flutter_plugin leads to a strong toolkit for building captivating AR experiences.
If you need assistance with building a Flutter augmented reality app, feel free to reach out to our expert Flutter developers at Aubergine Solutions. Contact us today, and let’s bring your AR ideas to life.