Learn about Pokémon Go.
Recently Pokémon Go is too red, plus their own technical control, see this article " Unbundling Pokémon Go " : how to use reverse engineering to get app source code, and analyze its operating mechanism, in this translation to share with everyone
This translation has been approved by Adrien Couque , as follows :
I do not know where to come from now, Pokémon Go swept the world in a week, and we found something interesting from it
Although but at The App IS Rate this page Currently only the Available in Three the Countries ( US, Australia and New Zealand ) , IT Still Makes Twitter and Facebook Go Pale . It Defeated Candy Crush to Become at The MOST successful Mobile Game in at The United States, not only proved to the bring Benefits To developers , local businesses also pay attention to Pokémon Go will bring those tourists , Nintendo's market value and then increased by 90% .
The game in such a short time to become a household name, inspired we want to see its internal structure. This article takes Pokémon Go App as an example to how to get the Android App program code from reverse engineering and analyze its network connection request for More information
Prepare APK
Before you can do reverse engineering, you must have an APK file, and it is difficult to get Pokémon Go 's APK file. Please note that installing APKs with unknown sources will be a high security risk, but Google Play will do some analysis to the reduce at The Risk of App , SO IT IS Best for people to download and install App through Google Play . But for Reverse Engineering, at The MOST like THESE Malicious APK , Because IT IS interesting. Here, WE are ANALYZING at The Release of Pokémon Go Version 0.29.0 for 7/7 .
First talk about, do reverse engineering, we still do not see some things:
· The Any now Build code -related Things
· The Any with the Continuous Integration and Testing (the Continuous Integration) Things
· Other Special versions ( EG Debug Version ) : During Development, you May have have some Special Features But by Will not BE Placed in at The Final Product, Because WE want to the Analyze at The Pokémon Go Version of at The Official Release, SO at The Special function Should by Will not Be put inside this
The code for the backend service. Many people may want to know how the algorithm determine the location where the Pokemon appears. But this algorithm is on the back end, we can only know how to transfer data with the back, can not know the logic Of the internal algorithm
APK content
Let's take a look at the internal structure of APK . In fact, APK is just a zip archive, which contains:
Here's a description of the function of each file ( green ) and folder ( red ) :
· Manifest is the Android Manifest , it's like App 's identity card, which provides information on the name, icon, version, privileges, restrictions and other hardware components. This is also required when the system installs or upgrades the App .
After the program is compiled on classes.dex , you can have more than one classes.dex .
The lib file is loaded with a library
The res and assets archive is a static resource file
· Resources.arsc IS Android A Special File, at The Compiler R.java Generated, Which IS and static Used to Link Resources.
· META-INF folders filled with the intermediary data (the Metadata) , but here we do not need.
Above is what you would see when you decompress APK .
We started to look at the first file: classes.dex .
Decompile the program code
De x is the Dalvik Executable abbreviations ( the Dalvik is Android system legacy virtual machine, now called new the ART , the full name is Android Runtime , but still use the file extension dex ) . This is the Android system-specific file format, and it Is not easy to read its contents. There are two ways to do this: the first use of smali disassembler will dex file content into easy to read bytecode , the second use dex2jar content into traditional Java files.
We intend to use the second method to convert dex into jar files (jar is a compressed file that contains all the .class files ) . Next we need to decompile the tool and then convert the . Script file into Java program code. There are a Lot of ready-made anti-compiler tools, have their own Jayx , you can use your usual, and even can find the online version of the anti-compiler .
We now have most of the easy-to-read Java program code, limited by the limitations of the decompiler program, and still some part of the program code can not be seen. In fact, there is a reverse compiler Procyon , which may have better output results.
It's important that the code we get is not the source code written by the original developer, just like using Google Translate to turn English into French and then turn back to english, you will get another string of new English. The reason is that when you Want to turn into French, according to the contents of the English word or phrase will determine the best corresponding word or sentence, once again turned back to English, according to the contents of the French will do a decision of the best best word or sentence Operation, the process of the back and forth are independent, the result will be different. This works like the reverse code of the program code: we decompile the program code, its operation will be the same as the original code,But the program code content will not be exactly the same as the original code, the difference may have a function name, variable name And annotations.
Fortunately, we can clearly understand the app used in the library:
· Android support libraries : support-v4, appcompat and support-annotations
· Various Parts of AT at The Play Services
· Jackson (JSON parser): core, annotations and databind
· Gson (JSON parser)
· Otto (event bus)
· Dagger (dependency injection)
· RxJava / RxAndroid (reactive Programming)
· Apache Commons IO (utilities for I / O)
· AdMob, now declared as firebase-ads (ads, analytics)
· Upsight (analytics)
· Crittercism , now known as Apteligent (monitoring and crash reporting)
· Unity classes.jar (interact between the Android framework and Unity)
· Lunar Mobile Console (Unity logger for Android)
· Voxelbusters's Cross Platform Native Plugins (mainly used for sharing from Unity)
If you are Android developers, you can feel strange: why there are two JSON parser ? One does the active programming ( annotation: author Ray Shih 's opinion on the relevant programming ) , a do event bus ? This is actually transitive dependencies : the library will be dependent on the operation, but write the program sometimes only call to several of the library, you can get here to understand how we analyze transitive dependencies .
Clean up some of the library without call, get a more concise list:
· Gson
· Crittercism
· Upsight
· Admob / firebase-ads
· Google VR SDK, Unity and associated
Another kind of dependency is from outside to inside, layers of wrapping up, like Upsight inside a large number of libraries, lists the list and the number of functions: RxAndroid (4k), Dagger (~ 200), Commons IO (1k), Jackson (10k), Otto (~ 50), various services (12k), the development of their own functions (3k) .
--- com.upsight.android:all:4.1.3 | --- io.reactivex: rxandroid: 1.0.1 | | --- io.reactivex: rxjava: 1.0.13 | --- com.upsight.android : Analytics : 4.1.3 | | --- io.reactivex: rxandroid: 1.0.1 (*) | | --- com.google.dagger: dagger: 2.0.2 | | Javax.inject: javax.inject: 1 | | --- com.upsight.android:core:4.1.3 | | Io.reactivex: rxandroid: 1.0.1 (*) | | | Com.google.dagger: dagger: 2.0.2 (*) | Commons-io: commons-io: 2.4 | | | com.fasterxml.jackson.core: jackson-databind: 2.6.3 | | | com.fasterxml.jackson.core: jackson-annotations: 2.6.0 | | | --- com.fasterxml.jackson.core: jackson-core: 2.6.3 | | | com.squareup: otto: 1.3.8 | | --- commons-io: commons -io:2.4 | | --- com.fasterxml.jackson.core: jackson-databind: 2.6.3 (*) | | \ --- com.squareup: otto: 1.3.8 | --- com.google.dagger: dagger : 2.0.2 (*) | --- com.upsight.android:google-advertising-id:4.1.3 | | --- io.reactivex: rxandroid: 1.0.1 (*) | | --- com. Upsight.android:analytics: 4.1.1 (*) | | --- com.google.dagger: dagger: 2.0.2 (*) | | --- com.android.support:support-v4:23.2.1 ( *) | | --- com.google.android.gms: play-services-ads: 8.4.0 -> 9.2.0 (*) | | --- com.upsight.android:core:4.1.3 (* ) | | --- com.upsight.android:marketing:4.1.3 | | Io.reactivex: rxandroid: 1.0.1 (*) | | | com.upsight.android:analytics:4.1.3 ( *) | | | Com.google.dagger: dagger: 2.0.2 (*) | | | Com.upsight.android:core:4.1.3 (*) | | | Commons-IO: Commons-IO: 2.4 | | | --- com.fasterxml.jackson.core: jackson-the DataBind : 2.6.3 (*) | | | com.squareup: otto: 1.3.8 | | --- commons-io: commons-io: 2.4 | | --- com.fasterxml.jackson.core: jackson -databind: 2.6.3 (*) | | \ --- com.squareup: otto: 1.3.8 | --- com.upsight.android:google-push-services:4.1.3 | | --- io. Reactivex: rxandroid: 1.0.1 (*) | | --- com.upsight.android:analytics: 4.1.1 (*) | | --- com.google.dagger: dagger: 2.0.2 (*) | --- com.android.support:support-v4:23.2.1 (*) | | --- com.google.android.gms: play-services-gcm: 8.4.0 -> 9.2.0 (*) | | --- com.upsight.android:core:4.1.3 (*) | | --- com.upsight.android:marketing:4.1.3 (*) | | --- commons-io: commons-io: 2.4 | | --- com.fasterxml.jackson.core: Jackson-databind: 2.6.3 (*) | | \ --- com.squareup: otto: 1.3.8 | --- com.upsight.android:managed-variables:4.1.3 | | --- io.reactivex : Rxandroid: 1.0.1 (*) | | --- com.upsight.android:analytics: 4.1.1 (*) | | --- com.google.dagger: dagger: 2.0.2 (*) | - com.upsight.android:core:4.1.3 (*) | | --- commons-io: commons-io: 2.4 | | --- com.fasterxml.jackson.core: jackson-databind: 2.6.3 (*) | | \ --- com.squareup: otto: 1.3.8 | --- com.upsight.android:marketing:4.1.3 (*) | --- com.upsight.android:core:4.1. 3 (*) | Commons-io: commons-io: 2.4 | --- com.fasterxml.jackson.core: jackson-databind: 2.6.3 (*) | \ --- com.squareup: otto: 1.3.8
This means that you have thousands of functions to analyze.
Although the library is a lot, but remove the analysis tools, monitoring tools, crash return and advertising, the most important left Pokémon Go game engine Unity . This is why you open the app will have a Niantic logo, in order to allow users to wait for a moment to start the Unity engine, and then a progress bar, the display engine to read the status of static files. All of your interactive operations are in the implementation of Unity environment, so will not see any Android native interface.
Another Note IS: the VR the SDK . The In The Stage of Pokémon Go Beta , someone Used The Same Way WE found Cardboard / the VR and OTHER words in The Program code, in The Official Version of The App use Statement Also Mentioned Cardboard . But from My Analysis , The future does not have the corresponding function of VR or Cardboard . From our professional point of view, VR SDK this library is used to connect the Android framework and Unity , but if you really want to integrate with the Cardboard ,You must let the Android framework and Unity can communicate with each other, it must refer to a large number of open source Can do it. But we do not see from the current code.
Here we spend a lot of time cleaning up the program, but there is not a real implementation of the project, but also need resources and assets , let us continue to look down.
Static resource file
To get resources and assets is also simpler than the original code. In fact, the assets will be packed into the App , almost all of the assets are used in Unity , so we temporarily regardless of them. Resources are interesting, they include icons , layouts and wording . The contents of Resources will become easy to read or edit after build . For example, the xml layouts file will be converted to binary format, and the 9-patches file will lose the basis for interpretation.
The good news is that there is a tool called apktool , which can help us to Manifest and resources files into easy-to-read format content, and produce an executable Android project. At first we didnt use apktool will be classes.dex into Smali file, rather than we want the Java program code.
Now with the decompiled resources and Manifest , there are also assets , plus the code will be cleaned up earlier, we can start to build and implement a complete Android project.
Compile and execute
In order to generate APK , we have to compile the Java program code before the need to create an Android project and build instructions. If you remember, because these things are not in APK , so we have to own, rely on: Gradle .
One of the interesting things is "the minimum demand Android version of the demand . " App 's minimum demand on Google Play is Android KitKat (Android 4.4, API level 19) , but in the analysis of the library, the maximum demand for Google VR SDK is only to API level 16 (JellyBeans, or Android 4.1) , we do not Clearly so Google Play claims are higher than the actual API requirements 3 versions. This is the beginning to exclude 20% of Android users ( according to Google ' S latest numbers ) , maybe intentional, and maybe a mistake.
But the most important thing now is that we already have a project that can be executed on the phone. If you want to install this reverse engineering version of the App , it is recommended in your build.gradle and Manifest components / permissions which first replace the application id , to avoid the conflict with the official version to ensure that the official version can be updated at any time.
After the installation is successful, you will find that you are stuck in the login screen. The first sign-in option is Google Sign-In . But when you click on it, it will verify the certificate signed by App , obviously we do not have credentials, so the error message is jumped out: GoogleAuthException: INVALID_AUDIENCE . In order to avoid this restriction, we have to spend a lot of effort to have a way, so the easiest way is to directly to the Google Developer Console to apply for a new App , so reverse engineering version of App can have their own credentials, and after login Get token , but still can not do with the back-end data exchange.
The second login option is through the Pokémon Trainer Club application account. But because too many people apply, the server seems to have been closed, and so it is restored, we will try to see if the reverse engineering version of App can log in.
Analyze the program code
Here we will examine look at the program code. Since this article is about an overview of reverse engineering, we will focus on Pokémon Go App , and the analysis of each app may not be the same
We saw most of the program code earlier in the Unity engine, because Unity is cross-platform, so these code can be executed on iOS and Android . But some are based on Android native features, such as:
· Sign-in / Registration (Inside at The Package Penalty for AT Penalty for com.nianticlabs.nia.account)
· In-App purchases (inside com.nianticlabs.nia.iap)
· Interaction with Location, Network and Sensors (inside com.nianticlabs.nia.location / network / sensors)
· Communication via Bluetooth with the Pokémon Go Plus (inside com.nianticproject.holoholo.sfida)
At first glance the most interesting thing is location / network / sensors program code ( if you fake your position or speed, the first time to know the location and type of appearance, and then you can catch more Pokémon words ...)
Communicate with Pokémon Go Plus , it should be when your mobile phone on the backpack or pocket when you can tell you near the Pokémon. This part of the code can be combined with the analysis of the network request to allow App only to inform you of the Pokemon you are interested, for example, you have not collected the only.
Slightly look at the code that communicates with Pokémon Go Plus :
Boolean notifyCancelDowser ( ); Boolean notifyError ( ); Boolean notifyFoundDowser ( ); Boolean notifyNoPokeball ( ); Boolean notifyPokeballShakeAndBroken (String STR ); Boolean notifyPokemonCaught ( ); Boolean notifyProximityDowser (String STR ); Boolean notifyReachedPokestop (String STR ); Boolean notifyReadyForThrowPokeball (String Str ); boolean notifyRewardItems (String str );Boolean notifySpawnedLegendaryPokemon (String str ); boolean notifySpawnedPokemon (String str ); boolean notifySpawnedUncaughtPokemon (String str ); boolean notifyStartDowser ( );
This is very valuable data! You can build your own device:
Intercept network connection
Do reverse engineering does not mean that it will take a lot of trouble to dismantle the code, you can from the App how to interact with the outside world, this method applied to any software.
App will be linked with the screen, to do the display or touch the interaction, in addition to: file system, sensors, networks and so on.
Here we are most interested in the network request. As we resume earlier, the most important game of logic operations are on top of the server, App need to exchange data with the server to do you can operate, if we can capture information on These transmissions, we might not have to pass through App can and communication.
An In FACT, Pokémon Go Used A Method, Called Optimistic Models the when Dealing with Network Requests . Optimistic Models the allow the Users to do AN Action ON at The App , do not need to the wait for at The Server's the Response, Directly to at The the Next Action to the Continue to Operate, SO That the player feel good smooth. If the server later error, it will jump out of the warning. So you can see when you are in the delivery of Pokémon, and did not show any waiting tips. App is currently in this mechanism has not been Very smooth operation, mainly because the server is loaded, I believe the next few weeks will improve.
So how do we fetch network requests? The easiest way is to have a proxy in the middle of the App and the server . But if the data is encrypted by HTTPS , you can only see irre something metadata .
There are a way called Man-in-the-Middle attack. This way you use proxy to deceive App you are Server , and then cheat Server you are App . When you receive the App request, use your app-side key first decrypt, And then server-side key encryption to Server to get a response, and then decrypt the server-side key , and then app-side key encryption back to App . You can get complete information, and App and Server will not know your presence The
Obviously, if the story is over, all the data on the network will be seen light. In fact, the use of encryption and decryption key is required to be third-party verified, that Certificate Authorities . Your phone or browser will only trust the Verified key , or back out of the warning message. Because the phone is our own, we can put the first key on the phone, to extract the information.
There are ready-made tools to help us complete the proxy settings, like mitmproxy and Charles . Charles has to pay, but the use of interface can guide us to make settings . The following figure is the Internet request that was intercepted when the App was Started
From which you can learn a lot of things, to see the first few requests:
· Https://android.clients.google.com/c2dm/register3: Sign up up notifications
· Https://stats.unity3d.com/HWStatsUpdate.cgi: Probably A Unity- Related the Analysis Event
· Https://bootstrap.upsight-api.com/config/v1/a9cc12f87adc420baf964f187672ecb4/: Upsight 'S First the Analysis Event
· Https://appload.ingest.crittercism.com/v0/appload: Crittercism 's first analysis event
· Https://pgorelease.nianticlabs.com/plfe/rpc: at The the Details Described below by Will BE
· Https://play.googleapis.com/log: Communicate with Play Services backend
· Http://lh4.ggpht.com/LakctgAXpXwe-3PMCWws8rCoVn1_TmyfAiWjWXm6VtsRjRl5v53n1JrWBumWmldzsBFxIUdRLXgsMewLjuyN: This is an Picasa request is PokéStop pictures
· Https://e.crashlytics.com: Communicate with Crashlytics , But Seems to IT BE failing
· Https://www.google.com/loc/m/api: GPS location
We can see that App frequently communicates with https://pgorelease.nianticlabs.com/plfe/ , and a 226 number is followed by the URL , and I guess it's done for load balancing : that's the first request is assigned to a server Go, then all the same session in the same request will lead the same server.
Finally, the "rpc" in the URL of the last thing on behalf of the app is through the Remote Procedure Call to communicate with the Server , so all the requests are sent to the same URL , which is not the same with the REST way The
Look at the contents of the request, neither JSON nor XML , and no compression or encryption: so we can clearly see UUIDs and "pm0015" and other strings, which may use using protocol buffers ( or flat buffers ) Do the serialized format. Charles Will help straighten it out, you can use protocol buffers in the Command Line , so from:
5, € â € "ÉßÛS # pgorelease.nianticlabs.com / plfe / 226: [
@ nrÝZ † ¡ ϯ½" 'ëXÖÐ _} ?? Î ~ -0' @ ... Ít '> - C ÷ ‰
Sorted into:
1: 53
2: 6032429073588813826
3: "pgorelease.nianticlabs.com/plfe/226"
7 {
1: "nr \ 026 \ 335Z \ 206 \ 241 \ 317 \ 257 \ 275 \ 224 \ '\ 353X \ 326 \ 320_} \ 220
\ 316 ~ \ 227 \ 361 \ 3670 \ '@ \ 205 \ 315t \ 221 \ 233-C \ 367 \ 211 \ r <j8y \ 024
\ 224 \ 312v \ 342 \ 2269 ~ \ 304 \ 202 / \ 036 \ 247 \ 276 \ 361 \ 266, \ 033s \ 027 \ 006 \ f ^ "
2: 1468599616357
3:" $ \ 002 \ 304 \ 337. \ 034 \ 270 \ 361 \ 214D \ 251nz \ 273fM "
}
100 {
}
100 {
} </ j8y \ 024
This is the request that pgorelease.nianticlabs.com/plfe/rpc returns, which has a new request endpoint: pgorelease.nianticlabs.com/plfe/226 , is used for all times requests.
You can also see a lot of "\ xxx" , which is "octal escaping" . Use the decoder , the content from:
Nr \ 026 \ 335Z \ 206 \ 241 \ 317 \ 257 \ 275 \ 224 \ '\ 353X \ 326 \ 320 \} \ 220
\ 316 ~ \ 227 \ 361 \ 3670 \' @ \ 205 \ 315t \ 221 \ 233- C \ 367 \ 211 \ r <j8y \ 024
\ 224 \ 312v \ 342 \ 2269 ~ \ 304 \ 202 / \ 036 \ 247 \ 276 \ 361 \ 266, \ 033s \ 027 \ 006 \ f ^ </ j8y \ 024
Becomes
Nr5Z617754 \ '3X60_} 06 ~ 7170 \' @ 55t13-C71 \ r
From the results speculated that this is like a list of objects that appear in the vicinity of Pokémon, each object has its own UUID and attributes ( eg pm0015 stands for pokémon 015 : Beedrill ) , the other may be coordinates, combat effectiveness and statistics. We can prove this from the request https://storage.googleapis.com/cloud_assets_pgorelease/bundles/android/pm0126 , if this request can get pm0126 related assets .
Continue to see the return of other requests. For example, the bottom of this should be the player's information:
{100
. 1 :.. 1.
2 {
. 1:. 1,467,925,951,134
2: "redacted: Player name"
. 7:. "\ 000 \ 001 \ 003 \ 004 \ A"
. 8 {.
. 8 :.. 1.
}
. 9:. 250
10 : 350.
11 {
}
12 is {
}
13 {
}
14 {
1: "POKECOIN"
}
14 {
1: "STARDUST"
2: 500
}
}
}
Number The at The 1467925951134 IS Unix timestamp , the Refers to at The 07/07/2016 21:12 , the this Should BE at The Player's Registration Time. An In at The Request and return at The Content, you CAN See Everywhere timestamp , some Accuracy to millisecond , some to Nanosecond .
And then further, we can see a lot of pairs of numbers, such as: 0x40486ddc40000000, 0x4002d99520000000 . This should be a coordinate, but not encoded in hexadecimal, but IEEE 754 doubles . The value of the hexadecimal value is converted into numbers:
Is the coordinates of our office! We will be able to get all the coordinates, guess its meaning, are marked on the map: the position of the user ( yellow ), points of interests / PokéStops ( red ) and possible spawn points ( green )
So far, we will read the network exchange of data, serialized format, but also to distinguish some id , timestamps and GPS coordinates, the other left to interested people to study.
Conclusion how to avoid being reverse engineering
See here, as a developer may feel no way to prevent others to do reverse engineering analysis, in fact, yes.
Blur your Java code is the first step: use Proguard . It will remove all the names of packages , fields and methods with random numbers, making analysis more difficult. If you want to do this analysis of the App , starting from the framework classes . Proguard is not only used in fuzzy program code, but also can remove useless resources and methods . Proguard is very easy to use, I think Pokémon Go should be used in the future.
There is also a way to reduce the Java program code, part of the function re% into native libraries , which will increase the difficulty of analysis, but the development is very convenient, and there are too many Java and native in series, will lead to Performance degradation
WE CAN Intercept at The Network Request Because at The App DID not use Certificate pinning . The Using Basic Android classes or OkHttp IS Very the Common and the Easy. But like Fuzzy Program code, IT CAN not resist at The Extreme Attackers ( Because at The Certificate CAN Also BE Reverse Engineering ) , But can lag them for some time.
Finally, this article is quite basic to the analysis, we did not expose the secrets of any game, open cheating methods make the game unfair. But for developers, you have to be careful to guard against professional hackers.
Here's a look at our findings
The program code is not blurred, it is easy to reverse engineer the analysis.
· We can rebuild executable project
· Dependency management library could be better
· Future without VR or Cardboard signs version
· May reduce Android version requirements
· We can read the code associated with the location / network / sensor and Pokémon Go Plus
· Easy to fetch network requests because of lack of certificate pinning
· Network requests is through protobuffers-RPC completed
You can find our reverse engineering version of the program code: Github contact us: Twitter