Hidden API of the mobile application: mitmproxy and proxy

```html

The site is protected by Cloudflare, Turnstile pops up on every second request, and the layout changes every two weeks. Meanwhile, the same service has a mobile application that connects directly to the backend and retrieves a ready JSON — without challenges, without markup, with a stable field schema. This is the "hidden API": an undocumented but fully functional interface used by the official client.

Let's break down step by step how to find it using mitmproxy, what to do with certificate pinning, and why everything falls apart during the scaling phase without a proxy.

Why even dive into the application's traffic

Scraping the web version and calling a private API are tasks with different costs. Compare:

Web. A headless browser is needed, bypassing anti-bot measures, parsing HTML, and regularly fixing selectors. One request = megabytes of traffic and seconds of CPU time.
Private API. A regular HTTP request with a couple of headers, the response is a compact JSON with typed fields. It often returns more data than the interface shows: internal identifiers, flags, service fields.

Mobile backends have historically been less secure than the web. The reason is simple: anti-bot platforms are designed for browser traffic (JS challenges, canvas, behavioral signals), and a mobile client simply wouldn't pass through them. Instead, developers rely on a static application key and TLS pinning — both of which are removed on the local device.

What you'll need

mitmproxy — an open-source HTTPS proxy interceptor (over 44,000 stars on GitHub, the current branch 12.2.2 was released in April 2026, requires Python 3.12+). It can be installed with a single command: pip install mitmproxy. It understands HTTP/1, HTTP/2, HTTP/3, WebSocket, and raw TCP, and works with TLS 1.2 and 1.3.
Android device or emulator with root. Experience shows that Android 7–11 is the most convenient: newer versions have significantly tightened certificate handling.
ADB for communication with the device and Frida (pip install frida-tools) — needed if the application pins the certificate.

mitmproxy has three interfaces on top of one engine: mitmproxy (terminal TUI), mitmweb (web interface, more convenient for beginners), and mitmdump (headless, for scripts and automation).

Step 1. Start the interceptor

Launch the web interface so that it listens for external connections, not just localhost:

mitmweb --web-host 0.0.0.0

By default, the proxy runs on port 8080. Upon first start, mitmproxy creates its own certificate authority and places the keys in the ~/.mitmproxy directory. Four files will appear there: mitmproxy-ca.pem (certificate along with the private key), mitmproxy-ca-cert.pem (certificate only), mitmproxy-ca-cert.p12 for Windows, and mitmproxy-ca-cert.cer — format for Android.

Step 2. Route the device through the proxy

In the Wi-Fi settings on your phone, select a manual proxy: the IP of your computer on the local network and port 8080. Next, open a special domain mitm.it in the device's browser — this is a built-in page in mitmproxy that automatically detects the platform and provides the necessary certificate format along with instructions.

On iOS, the procedure consists of three parts, and everyone forgets the second half: download the profile via Safari, install it in "Settings → General → VPN & Device Management," and then separately enable full trust in "Settings → General → About This Device → Certificate Trust Settings." Without the last step, the certificate is installed but does not work.

If you don't want to mess with Wi-Fi settings, mitmproxy has a VPN server mode: mitmweb --mode wireguard. The device connects using the standard WireGuard client, and traffic is intercepted transparently, without manual proxy setup in the system.

Step 3. The main wall — trusting the certificate

This is where most attempts fail. There are exactly two problems, and they are different issues.

Custom CAs have been out of favor since 2016

Starting with Android 7 Nougat (API 24), applications by default only trust the system certificate store. A custom CA is ignored unless the developer explicitly allowed it in the Network Security Config — through the <certificates src="user" /> block in trust anchors. This was a conscious decision by Google to reduce the attack surface, and it cannot be bypassed with phone settings. Chrome, by the way, does not trust custom certificates either. In Android 11, the restrictions have become even stricter.

The practical takeaway: on a rooted device, the mitmproxy certificate must be placed in the system store, not in the user store. That’s why root is on the list of requirements, not just "recommended."

Certificate pinning

The second wall is pinning: the application carries a fingerprint of the expected server certificate and refuses to communicate with anyone else. Even the system CA does not help here. A 2022 ACM study showed that pinning is prevalent in "high-risk" verticals (banking, taxi, crypto), but is often implemented incompletely and can therefore be bypassed.

There are several tools for this task, and they solve it in different ways:

Frida — modifies behavior at runtime: we hook the certificate verification functions and make them return success. The application is not modified in this case — the most flexible option. A typical command: frida -U -f com.target.app -l ssl_bypass.js --no-pause.
apk-mitm — automatically removes pinning from the APK file statically.
android-unpinner — rebuilds the APK, injecting Frida and unpinning scripts.
objection — a toolkit on top of Frida, works with both iOS and Android.
ssl-kill-switch2 — disables pinning in iOS and macOS applications.

If a specific domain is pinned tightly and hinders operation, it can simply be excluded from interception using the ignore_hosts option (accepts a regular expression) — traffic will bypass mitmproxy without decryption.

Step 4. Find the necessary request

Next comes the routine. Open the application, perform exactly one meaningful action (open a product card, scroll through the feed, apply a filter), and see what requests have appeared. In the terminal interface, this is done quickly: Z clears the stream list, Enter opens the selected request, E exports it — including a ready curl command.

What to look for in the intercepted request:

Endpoint and parameters. Often there are noticeably more than the application interface uses.
Client key. A classic example — a static identifier embedded in the application. In a well-known analysis of the public API of MyAnimeList, this key turned out to be the header x-mal-client-id with the value 6591a087c62b3e94d769cd8e35ffe909, which opened access to the endpoints api.myanimelist.net/v3/anime/season and /v3/anime with two dozen parameters.
User-Agent. Mobile clients have a specific one that serves as part of the "pass" — in the same example, it is MAL (ios, 139).
Tokens and their lifespan. Immediately check whether the key is static or updates: this affects the entire subsequent architecture of the collector.

The exported curl can be conveniently converted into code using curlconverter — you will get a ready request in requests, and then you can work with a regular HTTP client, without any browser.

Step 5. Scaling — and where everything breaks

At this point, disappointment sets in, familiar to anyone who has tried: from one home IP, the private API responds perfectly for the first half hour, and then starts returning 429 and 403. Mobile backends are less protected against client spoofing, but IP limits are stricter there — the server assumes that one phone is behind the address, not a parser with twenty threads.

From here come practical conclusions.

Keep the request profile plausible. A real application does not make 50 requests per second and does not operate strictly on a schedule. The order of calls also matters: a real client first requests session configuration, then content.
Distribute the load across addresses. One IP = one "phone." Strategies for rotation, jitter delays, and exponential backoff were discussed in detail in the article how to bypass API rate limiting when scraping through proxies.
Consider geo. Many mobile APIs return different content and prices depending on the country of the address — this is both a limitation and an opportunity.

Debugging is conveniently done without leaving mitmproxy: it can connect to an upstream proxy. The command mitmdump --mode upstream:http://example.com:8081 redirects all traffic upstream, and authorization to it is set with the --upstream-auth option in the format username:password. This way, you see the same requests as before, but they are sent from an external address — you can immediately check how the API responds to a specific country or type of IP.

What type of proxy to choose for a mobile API

The choice here is not abstract; it stems from who you pretend to be.

Mobile proxies — the preferred option. You emulate application traffic, and the mobile operator's address looks completely organic to the backend: behind one address, thanks to CGNAT, hundreds of subscribers actually sit, so limits on such IPs are softer. Suitable 4G/LTE mobile proxies.
Residential proxies — a working middle ground if the volumes are large and the operator binding is not critical: home IPs from providers offer broad geo coverage at a reasonable price. These are residential proxies.
Data center — only for endpoints without serious address reputation checks. Their ASN is recognized instantly, and it looks strange on a mobile backend: there are no phones in data centers.

Pitfalls that are learned too late

HTTP/3. Support for QUIC in mitmproxy is available and enabled by default, but on real mobile traffic, it is limited: often the connection has to be forcibly rolled back to HTTP/2 through manipulations with ALPN. QUIC works best in reverse and WireGuard modes.
The private API changes without warning. It has no obligations for backward compatibility — this is an internal interface. The application version in User-Agent will eventually stop being served, and the collector will silently start receiving empty responses. Monitor not only response codes but also the structure of JSON.
Don't confuse "found the key" with "gained permission." A static client key is not a permission for unlimited scraping.

On the legal side

Intercepting traffic on your own device is a legal and common debugging practice used by mobile developers and security specialists. The boundaries start further: comply with the service's terms of use, do not collect personal data without legal grounds (in the EU, this is directly regulated by GDPR), do not touch endpoints that require someone else's authorization, and keep the load at a level that does not interfere with the service's operation. A practical guideline: if the data is visible in the application to any user without logging into an account — you are in a relatively safe zone; if access requires someone else's account — you are already beyond its limits.

In short

The scheme works and saves weeks of hassle with anti-bot measures: we start mitmproxy, place the certificate in the system store of a rooted device, if necessary, bypass pinning through Frida, catch one meaningful request, export it to curl, and rewrite it in Python. From there, the task of "bypassing protection" turns into the task of "carefully distributing the load" — and is solved by rotating addresses, reasonable pauses, and the right type of proxy. It's easiest to start with mobile proxies: they are closest to the traffic that the backend expects to see.